Scale By The Bay 2018: Pat Helland, Keynote III: Mind Your State for Your State of Mind
Recording: Scale By The Bay 2018: Pat Helland, Keynote III: Mind Your State for Your State of Mind
my name is Pat Hammond I've been building distributed systems databases transactions messaging systems multiprocessor since 1978 and so a lot of things have changed since then at that time Jimmy Carter was president and at that time I had hair and so we're gonna have to move ahead now now these are some thoughts I've had on how storage and computation are evolving and how they can lead to interesting challenges I'm going to go down a little bit okay so these are my personal observations this is not actually to do with Salesforce it's just how I think about things and I do for recreational reasons write papers including this one and so this paper is one of a number that I published in communications of the ACM if you like this topic you can go just you know search for it online and there's everything I'm going to tell you today is in a paper and so if you care so introduction we're gonna talk about this we're gonna talk about what state is what's this state stuff we're gonna talk about how durable state semantics have evolved we're gonna talk about session state semantics and how that relates to transactions some of the work I did in the 90s how that's evolved away how microservices impact with that identity immutability and scale how do they work together with this crazy systems we're building walk through some example application patterns so I want you to see that the patterns are real what we're doing in here have different implications based upon what business needs in the conclusion and the big takeaway here is different business needs cause different systems to be built and so there's no one right thing it's not one size fits all so trends in storage Sturgess vault used to be you had a computer it had a disk it had some stuff and you used it and that was it it was all you had this all she wrote then you had shared appliances like a storage area network then you started having clusters in the network and then even rest api's over microservices in my mind that is an aspect of storage it's a way of thinking about it one of the things that's interesting about that is the a poppy comes the store and so that's a piece that I think is fascinating we won't dig into it too much today computing has evolved back in the day there was a single process a single mainframe region on the big mainframe then you started getting multiple processes in the same server you could work with your app and how did they talk to each other and work together then you you know what does it mean to do an RPC across a tiny cluster I was there watching these things begin what does it mean to call the computer that's plugged in next to the first computer how does that work services and service-oriented architecture I was the first one that I know who was talking about with it at the time I called a fiefdom a protected environment whether it's with its own data and you would call into that app and it would do the data stuff and that's today a service-oriented architecture it's evolved or DeVault or both to micro services it's both and so how does that fit together with this world micro services with little or no state what are they like now computing use of Storage has evolved with this direct file i/o you used to need to do careful replacement we're gonna talk about care for replacement care for replacement has a couple of variations what that means is first of all if I do a write and it annihilates the contents of what I'm overriding before it makes it good which happened back in the day right so you would have the old value then you'd have trash then you'd have the new value if that happened and I needed one or either the old or the new value to put it all back together I'd have to write somewhere else and then come and recover it that's the first one the second is that the client crash might interrupt a sequence of rights and I've written two out of five things that I need to write and now what happens if I only have two out of five after a crash how do I put that together now transactions a lot to do with transactions I've been working on them since 82 in Oh 78 really right how does this transaction stuff fit together and it provided careful replacement for the app it was great you'd say begin transaction here's a bunch of stuff do it all in transaction I'm a happy guy you didn't have to worry about it being half done that was huge for people writing the apps right and so we provided the stuff that took care of that plumbing underneath of trashing the disk when you were doing a right I know cuz I coded some of that later sands were providing the storage area networks big arrays which looked like a big disk we're providing you know battery backed up ram and stuff like that so you could actually not have to worry about trashing the contents and then you get into stateful to tea tear transactions and how did that work now we've grown to the point where that doesn't scale so people are saying hey let's do a key value store and you don't have to worry your head about where this Keys stuff is you just come at it we'll figure out how to find it we'll update the key and the value but you can't do two of them together if you do two of them together get a transactional behavior well we'll move one of the keys and it'll be broken later so that's not so good right now rest puts they invoke app code but who knows what that does to the state that's a whole other talk right so there are challenges in modern micro service based apps today micro services power many scalable apps there's pools of equivalent services incoming requests are load balanced across the pool I love this picture this is from the 2007 SOS P paper on Amazon dynamo I helped with the initial design of dynamo I didn't write the paper and one of the things that they're articulating is client requests come in there they're spread out a bunch of pool of equivalent servers at the time it was not it was full servers but it's the same as what you're seeing with micro services today and then you would call something which would go through request routing out to another pool of equivalent servers and then do again to know their services and each of those services their pool would have access to its own storage which probably was built to be scalable right and you have the semantics of that so the load balancing across that meant the second or third request was not guaranteed to hit the same computer that's huge it's huge in the ability to build and deploy systems it's also a real pain in the rear when it comes to state semantics we're going to talk about that micro services must support many operational needs for example there's health mediated deploys that's known as Canaries so with a health mediated reploid deploy say I've got a hundred computers doing the same work and sometimes it's not enough I can add another 20 to get 120 servers doing the same amount of work right if that's too much I could pull it back to 80 as long as my response time is managed in terms of threat traffic on to that stuff when you're rolling out new software it could break things and you got 100 of them running you don't install it on 100 them at once you install it on five and then you stand back can you watch how'd that do is there Eddie happy things getting sad we're getting complaints how about another five and you'd roll it out that's called Canaries because coal miners back before they had things to sense methane would bring little yellow birds into the mine when they're digging coal and when the canary would go thunk then you would say oh oh and you get the heck out of the kit out of the coal mine that that is why they're called Canaries today in this system micro-services are phenomenal for helping with health needed helping with health media to deploy in Canaries so rolling upgrade you want to support fault zones you don't want to break multiple fault zones with the first tentative stuff and you need to be able to design this for fault tolerance durable state is not usually kept in micro services because you can't easily get that state atomically updated you have to kind of roll it out like you did the software especially when these things are coming and going and crashing and restart it's even hard to get something went down and went back up and it missed the new version of the software or the new version of the data so typically the latest state is kept elsewhere and versions are cached and sometimes you read through the requests to the place you kept it like in here you're going back down to these Dynamo instances to get the state that you need for that it's a typical stuff so what is this state stuff what's it mean so when I hear about durable state accession state when I hear durable state I think the stuff is remembered across requests and the stuff survives when things break now there's a whole question I'm not going to dig into which is how durable is durable right can you read it subject to computer failure can you read it subject to data center failure can you read it subject to thermo nuclear exchange right it's it's all of these questions about durability need to be considered but I'm saying kind of this does it survive a system crash a single node failure is what we usually think of for durable state and again there's a broad discussion there so databases file systems key value stores caches are all that way how is it updated is it updated with a single update is it updated with a transaction is it updated with the distributed transaction do I can I could I should I have careful replacement and if so what's the granularity of the pieces I have to replace we're going to talk about that a lot do I need messaging semantics to update the various parts of the data with the state that are there the next really interesting question can you read your rights consistently if I write to the store and I go back and I read it do I see that latest thing that was written that's known as linearise ability because the key value if you will think of it as key value the value goes through linear updates one after the other after the other okay and weekly consistent stores and cashing make it hard to do read you're right you don't necessarily get the answer you wrote most recently next we have session state session status stuff that is remembered across a session to two computer soft pieces of programming talking to each other but not necessarily across crashes so session state exists within the endpoints associated with the session multiple operations end up being a form of session state session state is hard to do when the second request to a micro service doesn't necessarily land in the same place as the first so does that make sense you're remembering what I'm saying to you as we talk but I'm actually to come back and talk to your twin brother twin sister okay that's not easy to do we don't usually do that right and so it's difficult different micro services can get different stuff and so session state kind of falls away when you have micro services now what's data I wrote a paper in 2005 called data on the inside versus data on the outside and I was trying to point out back in the day the only data we talked about was inside the relational database that's all it was and if it wasn't in there that was like whatever we don't want to think about that don't worry our pretty little heads about it now the tables the rows the columns the values within the cells are what we thought of and that data lives in at one place the database and one time the transaction transactions provide this ordering that's actually called serializability can I make the entire database look like one thing after another happened it doesn't have to be serial it has to be such that is serialized double so if you update this and I update that I can put them in one or the other and I can make it look like one thing at a time happened loosely translated the pro doesn't see change underneath them during the transaction all phenomenal stuff all getting orders of magnitude more crazy as we get bigger into distributed scalable systems but that's what it was that was all we did back in the day and we pushed it till it was fallen over you know what his captain she's gonna blow and now you got other things to think about because we've broken that ability to make that easy there's ways to mitigate it but it ain't magic data on the outside is different messages files events key value pairs those scalable key value stores don't offer transactional serializability so the ordering is different it's not data on the inside it's data on the outside unlocked data is not stored in the classic database but what I've observed is that the act of unlocking it means you atomically get you actually get identity an optional versioning immediately when you end up and you know adopt that stuff when you unlock it it becomes ID it with identity in versioning outside data is immutable but maybe versioned each file message key has a unique identifier that might be a URL might be something else might be implicit on the session but there's a thing that identifies it now that it's popped out of the database how has durable states have changed well we used to have to do careful replacement I had to code this way back in the day way back in the day used to be disk writes my trash a block you would see old value unreadable trash new value a power failure was a real mess because you'd wake up and that block was nothing you didn't have the old value anymore you didn't have the new value so imagine the following I'm updating version 1 of the file the act of updating the block creates trash and then I get version 2 a power failure in the middle left me stranded and doomed so careful replacement for a single block right with mirrors what we coded what I ended up coding was when you write the new value in to some other place a parallel file so I have two parallel files I get mirrors I write it into the first one I get it there now it's there great I'm good I got it if we power fail I got the first file and then I write into the second one and then we're good and that way there was no window of time in which a power failure left it unreadable if I came up and one of them was unreadable I'd rewrite it so I could get it to be good again careful replacement with a single block writes for non mirrors so you could do this game when it wasn't with mirrors where you'd write to a different block you'd get a new version on the different block you'd come back up with that new version and then you'd write it back out and now you didn't lose things the point I want to make is that you had to walk through this it was like walking one step at a time across a creek where you had to have one foot on solid ground because things would go awry we still have it different granularity we'll talk about that careful replacement with record rights so updates to records and pre sequel databases had to be carefully ordered in many cases and update to one record X before updating Y allows the record the application to recover so if I update record X before Y I can figure it out after a crash if I update Y b4x I'm doomed a great example of this is an application Q if I write a record in the queue which as I'm gonna do this stuff to the database and then I commit and then I read that and I do this stuff to the database in a fashion where if I crash part way I can restart it that gets me out of this is a careful replacement but the granularity is records in the system whatever system it is values and key values whatever it is it's a way of walking your way forward and knowing how you stand and can recover transactions bundled and made this awesomely easier for decades they solved careful record replacement because you could wrap a bunch of records together I can say begin transaction change 20 records end transaction I never got 19 I never got one I always got 0 or 20 and that was huge solve a lot of problems it solved careful storage replacement because when writing to the physical disk the database and the transaction system made sure to cover this I'm gonna write it in this block and write it in that block you mister application programmer but don't have to think about it huge huge working across time though oh my gosh I want to do some stuff and then wait till something happens and then do something else that was multiple transactions so I had to think about how do I record my state while waiting for the event that's gonna move me to the next date so I have a careful replacement action in order to move transactions across time but at still careful transactional replacement and finally working across space and working across trust boundaries company a talking to Company B I have to do a transaction and remember I'm gonna ask him to do this I'm gonna figure out how to do that he's gonna do this we're gonna walk our way forward and restart Andrey Drive when needed it's all the same kind of stuff we've seen Yogi Berra said it's deja vu all over again when I see this careful replacement I think back to my youth when I had to worry about thrashing the disk blocks that makes sense so messaging semantics are pretty cool transactional message is pretty cool I can do transaction one I can write down transactionally I want to send a message and I could atomically consume a message it I could do transaction two which is yeah suck up the message and do these changes I can make sure it's exactly once how do I do that well when I do this I keep trying until I hear it's been sent that's at least once delivery and then I do it most once consumption at most once processing but this has got some onion disgust challenges to do at most once processing I actually have to remember the messages I've received now that may mean I know I'm talking to 20,000 different partners and partner number seventeen thousand nine hundred and forty two has sent me five hundred and thirty one messages but that actually doesn't scale over time and so I argue and I see in real application systems this works by aging things out over time let me give you an example you all know when you write a physical paper check if any of you still do write that that paper check is an expiration of a year and it has a physical number it's check number 103 so the bank will remember for a year that check number 103 has cleared that allows them to be idempotent if it bounces and comes back a reach reach resubmits they'll only process it once but there is a window of time because they don't want to remember infinite state to ensure at least once delivery does that make sense to everybody okay all this junk weaves together so how do you remember the messages you got to detect the duplicates how long does the destinations if I do this for a server and it's actually to a particular piece of State on the server and I cut it into how do I manage that this scalable computing stuff is really hard while guaranteeing at least once this one's really interesting read your rights yes or no used to be back in the day if you wrote something you could read it you know used to be when I was a kid if you picked up the telephone there was a dial tone but you had a wire to the wall okay linearizable stores offer read your rights so if I write X to be value 20 and I go to read it it's going to be 20 until there's another completed write that takes it to 21 or whatever right linearizable means occasional delay for a long time when a server is sick or dead let me say this again if I'm going to have a store which handles normal triple failures behavior in the database or even if it doesn't handle triple failures sometimes I go to change it and it will be slow why is it slow one of the three computers died and the rest immersing is he dead I think he's dead what do you think he's dead I don't know is he dead before they decide to kick him out and then move on right or one of the three will be garbage collecting or something servers have this incredible low latency until they don't and then it spikes aggressively if you have the ability to read your rights in a store I assert for discussion and we can debate it in a break that you have a most systems that implement that will offer delays sometimes so that's a challenge to be considered so do you get fast predictable reads fast predictable writes and read your rights in a linearizable store I argue you don't get fast predictable reads and you don't get fast predictable writes but you can get read your rights the reads are not as hard the rights are harder a nonlinear izybelle stores do not offer read your rights a nonlinear izybelle store there's no guarantee the right will update all of the replicas might read an old value reading and writing have a very consistent SLA skip over sick or dead servers great example is that Dynamo system 2007 SOS B okay the Dynamo system what you do is you write and it will find three servers to write to and if one of them is slow or sick it will keep going around the loop to find another one so you can skip over making the change to a value to a server that has an old value but by golly you're gonna write it three places and when you read 99 plus percent of the time you'll find the new value and it's fine one you know way less than one percent you'll go and try again you'll get the old value you say why would you do that well because it always reads fast and it always writes fast and it turns out there's business cases for which that's better that's the the major theme of this discussion okay we're gonna get there so in a nonlinear izybelle story fast predictable reads yes fast predictable writes yes read your rights no cached data offers scalable read throughput with great SL a's if i'm updating a read value pair and i'm cashing it across 100 servers you guys aren't too surprised at the fact that sometimes i'll read the old value as it's updating the cache that's to be expected and it's very valuable for certain business uses it's actually not valuable for transactionally correct business state that you want to make sure you got the latest value but it's really value for a lot of valuable for a lot of other things so scalable cash fast predictable reads yes with scale fast predictable writes now so much read your rights no so these are different stores for different uses is it okay to stall on a read is it okay to stall on a write is it okay to return a stale version you can't have everything so there's a great paper great blog article called linearize ability versus serializability by Peter Bayliss so immutability it's a solid rock to stand on sometimes we store immutable things one example just an example is I got a bit pattern it's associated with a UUID that's 128 bits no other bit pattern will ever be associated with that unique ID that means I cannot get an old value because there's no old value there's one value that's what immutability means one identifier one value never changes with the value storing immutable things can change the behavior of the store because you don't ever get an old version and you don't worry about all this read you write junk right so this is the slide the picture from the previous slide if I put immutable data on top of it well the linearizable store it still reads your rights but is slow the nonlinear as well store gives perfect behavior for immutable data perfect behavior because it's fast it's fast it gives you the one answer right and scalable caches are fine they work fine too great so interesting options with applications with immutable data nonlinear as well gives fast and predictable writes and reads and does what you want it's at C R it's an immutable store it gives you back the only answer for the data in serial caches give you lots of fast and predictable reads that's cool interesting stuff and as Alexa you mentioned there's a paper I wrote three or four years ago called immutability changes everything which talks about all sorts of quirky aspects of immutability so session state semantics and transactions same process different process honest to gosh when I started working on databases they were libraries running in the application okay that's how it was there was none of this cross process stuff and then we started to see that the application would trash the database and maybe that wasn't so good then you started moving them into two processes on the same computer so the database and app were split apart and connected by a session and then you started to put the state on the session oh yeah what user ID is this coming into this database what transaction am i working on what's going on and that came across by decorating the ability to call another process in the same server in order to let the app be separate and to know how to keep the database straight and this is by the way the heritage behind JDBC connections that you see today that are so common so later still the app and the database moved to different servers so we were running them in the tutee or an end to your environment to tier at this point where you've got an app servers a bunch of them surrounding a database server stateful sessions and transactions so stateful sessions were the natural outcome of shared processes it's like you're going to talk to me ok you can talk to me again we got this session I know you're in the middle of the transaction that's fine send me another request on the session that's cool it's still in the same transaction we'll go along for another 10 minutes it's fine whatever we could do this stuff right you knew the who the person was you knew how it worked they worked well for six service-oriented architecture a message is coming in from a distress diss trusted party into an app server who is trusted who was taking the message apart scratching its head saying do I want to do that for that other partner and then doing the work against the database so you were had the database was surrounded by the protectors of the application who were then dealing with other things it's a classic service-oriented architecture so here I have database one in database two and I could have client one talking to app a which was talking to app B which is talking to database one now this is I built the first one of these spec at Microsoft client two is talking to app C you've talked to app D again back to those two and if you're doing a transaction you had to have distributed transactions and Microsoft transaction server did this the propagation of transaction State across sessions across n tiers was something that was cool and new in the middle of the 90s I was very proud of it it's worked out to be more brittle than I wish it were and so it's not what people do today because when things break it kind of gets locked up and so but it's a very interesting I'm proud of it with a lot of fun so transactions sessions and micro services so I had now had an interior system now all of a sudden we're going kind of gonzo with micro services but micro services are cool right they stink when it comes to session state they just do that session state that we talked about breaks down now because I can't go across multiple servers usually go back to the same micro service instance but it doesn't always and so you can't easily do a transaction you need that session state to do a cross request transaction let me say this again if I want to send a request in the middle of a transaction get any answer and send another request in the middle of the same transaction and get an answer I need session State to correlate those I can't do that when the request is talking to micro service a micro service instance number one who's talking to the back-end data and then he gives me the answer and now it goes to micro service instance number three and off we go that doesn't work well at all it's very brittle so micro services and transactions are typically one store requests so you get to the micro service this is what am I going to do I'm going to update the key value store great that's great but guess what you don't end up with two-phase commit on that because it's really hard to make it work microservices are worth the restrictions fail-fast load-balanced health media - poi Canaries rolling up grades fault tolerance they're good they're worth it but now we have pushed micro services back into careful replacement of the key value store so they're gonna update value one scratch their head update value - and the programmer has to reason about failure modes between them I didn't say it's wrong I said it's trade-offs it says stuff we need to understand so it's not your grandmother's transaction anymore transactions only work on a single call to the store scalable microservices as the application microservices scale more instances are made as micro services compose they call each other scalable stores transactions across multiple identities leads you to distributed transactions across stores but that's a pain in the tush scalable linearize ability do i have per identity read your rights scaleable nonlinear eyes ability do i have per identity read one or more old versions so let's talk about identity mutability in scale what's identity each identities represented by some numbers string uri i can talk about a specific New York Times for a specific date I have an abstraction that's different which is the generation Eric today's New York Times each version of the identity is immutable once I go to today's date of the New York Times and the San Francisco region for the New York Times I get a same bit pattern that will be forever right that's not today's New Times necessarily it's an immutable value version history may be linear version history may be a directed acyclic graph this is what happens when you can't read your rights this is I always read the right I wrote the new one I reread the old when I wrote the new one this is I didn't get a chance to necessarily see the oldest one as maybe saw the latest when I saw one that was previous and now when I'm writing this one it's combining two things and so you end up with the state of your data and that happens cross identity relationships using care for replacement across identities as tried and true so if I do a first I do B second then I've got no window which it's bad everything is good I made sure that the update to be happened after the updates a careful replacement is predictable when the store behaves itself if the store is not one that behaves itself well-defined behave yourself is not linearizable then the replication of stuff can mean that I get the window out of order and so I can see this window here in the middle where these things I've updated this one before this one as far as I can tell when I'm looking at the replicas and so your app will break if you're careful replacement workflow can't handle this being updated before that being updated you just broke it by using a nonlinear as pull store so that's a thing to consider care for replacements buggy / nonlinear as opposed doors okay let's dig into some app patterns care for a placement / a key value obvious are uniquely identified by their key workflows captured scalable apps can be built over key value stores this is what you do when you're working with another business you're doing b2c you're doing much of b2b depends which parts of it but you're trying to make sure that the state is accurate and so now you have your workflow you try to manage all of that stuff in the time window careful replacement / key value is very valuable transactional blobs by ref imagine I have an app which is storing an immutable up but immutable blob I do a transaction to identify the blob I copy a big document out into some store that holds immutable stuff and then I do an answer this could be a nonlinear as well store because the data is immutable and the blob is implemented with many commodity servers and because it's nonlinear as well it's fast this one's fun shopping carts nobody believes that multiple customers have to have transactional behavior across two shopping carts that's not interesting right you want to make sure each shopping cart is updated by itself but it turns out that customers are very unhappy if the shopping cart stalls you can measure ecommerce sites you absolutely know if you take 10 seconds 20 seconds the customer is likely to go do the dishes and they won't buy something if you give them back an inaccurate contents of the cart the go huh and they'll fix the cart and she keep shopping so it's actually much much better to give an inaccurate shopping cart quickly than to give stall to get a accurate shopping cart shopping carts should be right now even if they're not right this is a tagline I want to introduce do you want your data right or do you want your data right now because you will design your systems differently for that in a nonlinear izybelle store sometimes multiple old versions exist in the version history dag sometimes you go back and you give an old one so again low latency predictable reads yes low latency predictable writes yes read your rights no wrong shopping cart rarely but sometimes yes it's ok meets the business need product catalog feeds and creeds feeds and crawls processed by the back end big batch system getting information from merchants getting information from manufacturers creating the descriptions of products each product has a unique identifier the identifier takes you to a partition the partition takes you to a replica the replica takes you to the product descriptions that make sense to everybody ok so I read this stuff I process it and I'm plugging the data in there the latency of updating the product catalog is not a big deal mostly it's ok it's not like a 2 second stall is gonna matter you're doing these big batch systems use your catalog lookups are latency sensitive incoming request comes in Product ID figure out the partition go to the partition ask a replica if he doesn't answer fast enough ask another one do I care if it's racing with a concurrent product update product description update no do I care if I get a slightly old one no I care that I get one quickly and that it's pretty good low latency predictable reads yes writes it slow write read your rights know lots of scale lots of scale search is fascinating especially for an old database guy who thought I had to be transactionally corrector was evil right I now know it's not true it's okay to be non transactional web crawlers search them index the index the updates to the index are not super latency sensitive a very important search requests get low latency response if you are a systems person you should read this paper the Taylor scale by Jeff Dean and Louise Andre Barroso communications of the ACM February of 2013 fascinating how they do retries to bound the latency but the retries by definition must be idempotent which means it doesn't matter if you get the different answer from a different server it's really cool really interesting but they do this to make the reads fast remember they have to hit all the partitions and get the search information back from all the partitions so one laggard causes big latency problems fascinating issues for systems guys lot Cisco conclusion I'm trying to get through this so I'm not late it's about the application pattern so low latency predictable reads writes read your rights care for replacement you don't really want to give the wrong answer because your workflow your algorithm you're careful replacement stuff gets broken so you wait to get the right answer it's fine I think it's important to recognize when you need that transactional blobs by ref is just one example of where the immutable storage of stuff can in fact be on a nonlinear izybelle store it's easier to manage easier to do a lot of things but gives you a really fast answer but because it's immutable it's never a wrong answer ever the worst that happens is you go to a server and it's not there and you try another one and then you get the one and only one answer that exists ecommerce shopping cart I want it fast I want it fast it's ok if it's sometimes wrong it's great ecommerce product catalog I want to read it fast I don't really care if I get an old version let me know when it's a new version it's not a big deal search I really want fast predictable reads I even want to do tricks to run to get multiple reads together that's what that paper is about I referenced you to the detail at scale linearizable reads and read your rights are not always required I want to say this again I am a recovering transaction guy I love transactions but I think they have their place linearizable reads are not always required depends upon what your business needs takeaways state means different things session state and by the way I always do my this slide first two takeaways let that's what I hope you remember okay state means different things I have session state I have durable state session states across running things durable states across failures most scalable computing comprises micro services with stateless interfaces well stateless interface means you know or remember anything when you go to another guy it's okay great but they need partitioning failures rolling upgrade stateful sessions are part problematic micro services can call other micro services to get data or get stuff done transactions across stateless calls usually aren't supported in micro service solutions I don't really know of any frankly micro services means no server-side session state which means no transactions across calls which means no transactions across objects okay let's move on let's work with it coordinated changes use careful replacement technique for computing early days each update provides a new version with this single identity complex content within the new version has include many things including incoming outgoing messages finally different applications demand different behaviors do you want it right where you read your rights or do you want it right now bound it humans usually prefer right now to write they usually would rather get a wrong answer quickly then wait for the perfect answer that's usually many app solutions are bounded on object identity have to be tolerant of stale versions and immutable objects solve a bunch of problems and so that is my last slide other than the fact that I work at sales first I like working at sales first I some of you would too and so that's all I'm saying you can ping me if you want so questions we have time for like I'm like two minutes early I'm sorry I'm cutting it close and so do we have questions I can repeat it if you want you awake it's not my job any questions okay we have lots of cool projects I mean I'm not an ml or AI guy I'm not although I'd like to learn more about what the plumbing is like for it I'm not close to it we're doing big data stuff we're doing database and infrastructure stuff I mean historically our focus has been how do you do classic enterprise computing classic don't make a mistake give it to people right give it to let people build their app I will tell you look my entire career well at least since 1982 has been focused on how do i do enterprise computing and i define that as make it easy for the person who's not a computer science specialist easy for someone who wants to think about insurance to build an insurance app right and that means we have to do all the hard stuff underneath for some definition of hard stuff the insurance apps got is huge challenges to but that I'm completely oblivious to but making that easy what's fascinating is doing that under scale right so historically the the Salesforce scale challenge has been we have lots and lots and lots of companies each of which has got their own environment and we have to support that correctly and run the the data center and the support and the storage and the disaster recovery in the failover and high availability all that stuff's fun do that so people don't have to think about it now we're seeing new models coming in you know IOT is starting to influence things we're seeing big data coming in and influencing things and so I love both the change of the industry and I love the scale and frankly Salesforce is a really nice fun company to work for with good people with good attitudes I will tell you you know going to work with people you have fun with makes a world of difference and so that that's kind of my answer did that makes it did that answer your question Thanks that was part of the sales pitch that was great any other questions come on you guys yes go ahead I'll hold it out so so I mentioned distributor transactions at Microsoft why is that a challenge I first did two-phase commit in 1982 when I was at tandem and this individual servers there when something broke they kept going for a long I mean literally there was an 11 year mean time between failure for the entire company's install base of over 4,000 customers okay and so I didn't see there that when a computer fails how badly the distributed transaction causes other computers waiting for it to get locked up so if you have a highly available component underneath a classic distributed two-phase committed it can do okay right but if you have just your garden-variety server things are fine until they're not and so that's where there has been pressure under that and I wrote a little bit about that in 2007 and a paper called life beyond distributed transactions I mean they're very interesting things I'd love talking and there's some really cool and interesting technology spanners interesting there's a bunch of stuff that's interesting but on a whole it's it's you have to ask yourself are the trade-offs are did that make sense yeah other yes I mean spanner works great but we're still trying to wait and see how well that works in terms of a readily available to the public protocol I mean they've been using it internally it's it's fascinating it's fascinating Calvin's a fascinating protocol but it's a form of optimistic concurrency control which means you take a bunch of stuff and you say is this gonna be okay without bumping into the edges and you go and you put it against and did it bump into the edges no let's keep those changes and it's really interesting it's really smart really cool but optimistic concurrency control works pretty well when you don't have highly contended data and that but that's been true in a classic database when you have a into your database so there's a the discussions are broad and long and I think we're gonna continue to see an exploration in the space it's a fun space but it's also but I include in that space how do applications deal with maybe this store doesn't do it for them so that's a broad space that includes both stores in application behavior okay so I'm wondering like with the application your system does like the ability with like so you say like if one server doesn't correct answer that the immutable version of that transaction bill call another server but helping it like know about like one service down so so so me I understand the question okay when I say immutable store what I mean is you never update a particular identity so I have an identity for this document it's got a really long URL or whatever and there's never the old version there's never the new version there's only the one version so now imagine if I have a store which doesn't have to do such correct stuff that you do when I'm updating pause for a sec let me go to the updatable linearizable store so i've got a key value store I'm updating value 1 - value 2 I have to work very very hard to make sure there are no value ones out there so if one of my servers that has value one in its belly is down I have to now reconfigure the system to never consider value 1 and never consider that server and so that's hard and it causes me to take some time scratch in my head to make sure that nobody is going to go back and get value 1 out of that down server in the nonlinear izybelle one I would be willing to give you an old version if I was storing version 1 and version 2 if I'm storing it like that sometimes they'll give you a version 1 most the time I'll give you a version 2 but if one's down and it comes back up the server's down it comes back up oh here's the old one what the hell so if I do that but i use it with stuff that doesn't have multiple versions it kind of works that's the observation it it kind of magically works because there's no old value did that make sense so yeah that does make sense so does that mean that that I guess like if one server is like slow to kind of like right create that computable version of it well if it if one slow then I just go to the next one I have a really quick time out really quick time out so yeah he's taken too long I'm gonna in the next one and the probability of them all being slow is really really really low okay I think it's time to wrap it up okay thank you guys for everything [Applause]