Scale By The Bay 2018: Ryan Knight, Data Consistency Patterns in Cloud Native Applications
you thank you welcome everyone thank you for coming out here here's hurt it hurt a funny you guys a couple people heard funny okay why is just a small part of this I wanted to talk about daily consistency in general and kind of look at like where we're at today with the different data consistency patterns as you know more and more enterprises transition to cloud native micro services architectures the challenges of data consistency become greater and greater because you have to deal with data orchestration and between the different services and maintaining consistency within an individual service the clicker works you know clicker doesn't work right transition doesn't work so first I'm going to talk a little bit about what what I mean by de consistency how I think about dating consistency some of the challenges of achieving data consistency and then what are some of the main patterns for consistency why does the transition network somehow I broke the trends via the transition anyways so when we talk of consistency you know a lot of people just think of you know is to my database eventually consistent or on my strongly consistent and really there's a whole variety of a whole different models or consistency that we need to think about and this is because there's a lot of different problems with the consistency of my data so if we think about all the problems that we have with consistency it's a lot deeper than just you know dealing with eventually consistent data the challenges are with things like reading your own rights if a driver writes a piece of data can immediately read it back ok read uncommitted data can I get dirty reads read skew non-repeatable reads in other words if you know client a and client B both read a piece of data are they going to see the exact same piece of data the exact same can't I ever so there gonna be some skew of that data am I going to lose an update if I have to concurring writers that write in parallel through my database exactly the same time could one of them overwrite the data you know could Ryder a overwrite the date of Ryder be is there any rights to you right skew is actually the hardest problem to understand and to fix in my database right skewers basically when I have two parallel operations that are both read a piece of data from the database in parallel and they both make a decision or what - right but they write different things and they end up skewing the rights based on this these inconsistent views they end up getting so the way we can think about consistency there's Peter Bailey's really good paper about this a Kyle Kingsburg put into a really nice graph we can look at the different consistency models and see the relationship between them in this graph at the lowest level is we have the lowest level of consistency such as you know the ability on the this left hand side of reading uncommitted data all the way up to you know serialize ability what's or whoa this is talking about on the left hand side is really your transaction isolation though it layers on the right hand side is the from linearize ability down is daily with a single register single partition consistency so this is a basically for a single session when I write into that that particular register do I see you know have an atomic operation so that starts with like monotonic reads is that read you know consistently you know read eighth and Rebbie does that always happen in the exact same order when I write when I write a write B does that right happen in the exact same order so these all build on each other to build up to the strongest levels of consistency so there's a lot of different ways of dealing with a consistency in the application tier there's a lot of different models I'm gonna just go kind of highlight a couple of them one spend a lot of time in detail on all of these distributed sagas are really common how many people have heard of distributed sagas anyone heard of them all right there popular model out there today for dealing with coordination of micro-services transactions aqua cluster is another pattern and so we're gonna go through all these really quickly and I think it's useful to when you look at all these different patterns that we have a way of evaluating them and the way we can do this is using kind of consistencies or lands to evaluate our different approaches and we can evaluate the trade-offs between these different approaches and we can also use consistency to understand what kind of problems we're gonna run into by using these patterns and one of the one of the things I really like to think about when I'm evaluating these different consistency patterns is how much complexity does this put on the developer so there's no one perfect solution there's a lot of trade-offs of plus and minuses all volleys the best way you know everyone talks about the cap theorem I think the best way for me that I've heard is actually Daniel body talks about really it's more than just cap it's this in the absence of a network partition I want to think more about the trade-off between latency and consistency so do I you know I was just talking to Pat healing and he said do you want the Gator right or do you want it right now so do I want exactly consistent view of the Gator am i okay to wait just a little bit to get it right and really when you look at these I mean there's a lot of different patterns and you know some you know they're all dealing with the transactional complexity of consistency it's just where that's happening so first really quickly look at patterns here ap with seeking connections in other words if my client always goes to the exact same session it's going to have it be a much easier to achieve consistency because I'm always going to the exact same service for that connection so this provides you know high availability and partition tolerance this provides you know if we look out in the the consistency models this is basically everything under the cause of consistency so peer an writes follow reads all the time I can read my own writes that type of thing Wow so aqua clustering is one example of this with aqua clustering I could basically pin a session to an actor that actors always going to process that session so I can guarantee that that session is always going to be able to read its own rights I can guarantee you know monotonic reads monotonic writes because that same actress always processing it I can use aqua clustering to scale that and to guarantee that you know provide some level of guarantees that it's nicely charted across the cluster akka has a lot of nice patterns for persisting that actors so I can provide high availability what is that's basically you do is that's going to persist my actor state to a database and allow you know if that actor dies that I can rehydrate the actor on a different node and on top of all these models there's the akka distributed data which provides a unique data model with CRD keys these are merged these are data types that allow you to do things like grow on the counters they provide like monotonic merge functions so that I have a round on pattern for emerging conflicts so like a grow in the counter if I add you know four or five and six to my counter doesn't matter what order those operations happen in so there's a lot of strength sock to clustering it's you know extremely high availability it has high throughput low latency because it's all in memory and it can be partitioned tolerant if I use a split plane resolver there's a number split rating resolvers that are out there there's a number of weaknesses with akka clustering it has a limited data size I think one of the biggest challenges you know when we look at consistency models is that there is no transaction isolation so there's no isolation in my data and then the consistency level really is dependent upon the backing store of that of what I'm using for akka persistence for my backing store just gonna skip down to AP consistency so this is everything in the blue you know this is sacrificing a high level of consistency for availability and partition tolerance the idea is that I wanted to maintain availability even if the network goes down entirely this provides you know a fairly high level of consistency with this you know monotonic atomic views of my data I mean can provide protection for we you know we committed uncommitted data so one example of this is distribu sagas they're in the family of AP consistency they're not entirely ap consistent distributed sagas basically have a central coordinator that manages the complexity of that transaction logic and that central coordinator basically tells you know one or more services to complete you know the transaction it coordinates those transactions so an example would be likening a travel booking the distributed saga coordinator is going to tell the you know car booking the hotel booking the airline booking to go and book a transaction and then those services will either complete or rollback and depending on the state of each of the services that the coordinator then describes the transaction as either a success or failure and they'll tell the other services to continue know to succeed or fail as well distribute sagas rely on my individual workers to be idempotent in other words I can replay him I can make that car booking and multiple times in a row and not end up with multiple reservations on the underlying workers also have to be able to compensate for failures so they have to have do they know how to roll back in the face of it transactional failure so this gives me a model that's effectively wants not exactly once because these things don't happen exactly once I might actually do a transaction multiple times before I actually get the entire transaction completed and so this has a number of you know this has a number of upsides it's very fault tolerant because I'm using a backing log to store all the events of that destroyed saga I do have you know a very fault tolerant highly available solution the executors for the underlying coordinator are composable and I can come up with new workflows to tie those together and I'm isolating all the complex code into a that central coordinator this for the central coordinator provides atomicity for the trans overall transaction and it allows that transaction to be a board to enrolled back the downside to this is there's a lie-down sites to distribute sagas they provide very weak isolation and you get into this really weird state where it's you know if your coordinator dies in the middle of a transaction and before they actually it finishes rehydrating you can get some unsafe states the versioning of that transit that transaction is fairly complex in terms of you know if I want to update the workflow and have like a new version of the workflow my backing log will have to change and how a version that there's a lot of complexity in do during the rollback and recovery logic and idempotency is impossible for a lot of services so a lot of services just can't fit into this distributive saga model so the the other area I want to talk about is the area with strongest consistency that CP consistency with global scale transactional databases this is the strongest level of consistency in the consistency models so if I look at so what this provides me is serializability right so serializability the difference between it serialize ability and linearize ability is you know serializability is talking about coordinating transactions across multiple tables or registers whereas linearize ability is a single register or single table so this level what we want to talk about is how do we provide serializability across multiple tables and the databases in this pattern provide true asset transactions so I have atomicity to allow for rollback of transactions this also provides complete isolation of the transactions so that's really what's talking about on this left-hand side like being able to you know preventing reading uncommitted data so I'm isolating the transactions from another transaction that might be happening in parallel and really when we talk about these transaction levels we're also talking about safety guarantees what levels safety am i providing in my database so what's the primary disadvantage upfront of these and Apollo said that bestest of transactions are hardest to do transactions they're harder distributed transactions over Wang LAN network or the final boss harness this level of databases are some of the most complex to deal with and build so you know he gave this warning on Twitter just recently that you know he's seen a lot of trends in the industry of a lot of people trying to build their own and distribute it you know transactional databases over LAN and this worrying that this is an extremely hard problem so there's there's there's a lot of databases in this space and it can get confusing to try to understand what is the difference between all these different databases and what they provide to the application level when you look at databases in general most databases provide what's called a multi-value concurrency snapshot in other words I snapshot my date at a single point of time and then I operated on that snapshot and so I can have you know concurrent operations that are happening on different snapshots the primary differences between these databases is in the transaction protocol is that global ordering done in a single phase versus a multi-phase do i pre do a pre commit or post commit transaction resolution and you know what are the levels of consistency provided what's the maximum scope of that transaction is it a single record or multiple records and this all ties into you know as I evaluate these databases and how you know what is the pattern that I'm going to end up using in my application tier because the application tier has to be aware that when I do my operations if I only have a lower level of consistency if I only operate on a single partition that I can't I have to then provide you know add in my applications here if I only have transactions this is a single partition the application tier is gonna have to provide consistency across multiple partitions so it pushes some of the complexity back up into the applications here the other complication is everyone says they're asking plants or what does a secured clients means it means that I have you know the this you know the asking plants to be anywhere on this asset spectrum so you know on the very left side I have you know very weak isolation levels my scope of my transactions a single row I really am in eventually consistent all the way over to the right hand side where I have a very strong isolation levels my the transaction is distributed across multiple partitions and I've serialize what consistency so probably the the the other thing really understand between these different databases is how they apply the rights the most common approach to applying my rights is in a non deterministic fashion so this is traditional like two-phase commits really I'm going to roll the dice on when the effects get applied roll the dice under what's going to happen and one which effects get applied because what happens is the coordinator is going to walk across the cluster and it's going to determine the effects of that connect contention and then it's going to on the second phase it's going to tell the partitions if they succeed or fail and so what actually ends up happening is very non-deterministic in the end of the day and it requires two rounds of global consensus the alternative to this is a deterministic database so one example of this is from fauna fun eases the Calvin protocol the Calvin protocol does deterministic commits and the way that does this is that when it receives a query into the database it takes a query in a non determine this is a query and it turns it into a deterministic transaction so I'm determining exactly what all the effects that transactions are in the first phase I mean I essentially turn that transaction into this kind of pure function that I'm gonna operate over my state and then I'm going to get a total global ordering of that log and commit that log and kind of the ordinary face that's the only round of consensus that have to do across the cluster and in the final state I have a deterministic results that get applied to my database so the other that's on the right side if we look at the read side the trade offs on the reading side are do i do my rican Sisseton see with clocks so this is like spanner spanner inspired right this is a big trend today as you know spanner in our databases the problem they're trying to solve is how can I do consistent reads with concurrent writes happen at the same time so I want to be able to read if I try to read using two-phase commit I'm gonna get a very high latency so I want my rates of fate of where the two-phase commit so the alternative is to choose a stable snapshot time and how do I figure out what snapshot I read the database to add and that's really a decision of how stale do I want that data to be and by having atomic clocks I can guarantee a very high level of jaren teeth at the marina the exact right time it's the alternative to the spanner protocol spam writes consistency without clocks so I can do a read only transaction that avoids the transaction pipeline and I can do basically it uses a log base total ordering to depart it's a guarantee the stability of my treats so because the calven protocol is doing a global ordering of all the transactions across the cluster I can know that exactly at a particular point of time that I can guarantee a stable read the challenges is that snapshot of when I read that data might be slightly the future from wind node has actually seen the data so I sometimes I have to wait until that node has caught up on all the transactions and they've been applied put in as soon as that logical partition has all the data it can provide a read about executive and snapshot without having to go through the pipeline so the looking at the pros and cons of this one Google spanner provides you know it provides a very high level consistency it has relational integrity constraints they providing your external consistency which is the highest isolation level they provide five nines of uptime it's really a phenomenal database the challenges is it's very expensive it's very proprietary it ties you into the Google cloud it does have the nice feature that when I do a write to a single shard I can avoid the two-phase commit process so the alternative to that is a fana DB so if I look at the database to date like spanner gigabyte cockroach all these are using some kind of atomic clocks hybrid clocks fauna is instead based upon the Calver protocol so it basically does a global transaction ordering with the log font is both in the cloud and on-premise temporality multi-tenancy has a roll level security built in from the core it has a document model for the schema modeling and the way that I query Vaughan is through indexes so each where we call a class or basically table it can have one or more indexes and that those indexes are like a materialized view into my data and that allows for different query patterns so the phonic consensus algorithm provides this total ordering so to give me a level of serializable consistency and that distribut log that's going to provide the total ordering of the transactions scales with by being able to partition to log the nice thing about font i write is the single phase consensus algorithm provides all those you know possible global latency because I'm not doing two-phase commit on during a single phase global consensus algorithm I have a very much lower latency I can do low latency snapshot reason to void the transaction pipeline and there's really no difference between multi partition and single partition transactions now final provides a very powerful query language so all you know you can have a complex transaction in a single query the the downsides to fauna are it is a proprietary query language you know so a lot of people want you know any great fun into their legacy applications and that's typically not the right approach because the proprietary query language is really designed to have you rethink how I'm building my tree my application and building a new type of you know trying to a transactional model in my application so this provides a much higher the other downside is you know like other distributed databases like spanner and cockroach because I do have global transactions I do pay a higher price for my latency and probably the primary difference between fauna and spanner is that fauna doesn't you always have to probably pay the cost of a multi partition transaction rights spanner can actually avoid two-phase commit with a single shard partition so they have a lot of tricks they can play with that because fani uses a global log for its total ordering I'm always paying the cost for multiple partition transactions and then the final thing is temper ality all right this is one of the really a very interesting unique to the aspect of fauna is that it maintains the evolution of data over time and so I can take basically look at any snapshot in time and understand where that's had what they look like that particular point of time this is very useful for being able to like do something like give me all the reads since this particular point in time so I wanted to do a quick demo I'm gonna put this down I just had to do thank you all right so I have fauna training here at the background I'm gonna run a process this is the northwinds data set from 20 years ago customers and orders this is going to load my data set into mode so Fonda all my customers and orders Oh wrong screen thank you sorry guys you know can I really quickly turn on Mary we're crawl so that's loaded the database now I'm just going to show some example of queries against that database so if fauna has a shell that allows me to connect into that database sorry say final shell northwest's puts me into command prompt where i can operate queries against that the northwinds data set this is using a I built a little workshop it's out on my github github retro Ryan on fauna so I can use this to query the database so I've loaded all the customers so for example I have an index called orders by customer ID so if I look at that index the index is the source that indexes the class order classes are tables and fonta terms are the fields I want to search on so I want a search on customer ID and the values I want to search on are the values that are going to be returned out of that index like I say it's a materialized view so I'm providing the values that are actually stored in the database so in this case I'm only going to be turning order date and order ID I could do something like return a ref to the extra class to pull back to full instance if I wanted to so I can execute this map query so I'll break this query down approximately so this query what I'm going to do is I'm going to do a match against the indexer orders by customer ID I'm going to look for the customer that with the customer ID loan P the loan P and so once I get that index back that returns back a set of data and then I want to paginate over got set of data and then it uses a a very functional like this construct I basically need a map over that side of data and do an operation on that set of data and the operation I'm going to do is this lambda operation so what I'm saying is for each item in my list I want to do get out the order and then basically just return it so if I run this this is just going to provide a total list of orders all right so I'm map you know got set of data and I'm really just getting the data back and pulling it back I can also do like get index on all orders right so this is going to get me all of the orders out of the database so I can this is going to give me a total ordering of all the orders in the database and then I can actually do operations on that order an interest of time I'm just going to skip a little bit but essentially what I can do is I can do a what's called a let operation a let is going to take the data set pull it back other databases assign it to a variable and then I'm going to do an operation on that variable and such as getting the last index out of that order details right so this did what this is saying is get the last index out of the database select the zero zero which is basically I'm selecting into the row and giving a particular value out of that row so I got out the value of xi xi 0 0 7 7 so that's the last order ID and so using that order ID I can then create a new order that uses that last index so in a single of this single query that query is going to be a single transaction this transaction I'm going to pull that ID back so basically I'm reading from the database pulling that idea in and then using that ID to create a new order so if you see down in here and taking the last index if a scientific variable and I'm saying give me the variable last index add one to that index and then store that as order ID and so that creates a new order in the database and that's an entire transaction so it's doing you know it's reading the database performing some operations on it and then writing it back down into the database and it does that all in transaction so cool to time for like one or two questions really quickly crawl any questions [Music] I do you know Matt what huh atomic DB do you know it uses O'Day Tomic the atomic do you no matter what kind of transaction protocol the atomic is I don't know either I don't know what kind of transacting protocol the atomic uses sir yeah go ahead the DSL that you're using it's kind of like it's like a functional closure so it has some similarities to closure but it's it's somewhat of a functional language so I don't know Matt wouldn't you would you boys your inspiration for Matt might be able to that's the CTO father I don't know what was your inspiration for the font of query language matter [Music] cool that's it hey thank you everyone for coming [Applause]