SBTB 2015: Duncan DeVore, CQRS/ES with Scala and Akka Persistence
Recording: SBTB 2015: Duncan DeVore, CQRS/ES with Scala and Akka Persistence
how you doing everybody my name is Duncan I work for type-safe and I'm an engineer i work on monitoring but today we're going to talk about something a little bit different akka persistence and eventual consistency and distributed systems in general and this is a pattern i think which can be very useful and as we build systems and as we reason about really two primary areas an area called identity in an area called state both of which can be challenging in a distributed system so for our outline here we have a lot to cover and i want to go through it relatively quickly because of the time constraints but my belief from that you'll be able to pull some things to at least if you're not familiar with these concepts that they can dig into them a little further to see if they're of use to you um so let's get started one of the challenges that we're facing is the reality of the context in which we design systems has changed in the last 30 years it's become much more larger and view the volume of data that we have to deal with has increased beyond even 10 years ago what we would have ever dreamed of and so forth and it's introduced all kinds of new variables that we have to account for and the result of that fact has led us into a slide which I have shamelessly stolen from one of my co-workers Jonas but the idea is is we had in yesteryear we had the notion of single machines today we have clusters of machines we had single core processors today we have multi-core processors we had expensive Ram now we have cheap RAM and all these technological advancements have come on the scene in the last 10 to 15 years and we need to utilize them to their extent we need to use the entire system expense of this cheap disc I remember when I had my first computer I paid three or four hundred dollars for I think a megabyte of disk space or two megabytes now you can get a terabyte for that price so the change is incredible slow networks fast networks network latency is one of the key things and distributed networks few concurrent users lots of concurrent users just 15 years ago I built a system that had I think five or six thousand users that was considered to be rather large at the time recently I talked to a client that was looking at three to ten million concurrent users so quite an increase small data sets large data sets large data sets are becoming so large that the data is really an exercise of reducing the noise or eliminating the noise so you can get to what is a valuable into the covers I heard once said that people that are effective or people that are successful have one trait which is very important and that is that they're able to accept criticism and they take criticism and they look at it as a gift and they look at it as a a jewel encased inside of something like coal and they crack away the things that are useless and they mine out of that the gift which is in which is what the criticism was intent for so it's the same thing with large data the noise is so great you have to mine out that the true meaning from all the excess noise and latency latency in seconds is now milliseconds you have the notion of real time and near real time data and so my underlying message implicit message if you wish is that this is much deeper than we might think it's beyond just coding it's it's it touches to our everyday lives if you will MIT Sloan business school management and capgemini did a review and 2013 and they basically said that companies that do not embrace this type of change and I don't mean event sourcing particularly but the change and how distributed applications are making a huge impact face what they call digital imperative and if they don't if they don't brace this than they can face technical obsolescence or competitive obsolescence and I think the canonical example is netflix versus blockbuster right a company that started out that was the king of video and then is no longer in existence because they waited too long and unfortunately that has an impact on us right the people who give us the money that we use the feed ourselves and pay our rents and mortgages and all these types of things that trickle down type concept will eventually come and affect us as well so it's an important thing a very important thing that we have to consider that the work that we're doing in the way that we reason about building applications and solving problems has a much deeper impact than we may realize and this quote out of our book basically the demand for these types of solutions underlying being distributed as exploded and it's in response to customers right they want immediate response no failure access anywhere the whole internet of things if you're familiar with it which I'm sure you are and I'm I'm in the camp of believing that one of the primary and main ways to solve this problem is distributed computing in fact I go so far saying only but I think it's extremely important so it types safe we have the reactive view of things it's the reactive manifest manifesto if you will you see at the bottom message driven and as an arrow that points up and then two arrows that kind of circle up it's kind of like the base of which two pillars are resilience and elasticity and agreeing with Jonas resilience is foundational to the support of this entire thing as well and then ultimately leading to the goal which is to be responsive in light of failure in light of increased load i am going to respond to you if i'm speaking to somebody and i asked them to do something for me and they do not respond to me I don't know what they're doing I don't know if they're dead I don't know if they're alive I don't know if they heard what I said but if they say hey I can't do that right now but I can do it in five minutes I'm happy they responded to me so the customer wants responsiveness and they expect you to be fault tolerant or resilient and be able to handle increased load so again kind of a strong statement but i believe that modern applications need to embrace this type of thinking incorporating this type of behavior into their DNA right to really kind of think outside the box and not just learn it but to kind of live it you can use the term religion if you want however you would like to define it but I believe it's very important and one of the tools in achieving this I believe is very useful with something called event sourcing so what is event sourcing historically I'd say the last 20 25 plus years the majority of business and I say that specifically business applications have relied on storing something known as current state right what is the situation now that's what I care about and in order to process this you process transactions and then if you want to know things that had happened but we're not recorded you have to implement some kind of audit logging perfect example was hibernate or hibernating bears or you can start to track things that you do instead of hibernate but the point of it is is that you require extra coding to do this it's not implicit inside of your model and how you process or reason about your environment so current state only these types of models have this has not always been the case I believe it's a side effect of our TV or DBMS systems interestingly enough internally they do not do that high performance systems mission-critical systems they do not store data in this fashion relationally if you will I come from an energy industry we use SCADA systems they're the skate and historian as event source based and actually tracks just deltas so if it has two or three readings that are identical it doesn't actually persist that the implied model is that nothing has changed then you can use the value that was recorded and last known and file control audit systems etc back on the mainframe with v Sam datasets and such keeping track of inserts and changes to files or track this way too so the notion of event sourcing is actually quite old so it provides a means by which I would say we can capture the real behavior right you're capturing behavior historical behavior and from that you can drive in 10 whereas if you're only capturing current state you only know what the current intent is you don't know what led up to it you don't know the history if you will what's the saying those who do not know history are doomed to repeat it repeat themselves so event sourcing is a way of achieving this it's behavioral in nature in the sense that you're capturing events to whatever granularity that your business defines them and it can be used in conjunction with the notion of commands which we'll talk about but essentially a command is something that comes in that can be converted into one or more events and current state is never persisted it's just an append only sequence of events current state is derived the nice thing about this is it gives you a very simple key structure you don't need compound keys you don't need complex keys and as a result it's much easier to distribute it additionally it can simplify the the idea of trying to synchronize your data model with your business domain right we try to do that a lot where we have a database and we want the table structure to look just like the way we model our domain and so forth and event sourcing that's not the case gives you a much simpler approach so commands what are commands everybody knows where the command anybody who has had parents knows what a command is right I issue them to my children and sometimes they abide and other times they don't but it's to give an authoritative order it's about behavior I say to my daughter clean your room right that's the command clean your room or in this case register client kline locale has changed and the idea is we have this verb noun form at the verb comes for first and then the noun and it's essentially a request to do a task or an action they're imperative and as such they can be rejected their request and mutate the state your room is currently not clean I want you to mutate the state of your room to a clean state and they transfer in the form of messages rather than data transfer objects as we think of one could argue they're similar but another thing that they imply is a task base you I because you have this notion of eventual consistency which we'll talk about shortly as I had mentioned that can be rejected they're not to be looked at as editing data or data edits there really a request or a behavioral request to change data and based on system design they have a tendency not to expose the internal state of your system whereas with a crud based model that can be an issue not always what it can be they can greatly simplify your repository layer and as I had mentioned before you can take a command and it can be split into multiple events so what's an event an event is a thing that happens especially one of importance more importantly it's something that has happened it's past tense and as such they're indicative and they serve as an indication that something has occurred they are immutable until we can event a time machine they're going to stay immutable and they cannot be rejected when they are processed and so they follow a noun verb comicon context so in the previous example we had register client and client locale or change client locale this is client registered or client locale changed or in the event of my daughter it would be she'll come out and say cleaned room no not actually but that's the idea their atomic right so if you think about crud create read update and delete it's kind of an interesting concept it could be looked at as create and created right eat in red I guess but the idea is is the more atomic you make your events the more intent or behavior you're capturing and who decides that the architect he has an input or she has an input but really the business has an input to its what's important for you to capture so verb noun implies behavior they're immutable the beautiful beautiful thing about events and event sourcing is they provide a natural verifiable audit log and in the energy industry security is very important and everything must be audited so it's really nice when you walk out the door with the system that is a natural audit log without having to implement anything that's the way your system was designed as as such they also cannot be rejected so the canonical model this is probably the easiest way to understand it is the checkbook an account register you can see here a bank account register that has some debits and credits ATM check withdrawals etc and you'll see Delta's there under the change column and then to the right for an optimization we have the balance that's calculated with the last known apply Delta and from an event source point of view you would have your events store and each one of those statements there would be an event that would be appended to the store if you will now imagine a scenario if your bank didn't model they're there register this way if they only stored current state but you however you know you reconciled your bank account you see an error you say you have ten thousand dollars they say you have five thousand dollars so you go in and you ask them a question and they say well mr. Duncan our account says you have five grand and I say well no here's my register you see I recorded everything and they say we're sorry we only record current state so the last update was to decrease your balance to five thousand dollars we have no record of anything you probably made a false entry into your register I mean that's ludicrous obviously that would never work so why is it only positioned for registries things of importance why does it have to be that way what doesn't it can be used pretty much for any model this is just a rehash of what I said but essentially you're going to persist each transaction is an independent event you're going to calculate the balance or the Delta of the current transaction and you're going to apply it to derive your balance it gives you a verifiable audit log that can be reconciliate it for reconciled for validity and can be replayed and you can derive current state at any point in the timeline that has an interesting impact on analyzing intent by observing behavior over a between time boundaries as I had mentioned the last point there and I and I believe that this allows you to derive or to study or to learn the real behavior of how your system is being used so a capper sistance akka persistence is a library that is part of a key it provides the notion of something called a persistent actor or a persistent actor model which is a trait that you apply to your structure and it will persist into internal state but it does more than just that it also manages the you'll see the last bullet their identity or identity and state so what is identity identity is who you are or what you are or what some domain object represents where state is the sequence of events that got the identity to the to its current position to where it is now so a cutpurse istence can provide both of those mechanisms for you and it depends on to what at what point that you would like to take it and we'll show some example or I'll share some examples here shortly so the persistent actor is a stateful actor its thread-safe just like all other actors you have to work real hard to make it on thread-safe ie by exposing or leaking mutable state it supports something known as command sourcing as well as event sourcing and I'll just talk briefly about command sourcing command sourcing is the notion of essentially persisting everything that comes in without validating it this can be useful in transient debugging right but there's a cost right you're the number of messages will call them that you're persisting is increased and you're also persisting errors in your system when you replay in theory your system accounts for those errors so your system is not broken and it does allow you to track and track down errors by replaying history to that point of the air and then you can see it so it is a toll I generally do not use that I prefer to go a straighter than sourcing where I'm going to validate my commands and reject them if they're unacceptable but you can do both and a lot of times command sourcing is used for super high volume where you're just going to persist everything Bam Bam Bam Bam and deal with it later in the model implicitly optimization for replay is built in and you can recover from both your journal which is the event store itself or snapshots so first the journal it's an append only storage model it controls the the the application controls which messages and or events get journaled that's up to the programmer it's pluggable there's several journals out there in the community plugins and the default journals leveldb which writes to the local file system replicated ones can be found at the community plugins page we also have snapshots which again or primarily an optimization you know what a snapshot is right it's an optimization I don't want to replay my entire history I'm going to replay knit to the last known snapshot I have the entire history if I need it perhaps for some maybe maybe a big data grooming or what have you but I can replay back to the last known snapshot excuse me and recreate my internal state from that point and then replay the events from their own there's a lot of journals some journals give you error if the last known snapshot is corrupt for some reason or roll back to the next node snapshot what have you so there's optimizations depending on the journal but the snapshot implementation is also pluggable the default one again uses level DV and rights to the local file system so in a capper sistance the first thing we have here is the notion of something called receive command and traditional events sourcing you have a command handler and you have an event handler and akka they're kind of merged together with this partial function here and you have two of them first the received block which is going to at first if you so desire to validate your command in this example it's going to fold over this function validate registration and it's going to create a list of non breaking error validations it's going to return them as a single group and then if it's a failure it gives it back to the sender otherwise it's going to call persist in this case that synchronous or persist async is the other option and then we're going to side effect with inside that partial function one key thing to note is that if you do use persist async akka will immediately process the next event so any side affecting that occurs within that block you have to account for in your state management aspect because it's going to process super fast so and I'll show you what I mean here I think on the next slide yes our identity and state so in here we have an internal mechanism for capturing state uh-oh we have the evil VAR right but this has been closed with inside the actor so it's thread safe and our state in and of itself is a case class which is immutable so we're doing defensive copying essentially so what we have here is kind of like an in-memory cache of current state we don't have to ask we don't have to hit disk if we want to know what the current state of now is and this is not i'm not talking about seek urs today but essentially it gives you a really nice thing to have here and then the recover is when you bootstrap the actor it will then spin back up and it will replay the events or replay the snapshots to bring it back to its current state so this gives you resilience if you're after crashes the system will spin it up again and it will recover where it was you don't lose your data a very powerful mechanism to reason about so cluster sharding jumps into this identity is the notion as i described i have identity I am Duncan right and as a result or I am a human being if I have a big system if my house is too small and I need to get multiple houses right for multiple people to live in a community and that's the idea here we can take that identity and we can distribute it across several nodes where one machine is not enough it's resilient it's elastic it will activate and passivate messages but the key the key thing the powerful thing here is i don't need to know which house the person is in i have something called a shard and a shard coordinator in a short region that i can simply ask person shard where is Duncan I don't have to have the path of the actor I don't need any of that and it will manage all that for me and surprisingly enough it uses I could persistence under the covers so a shard region is on each node it's an entry point which ever knowed you come into when you ask your question the shard region will either know where the actor is or last the central coordinator for the first time only and then it knows the path to traverse if the actor does not exist it will create it if the actor in other words if it has passivated or has never been created and there is a mechanism that you can apply you don't want actors hanging out in memory if they're not being used so they'll passivate and go away but you don't lose your state as we were just showing earlier so again it's just explaining the shard coordinator and just keeping track at the time here the first message comes in and gets resolved there and from that point on it goes to the target destination so this is just an example of how you set up sharding with a cluster sharding there are some things that you need you need an extractor and a resolver etc but it's pretty simple and you can kind of think of it as a proxy so when I send the command which is change client name i'm going to shen to send it to the the region and the region is going to extract that first value one two three the ID out and it will determined through the shard resolver where to send that message to the correct actor so this brings us to one of the challenges behind this is consistency conformity in an application of something typically necessary for the sake of logic and accuracy right we want to be consistent across our distributed environment in a monolithic system they are trivially eventually consistent in that there is still eventual consistency even though it might be at the micro second level but we take it for granted because that we have a tightly coupled system and we a lot of times use a centralized database yeah okay let me bounce through this can I get two minutes okay so strong consistency is an important part of that the other aspect of it is and distributed systems you lose that and if you're familiar with cap theorem you have the three pieces of a consistency availability and partition tolerance you can't have all three at the same time and that's this diagram here where the center na is not achievable and distributed computing so you have to deal with that um the reality of it it comes down to eventual consistency and the idea is that you're going to make the different nodes in your system eventually consistent as long as there's not a right in between that eventually things will be consistent and that's the way we live our lives we're eventually consistent eventually the message gets there and we all see the same thing so I'm going to stop there only because there's some other things I'll be around for some questions if you have any and we can do them offline because I want to give the next person time to get up here but I appreciate you guys coming out and I hope that if you're not familiar with this that at least it'll give you something new to take a look at and if you are perhaps you can you've learned something or maybe you can come up and teach me something so I appreciate it thank you very much for your time you