Chris Richardson, Functional Domain Models with Event Sourcing, SF Scala @Sungevity 201502
you so I'm going to give a talk on functional domain model but kind of work from a event centric perspective and talk about an architectural pattern known as event sourcing so a little bit about me so we are way back in the 80s programmed in les building Lisp systems compilers runtimes garbage collectors and that kind of thing eventually ended up in the Java world wrote the book pojos in action which is all about spring and hibernate which are two favorite topics at a scholar meetup did a cloud computing startup whoa cooled cloud foundry OOP seem to have jumped ahead not very good with a microphone hey so did a cloud cloud computing startup called cloud foundry which then got acquired by spring source right before spring source was acquired by VMware and now these days doing some consulting and I'm actually working on another startup and if you want to know more about what I'm talking about there's a whole bunch of links so got some example code in my github repo as well also i created the website microservices dot io which is the beginnings of a pattern language for describing microservices and right now there's only three patterns there but i actually have five more to go and sort of draft form that i plan to publish in the next week and follow me on twitter so the kind of the structure of the torque is you know why event sourcing what is it why should we care and then look at a domain model really simple domain model that was built using an event sourced approach so it's funny for the longest time I mean Pratt you know for probably like eight plus years I've always used this money transfer example the idea is like you know you go to your bank website and you want to transfer money between to a bank accounts you pick the from account the to account and the amount and press a button and the money gets transferred and for the past eight years i always apologize because i've never worked for a bank and i have no idea how it works in practice but this is my my book so this is actually a version from a slide in 2007 so you got a service there's a domain model you know traditional 00 kind of domain model where state and behavior wrapped up inside a bunch of domain objects you know if you to look at the code you know so it's sort of the standard mutable domain model so you got the account class and so on all of the entities are persisted to the database using hibernate persisted to a relational database so it's very traditional the service that's coordinating the transfer between the two accounts is made transactional with spring so pretty standard stuff now what's interesting is so that was from 2007 and you could say that was like how you built apps back then but if we fast forward a few years the way we build apps has changed quite a bit so I really like this book the art of scalability it's all about building scalable engineering team scalable architectures and so on and in the book there's a three-dimensional model of scalability where you've got x-axis scaling this horizontal duplication you've got said axis scaling that's all about partitioning database sharding and so on and then there's y-axis scaling or functional decomposition which these days goes by the term microservices you know which is very fashionable so we don't really build monolithic apps talking the talk to a monolithic database anymore you know things are quite different so you can imagine if you apply y-axis splits or functional decomposition to the application it's now a collection of services each with its own database and then that you know the database is split up as well it's functionally decomposed you might have accounts in one table money transfers in another and then we also could actually applies that access splits or partitioning or sharding and so a the two accounts in a transfer could actually reside in two separate databases based based on their primary key so you know three entities potentially three different databases yet we need to sort of have some kind of business transaction that you know maintains consistency between all of these different databases and yet we can't use two-phase commit any more sort of gone out of fashion and it doesn't work because of the cap theorem and so on and even if we were to pick a more mundane example of like well I'm storing product information in my sequel and then I'm using elastic search to do text search how do i maintain consistency between those two databases right kind of in real time and so on and unique once again you can't do two phase commit so you sort of have all of these problems with multiple databases and maintaining consistency between them and a good good solution to good way of solving that problem is to use an event-driven architecture where when a component or service changes its state it publishes an event and then consumers can subscribe to those events and update their own state so the components say that updates my sequel can publish an event saying the product has changed and then the indexing component can update elasticsearch but interestingly that actually creates another problem how do we atomically update state and publish events without using two-phase commit so we sort of kind of moves the problem along and that that sort of leads you to event sourcing which is really an event centric way of designing your business logic and designing your system and the big idea is for each aggregate for each key entity in your system you identify the state changing events that the domain events and you actually define an explicit event classes so for an account it would be a count opened account debited account credited and so on so we're kind of just treat domain events is first-class citizen in our domain model but you know that's just the beginning so sort of the big idea where the event sourcing is that we no longer just persist the current state of an entity you know you could imagine where we have an account table that's got a balance column we don't do that anymore and instead what we do is store the sequence of state changing events for each entity so for an account it would be it was opened with a certain balance then it was credited by a certain amount little debited by a certain amount so you actually store all of the state changes for each of your entities and then if we ever wanted if we actually need the current state we actually just load all of the events and replay them later on you'll see how that's really just doing a left fold over the sequence of events and that will just reconstruct the current state and you know you might be wondering okay so what's the big deal but what what what has changed is that whereas before you had to update state and publish events now or wherever doing is actually just persisting and publishing events and there's no no separate state that has to be sort of part of some transactions so we're just doing one thing and the whole and it's a lot easier to do that in an atomic fashion so that that's kind of why I sort of ended up in this world of event sourcing because it solves all E or a number of problems that arise when you're using micro services when you're using no sequel databases and so on so the way in your system that you know the way requests get handled you know whereas before we sort of load an entity a request comes in we load an entity you modify it and then you save back the updated state in an event source system it's a little different you actually first load the event and then you create in this case an account and then you actually replay all of those events to reconstitute the current state of the account and then you process the command that's derived from the request and that gives you new events which you then persist back in the events store so a loading event generating new events and appending them to the entities events in the event store and that's actually done with optimistic locking so if there had been two concurrent up to concurrent update attempts only one of them would succeed and then when it gets really interesting is off the back end on the other side you can have subscribers that express interest in certain event types and then that when events get saved into the event store the subscribers get notified which can then enable them to update their own entities or update views that they're maintaining so you could up so they could update elasticsearch and so on so it's kind of a totally different event centric way of building your systems and there's various event store implementations out there a lot of folks I'm gonna see them some folks in the room who have ended up building their own event stores right Alexi there's one commercial one out there get event store by Greg young who's been advocating event sourcing and seek urs off in the.net world for a while and I'm actually built built building one myself it sort of grew out of it and the first startup idea sort of then got consumed by the actual event driven platform so I'd love to talk more with folks about that whole bunch of business benefits to to event sourcing you know number one you now have a built-in reliable audit log right so any update to an entity has to be done via an event if there's no event the update didn't happen and that's quite different than the traditional way of doing auditing where you just kind of write your business logic and then you kind of add some auditing code on there which can get out of sync can be forgotten and so is not very reliable another really interesting aspect of it is that it enables temple queries because we've actually stored all of the state changes for an object you can go back in time and ask an object what was your current state five days ago so that can be useful in some domains it also fits in nicely with machine learning because it's generating the stream of events that can be left fed into a machine learning predictive analytics engine which can then sort of help refine the cat pictures that your system shows to to the world and it also preserves the history of everything that's happened in your system so you can actually go back and and when you implement a feature you can actually implement it as in a way as if you as if the head of the word had always been part of your system because you've got all of the past changes to then go and process so you can kind of more easily and sort of implement unanticipated requirements and then there's also a whole bunch of technical benefits right so kind of solves a whole bunch of data consistency problems in a in a in microservice or no sequel architectures and then it also eliminates much of the o.r mapping problem right you know we've sort of all been struggling with how to map a rich domain or domain model to a flat relational schema I'm given that events tend to be simpler in structure it's fairly trivial to serialize them um actually and guess you see the slides a whole lot better in the dark you know there's a bunch of drawbacks right you know it's kind of a weird and unfamiliar way of designing your your domain model takes a bit of getting used to and sometimes events can you know be a sort of hit because events never go away once they've been generated you sort of got this historical record of all of your bad design decisions which can be to be a challenge they're usually events sort of correspond to things from the problem domain so they tend to be fairly fairly static and also with any system you have to message oriented system you have to deal with the possibility of duplicate events which can be a challenge but the system's Ivan what building every event that gets published has monotonically increasing I time-based ID so you can you can easily detect duplicates and then also there's because we've now got eventual consistency your application might actually have to deal with that you know you could actually update an entity and then go looking a view that's that's sort of a denormalized view generated off the events and that because of a replication lag that you might not have been updated yet and so you know you've got your code has to be a little slightly more complex to deal with that and then there's also the limitation that the event store is all about retrieving events and updating events by primary key there's no support for joins which is a whole other talk about an architectural pattern called command query responsibility segregation which is a way of solving that anyway so that's sort of event sourcing and kind of why it matters and what what the benefits are and what some of the drawbacks and I kind of wanted to actually look at some code see any you think about you know building a domain model you know some things have stayed the same so you've still got the familiar building blocks and your domain model is still built out of entities value objects there's a concept of aggregates in fact in this system at the notion of aggregates become a lot more important you've actually break up your domain model into aggregates and each transaction only actually updates one aggregate at a time and then publishes event which can then trigger the update of other aggregates the actual nature of the aggregates change so rather than a mutable kind of classical sort of domain model object-oriented domain model where you've got debit you know methods that mutate the state we now go into sort of a functional immutable event centric approach so instead of a debit method that changes the state of an object we now have a debit command and a debit account debited event and the actual each of each aggregate has two methods process command which takes a command gives you back a sequence of state changing events and then an apply event method that actually takes an event and gives you back a new object the new entity that has been updated to reflect that state change so it's sort of sit the you know there's kind of a one-to-one mapping but it's kind of quite different you know one of the challenges you have is designing domain events right you know the naming they tend you know they sort of have to be in past tense because they're representing things that have happened hopefully in your domain you can have specific names like that mean like debited and credited as opposed to something vague like oh I just updated this object this domain object you know they have attributes that it you know I mentioned there's an ID that's basically a time base uuid plus any attributes from the command that are necessary to then call it to then update the state and then there's also the idea of event enrichment we're not only do you just have the attributes that are necessary to update the state but you also throw in some extra data that is useful to the consumers elsewhere in the system of that event so instead of just when you say you add a product to a shopping cart just having the product ID you'd actually have the product info because the downstream consumers actually need the product info so rather than having requiring them to go back and get it that information is packed the event and there's a kind of a whole bunch of issues around balance like you don't want to make the events too fat so to speak but you don't want to make them too skinny that the consumers have a lot more work to do so you got to kind of have balance so kind of have slightly two different ways of representing domain objects one of them is kind of a hybrid sort of 0 0 plus FP approach where I've got a couple of traits aggregate that has the apply event method that really the returns a partial function from event to updated aggregate and then the command processing trait that has a process command method that returns a partial function from command to a sequence of event so these are actually based rates so the actual aggregates have to implement these traits so here's an example of the account so you know so count implement command processing aggregate blah blah blah and you can see like there's a process command method that's using Scarlett pattern matching to math teach incoming command to a sequence of events you know that and there's a overdraft check in there as you'd expect and then there's the apply event method that's there's a partial function that's doing pattern matching from event and to updated domain object once again because if you know it's a case class and so you can just copy it and modify the attributes that you want their interestingly you know their scope here for lenses if you want to go fancy and you know hardcore if you want some of the other examples i have do that the underlying event store you know has like save update find methods you know with kind of type parameters and generally looks like someone sat on the keyboard you know there's there's another another interesting that you actually do not have to interact with the event store directly so I've got a couple of sort of simple dsl's so here's the account service dsl so here's the account Oh open account method and the dsl just says so go create a new entity of type account and have it processed the open account command and under the covers it's actually instantiating an account you know calling the default constructor processing the command applying the events to update it and then persisting the events in the event store so all of that boiler place hidden just by using this kind of nice little dsl and likewise the code that subscribes to events being generated by other aggregates also uses a DSL as well so it's like handler for this kind of event in this case is updating an existing entity whose ID is this by processing that command and so under the covers it's loading the aggregate from the event store processing the command applying the events and then persisting the new event back into the event store and that's all hidden with the behind the less than equals equals thing so in that you know that i found that that's actually worked quite nice then recently i've been working on a more sort of pure functional approach and that actually has similar kinds of traits so like aggregate which has got apply event and you see apply events is doing a fold over the event stream and then there's the command process with the process command but rather than the actual domain entities or domain aggregates actually implementing these interfaces they actually play the role of they actually function as implicit or kind of like multi parameter tight classes which is actually a concept in high school but not so much in scala so the code ends up being this is a little hard to read but here's the actual domain class the money transferring it's just a straightforward case class with absolutely no behavior just the state and then there's a implicit object a money transfer aggregate that implements those two interfaces that has processed command and apply command and so there's this sort of the standard functional separation of State and behavior which seems to be popular within the functional programming community so oh yeah so yeah then you know here are the events so they all inherit like they'll they will extend the money transfer and here are the commands and so on and then the actual events store is a little the signatures where the method names are the same but the signatures are different because like because it needs to take these and needs to take via the implicit the aggregate that tells it how to actually instantiate an aggregate and how to actually apply event and I sort of find writing stuff writing this kind of code little painful like trying to actually eliminate the sort of how to avoid having to specify all of the type parameters and I found just by passing in the class objects all the titrant for inference think I know figured out what tears and figured out what EV is without you ever actually having to write it out but I sort of playing around with this for a few hours anyway so hopefully that's given you sort of a flavor of cut in a kind of a different way of writing sort of domain models of functional that are both functional but also event centric and you know I like Scala for this right you know case class is this great thing that just eliminate a lot of sort of boilerplate code patent matching is like super useful and you know just the fact that recreating state from a sequence of events is really just a fold you know that that's really quite nice as well so anyway so that that's my talk I had no idea how long I was meant to speak for but that's all i have to say so thank you yeah incoming commands and derived event well say that again the bottleneck I mean if you want to be able to process a set of commands or any big events from that it takes some computation validate that parts of command and the validate doesn't make sense in context and so all that Vincent's done before that's emitted and so it in my experience it tends to result in kind of a backlog of commands because they need yeah so sort of that the question was were was well kind of all of this processing of a command takes time and so can create a backlog and delay when events get published is that sort of roughly it um I I don't know I guess I haven't really seen that problem in a way I view the processing of commands in this architecture to not really be any different from processing requests in a more standard architecture and oh you know of it you know that you I mean the one way you could end up with bottlenecks is if you end up with a whole bunch of concurrent requests for the same entity and then you get a lot of Optimus optimistic locking failures and and retries but apart from that everything you know there's great scope for request level parallelism right the event store is super scalable because it's very shot able you know and then the actual app side of it is also very scalable as well so a concrete example I could sort of make up is he wanted to be able to scale out and where were you what approaches to shark and in that case you're sort of I guess creating different aggregates that maybe don't step on each other's toes but what if a specific shard itself would benefit from scaling out is there not a problem in for example many users wanting to access that is you want to spread it up the machines but if it's if it's all part of the same aggregate still start to run into oh yeah I mean so sort of the comment was war what's the granularity of the aggregate sand and I view it as things like a user account shopping cart product order so so very kind of somewhat fine-grained and yeah if you go read the literature that they will talk about yeah I mean like you know the exit interesting read is implementing domain driven design book the kind of the more practical guide to ddd than the blue book it's sort of the red and black one and you know one of the one of the examples they have in there is like the initial version of some domain model had some giant aggregate that resulted in like massive concurrency problems and it's like oh we just need to need to have finer grain aggregates and the plot and the concurrency issues go away any any other questions yeah if you need to maintain some sort of transactional semantics between different aggregates like I can imagine that if you have two accounts and you want to do a balance transfer that might be a single event that mentions both accounts so how you do that oh yeah so if you need to kind of maintain sort of transactional integrity that's actually like an i have an example of exactly that in another talk which you can see online but briefly right so it's kind of like you create a money transfer object that says need to transfer money from a county door can't be or you know ten dollars that published so that's one transaction that publishes an event then the account is subscribing to the money transfer created event and goes oh well I'm going to go debit this account which generates NIV an account debited event which then triggers the update of the money transfer to say that the debit part happened which publishes another event that triggers the credit which then triggers the update of the money transfer to say that the transaction is complete so there's a lot of you know it's a sort of a multi-step process with events bouncing back and forth that's a very interesting solution what I haven't thought of it does seem a little risky because it suggests that or it allows for transfers that are not that transactions that are not part of the money transfer that is you no longer get sort of a robust requirement that only transfers happen via your money transfer object because any increase in balance is going to be accepted by that account what yeah so you're sort of saying it's sort of less well sort of the the enforcement is looser yeah I mean I I suppose it is I mean the one you know the one thing that troubles me about it is right you can't look at the money transfer workflow because it actually ends up being split between the money transfer class that's that's subscribing to events from that are being published by accounts and then it's being and then there's the accounts that are subscribing to events being published by money transfer so it's kind of like spread it spread around but it seems workable yeah like a transaction oh yeah it's fun funny you should say that because yeah the actual so a transactional money transfer has a transaction it has a has a primary key so that's the transaction ID and then when a debit happens its or credit in an account it's because of that transaction ID so there's so there is this traceability thank you well thanks and you can now find me online somewhere