Developing functional domain models with...

SBTB 2015: Chris Richardson, Developing Functional Domain Models with Event Sourcing

Recording: SBTB 2015: Chris Richardson, Developing Functional Domain Models with Event Sourcing

so I'm so welcome to my talk on event with functional domain models and so the goal of the talk is both to introduce the concept of event sourcing to those who for whom it's new and then just talk about an approach that I've been using where I sort of write domain models using a functional style which you know is the popular thing to do these days so that that's that's the talk a little bit about me so I actually got my start in programming back in the 80s building Lisp systems Lisp of course being one of those early functional and then object-oriented languages kind of a hybrid somewhat analogous to scholar in a way and obviously you know I ended up programming in Java for a number of years back in 2006 I wrote the book pojos in action which was all about how to build apps in spring and hibernate which back then were revolutionising enterprise development and then I started tinkering around with this obscure service known as Amazon Elastic Compute cloud back in 2007 which I think many of you have now heard about and that that that work evolved into cloud foundry which was a pass that then got acquired by spring sauce just before spring sauce was acquired by VMware I ended up spending four and a half years at Spring sauce VMware pivotal doing doing various things and now I do consulting and training around micro services so please contact me if you're looking for help and I'm also the founder of a start-up that's building a platform that's gonna make it easy for application developers to write event-driven micro services and I'm looking for people to kick the tires on the platform so go check out the the signup URL and you can find out more information about me at these lengths so github I've got a couple of different examples of micro service applications I also have micro services ohayo which is a collection of patterns for microservice architectures and my blog and Twitter and so on so please reach out so here's the agenda first kind of provide the motivation for event sourcing and explain what event sourcing is and then dive into the details of designing apps using a functional event sourcing approach alright so let's get started so you know I'm sure everybody's familiar with the typical enterprise app that's just one big monolith right and on the one hand it's not bad it's just simple right to build apps that work this way having a monolithic database is extremely simple gives you nice asset transactions and all of that so it's kind of good right but if you're building an app that's just you know constantly growing sooner or later you're gonna end up in monolithic hell right where you're just gonna have this monstrous app that's difficult to understand deploy maintain test you know all of those those bad things and I think many of us have worked on those million line apps that are just just how to do anything with and of course the right approach these days is to adopt the Micra service architecture break up your application into lots of little server or lots of services some of which may be little some of which may be large but just meaningful chunks of functionality that can be deployed developed and deployed and scaled independently of one another and then at the same time you break apart your database as well each service has its own database and then to make things even more complicated maybe you're going to use a polyglot persistence architecture where you'll use a mix of both sequel and no sequel databases and those days some of those databases can be sharded as well so we've gone from a monolithic app monolithic database to lots of services lots of databases and unsurprisingly that results in what you could call distributed data management problems you know and it really stems from the fact that we're using non all databases and and many of them right and so one classic problem is kind of it's sort of typifies by this so imagine you've got just orders and orders belong to a customer right incredibly simple domain model but then let's imagine that there's a constraint where the the some of the totals of the open orders for a given customer must not exceed the credit limit you know in a simple one in a standard relational database you can enforce that constraint just by using transactions right you begin a transaction you check the credit that you create the order you commit the transactions and kinda-sorta the properties of serializable transactions will enforce that that will ensure that that constraint is observed but then you go and break it up in the microservices where you have order management customer management how on earth do you do you enforce that invariant in this architecture right we can't use two-phase commit you know that's just sort of totally out of fashion these days and you even have that within a single no sequel database where you where you don't have proper asset transactions and you can only update a single act single entity atomically so you couldn't even you know if you put orders and customers in the same database that no sequel database enforcing that constraint is a challenge as well so what can you do about it well the standard approach today is to use an event-driven architecture right and the way that works is you change one thing in one database at a time and when you do that you publish an event to say I've changed it another service can consume that event and update its own state and publish more events and so you end up with it's eventually consistent event-driven workflow so for the order placement scenario you create an order for a certain amount of money that publishes an event an order created event which gets consumed by customer management which reserves that credit publishes an event to say that the credit was reserved which gets consumed by the order which can then become accepted or open and then that could pop send an email to the user your order has been accepted at cetera and have credit the credit check failed a different event would be published and then order management could cancel the order and send a notification back to the user so that that's sort of the standard event-driven approach but then that actually leads to another problem how do you atomically update the database and publish events without using two-phase commit because we're trying to avoid that right and the solution there is to just don't update the database just publish events and that's what leads us to event sourcing which is a very event centric way of writing your domain model and also persisting the state of your business object business entities so the big idea there with with event sourcing is that for each aggregate to use domain driven design terminology or business entity you identify domain events and those events signify both state changes and also just things that have happened that a notable like the credit check the credit was reserved which is the state change or the credit check failed which is kind of a notable event but doesn't imply a state change and you actually define explicit event classes as part of your domain model so for an account it would be the account was opened it was credited it was debited for a shopping cart it would be the item was added and removed or and the order was placed so you have these events and then where it gets interesting is rather than persist the current state of an entity you know standard account table with a balance you actually persist the events instead and then to actually reconstruct the current state of an aggregate you load those events from the database and replay them to kind of bait to reconstitute the current state of the entity and then if you actually have a lie number events you can periodically snapshot to avoid having to replay everything from the beginning of time and what's kind of interesting there is that if reconstructing the current state in functional terms is doing a fold over all of those events so you can sort of think of the present as a fold over history which is kind of an kind of nice encapsulation of the idea and then the other part of it is aggregates of these entities that commands flow into commands represent requests from the outside world and events representing both state changes and things of note flow out the other side and the way this would work in a particular application is that a request comes in to update a particular entity you load it load those events from the database you instantiate say using the default constructor a blank entity and then you replay the events you sort of do the fold at that point to reconstitute its current state you process the command which yields new events which then get persisted in the event store and then you can have subscribers are you also doing up to an optimistic locking check as well and then you can on the other side you can have subscribers to those events which then get notified when those events have been saved and they themselves can update new aggregates update CQRS views and also send out notifications to the user emails text messages and the like so that's sort of the big idea big idea behind event sourcing and how you would use it in a in your application so a bunch of benefits to doing things this way it solves a whole bunch of challenging data consistency issues in a micro service and also no sequel architecture it also gives you this nice reliable way of publishing events right because state changes correspond to events and you can't have one without the other and so you get you know you get a guaranteed reliable stream of events out of you system which could drive predictive analytics feed into spark streaming for example but also just send out emails your order has been been received etc it also solves the o.r mapping problem because your serializing events not your domain objects and the structure of events typically is quite simple and you can just say serialized to JSON for example and also this model is reifying state changes so you actually got an explicit sort of representation of the state changes of a given business object so that actually provides a reliable audit log so you can always go back and see what has changed by whom to a given business object and you can even just display the history on the screen like I recently started using a sauna right and you look at a task in a sauna and you see the history of that task who created it updated it and so on you get that kind of behavior for free with this model and what's really interesting it also gives you the ability to do temporal queries as well you can look back in time and see what was the state of a given entity three days ago for example which is really useful and some heavily regulated domains like finance but also it actually because it's preserving the history of everything that has happened within the system you could actually implement a new feature today and then but then replay through that module all of the past events and it's sort of behaving as if you had implemented that feature in the past which is kind of an intriguing idea okay bunch of drawbacks right you know it's kind of a the model itself is different so it's sort of weird and unfamiliar and there's a sort of period of getting used to it events once that publish do not go away you can't effectively like we kind of reply schema migration so you could sort of dynamically on the fly but then the events never go away so they could in theory rep you if you have a badly designed event you might actually have to deal with that for for eternity which can be a problem you know because this is a message based system there's the possibility of duplicate event delivery right so you have to design event handlers to be idempotent so that that can be a challenge the-the-the sort of the because you don't have acid in the in this in this model applications have to be designed to deal with eventually consistent data so for example an application could update some entity and then do it then query a view to retrieve that entity in that view might not have been updated yet so you know you might have to sort of carefully design your UI like make the assumption that that update had happened and then later fetch the data for example and then also the event store itself only supports primary key based lookup of event of the entities events so if you need to do queries which now involve joins and filtering and that kind of thing you have to use command query responsibility segregation and implement separate views that are maintained by folding over the event stream so you know it's another sort of complication there but it means that you end up with a bunch of views that are heavily optimized towards the queries that they need to support so you end up with quite a nice scalable performance system so it's probably something you'd need to do anyway so yeah both so it's both good and bad but it's something that I found quite that it's worked quite well in practice that will come Shh yeah sure quick one oh you know that's a really interesting question so yeah so does the messaging system need to be reliable I guess the short answer is yes it needs to be reliable give you at least once delivery semantics and also preserve ordering as well but that's sort of the kind of functionality that gets built that the event store itself provides or yeah so if you're like I'm building an event store I'm making sure that it provides those guarantees and if you build one yourself you have to you should build in those guarantees as well good question all right okay so that that's quick out intro to event sourcing kind of want to talk about why okay so how do we use this to design you know business logic how do we design domain models so what I found actually in all a lot of the Micra services an event sourcing work I've been doing domain driven design is really kind of a good guideline so go read the blue book go read the red book right implementing DDD so you kind of want to use all of the standard building blocks of domain driven design entities and value objects services and repositories and lastly aggregates which are kind of pretty much is which is sort of something in the past I kind of really didn't embrace you know I just kind of did oh oh that was sort of DDD flavored but I think when you're using event sourcing and just in jet in particular in designing micro services and in particular in general you kind of want to think about designing aggregates for it you know explicitly in your work so the I'm going to talk about you know what aggregates are briefly so the big idea right is you've got your domain model a bunch of classes entities value objects and you really want to partition it out into aggregates so afraid it might be a little hard to see but an order becomes an aggregate customer is another aggregate a product is the third aggregate and they end up so you're sort of chunking your domain model into these group little clusters of objects which actually have fairly loose connections between them so your domain model is no longer this tangled mash of objects it's sort of comprised of the aggregates that have constrain strained relationships between them and so in aggregate right it's a graph consisting of a root entity and and then other objects entities or value objects are hanging off it and they tend to correspond to business objects like orders and customers and products and shopping cart but there's sort of constraints right so a reference to other aggregates to other aggregate roots is not via direct object references we would normally do it in a traditional oo model but instead it's by the primary key so for example an order line item does not actually have a reference and object reference to a product but it just has the product ID so which takes a little bit of getting used to but it actually breaks this tight coupling and it's just sort of like a almost like a foreign key reference and then it gets a little more interesting so the actual order line item could contain a copy of the products name and the products price which on the one hand sort of is like well that's kind of like denormalized and you know violates you know normal form and all of that and sort of it's you know but on the other hand it actually sort of makes sense from a business perspective of well when I place an order you know that order it happened at a particular point in time and the price of that product was you kind of want to capture what the price of that product was at that particular point in time so it's from a business point of view it actually makes some sense so you've chunked up your your domain model into aggregates and there's just these ID based reference as to other aggregate roots so that's broken it apart made it a lot more modular which from a microservice point of view gives you a choice of very flexible deployment you could actually put customers and orders in the same service or you could break them apart and deploy them separately so that that's kind of a nice added benefit and it's also important to think hard about the boundaries of each of your aggregates because one transaction is the update of a single aggregate and so you don't actually have the opportunity to update multiple aggregates within a single transaction and if you think about what if I was using Orcas Island why let's just stick with right you know these aggregates map very nicely to a document a monger only lets you update one document at a time you know atomically right and so it's sort of surfacing that constraint at back up to the sort of the design of your domain models and so it's forcing you to think of that constraint explicitly a design time and then also you think well in a relational database but I can't update many things but once you start scaling out and your sharding for example having this constraint of only updating one thing at a time actually makes sense as well otherwise you end up with the need for distributed transactions so in a way so you kind of have to think explicitly about this and then you know design your granny ledge granularity correctly so you have choices right like you could I mean I talked with some folks once who basically I think made their entire system and I a single aggregate right and that was kind of good in a way because it meant that there that model was always kunia updates were serialized and so everything was guaranteed to be consistent but you probably don't want to do that you know you probably want to adopt a more fine-grained approach so customers and products and orders would be that would be their separate aggregates and it ends up being a trade-off between on one hand you have great consistency if you have a single aggregate but on the other hand you'd you're giving up on the ability to decompose your system into services it's not going to scale because every update and your system has to be serialized through this one aggregate and the user experience is going to suck as well right so so you kind of have to find the right balance be in the system another part of this of course is domain events right you know so those represent notable occurrences within the system and a little server cause the state it's part of the public domain of the AP it would sort of the public API of your domain model so you kind of have to think carefully about the design of them they're gonna have metadata like the ID they're gonna have essential attributes that are required in order to process that event by the aggregate so the aggregate needs certain pieces of data in the event in order for the business logic to make sense but then there's a concept of domain event enrichment where you add extra attributes that are not needed by the aggregate but they're used by consumers the events you actually eliminate the need for a consumer to go fetch that data when it receives the event so you can start stuffing extra things you know carefully you don't want to overdo it you also have to think about the design of the commands right so these are created they capture the data that's coming from the request they're processed by the aggregate to your defense and they just tend to be immutable they're just collections of value objects right and you know so what's is this data that's needed to validate the request create events do auditing and that kind of thing so at the bottom if you can see that there's an example of some commands okay so there's sort of first kind of piece of code I want to show you it's just sort of a hybrid or approach that I started off with and that just mixes State and behavior in to the same class and in a very sort of traditional oo state so a customer has like a credit limit field as well as you know map of credit reservations plus the actual behavior for processing commands and applying events so classic sort of oo with a bit of FP thrown in so there's some basic base traits that kind of that these aggregates have to implement so there's a aggregate trait that's parameterize by T and that that has an apply event method that the pastural function from the event to T so that kind of captures the idea of applying of updating state in an immutable fashion by applying an event and then there's a command processing aggregate that has a process command method that gives you a prototype or shal function from a command to a sequence of events so those two things sort of capture the essence of what an aggregate is and then you look at the code you know it's fairly straightforward right so so a customer aggregate is a case class there's some available credit reserve credit methods for doing basic comp computation process command does pattern matching against the command with a little bit of business logic to check to make sure you're not exceeding the available credit and it just returns events or sequence of events and then on the applying event side apply event is a you know it's doing pattern matching against the event and just using the copy method to actually copy we returned a updated copy of this immutable case class so you know there's not a whole lot of business logic in here but I think you get the idea and the actual underlying events store API which might it's easier to read on that one than this one you know it's got basic save update find and then at the bottom subscribe methods and it's all parameterize by T that's constrained to be a subtype of of an aggregate so you know in a way that that's kind of good right but if you kind of dug into it there's a whole bunch of runs unsatisfying things you know like it's throwing exceptions right that's which is not very functional the actual typing in terms of the events that in the signature it was it was event rather than the events that this particular aggregate and knows how to consume produce and consume and you know and also this is a sort of FP conference right and and sort of oo is a bad thing right well actually I kind of like oo but but it's you know it's not not that functional so so where I got interesting is I belong to this guy Oakland scholars study group when we get together once or twice a month and recently we've been studying Haskell which has made a whole bunch of us extremely happy intriguing language I still don't understand what I'm doing half the time but you know one of the things I did I stumbled across this high school type class that for event sourcing and it's just sort of incredibly simple and elegant right so it says you know a Type S is an aggregate if it has these methods or so if it has these hoc operations execute which is the takes aggregate and a command and gives you back either an error and an event and then apply takes the aggregate and an event and gives you back an updated aggregate and then there's seed that creates a blank one but where it gets also super interesting is is that the types of the errors the commands and the events are actually defined abstractly at this level these are kind of analogous to abstract type members in a trait and this is part of a Haskell extension ownit known as type families and I just thought that was just nice right it's just elegant lets you write code like this which is actually doing a fold over events to reconstitute the current state so it's doing cooling apply to apply an event and seed to create the initial state and I've actually been building on this implementing a tic-tac-toe game as a web service that's actually using a rest invoking the events doors REST API so here's the and I actually I took the tic-tac-toe game off found it on on a web page somewhere and I've sort of layered an aggregate on top of it so you can see that it's defining the concrete types for error and event and command it's also defining what executing a command means and it's just actually trying to make a move like place a zero or X or not or or X at a particular Square and if it succeeds it returns the move made event otherwise it returns an error and then at the bottom there's the apply event logic that's just kind of kind of a quite quite nice definition yeah but along the way I mean actually working at this level it's quite straightforward but then I had a deal with advanced concepts like in high school like getting the value of an environment variable containing the URL to the server right I mean that was the hour that was like so a huge sense of accomplishment when I did that right and then I made a rest cool that that was awesome as well so so this level was easy it was sort of dealing with the outside world got harder ok so that made me think let's just save questions to the end that made me think that worried I could do better in Scala right and then I came across this paper that was talking about how you can implement type classes in Scala and in particular it had a section about how you can represent those type members it's sort of what the Associated types as abstract type members and that sort of inspired me to just do it better and so is the new model right where I've got a functional separation sort of function of state and behavior so I have a customer data type that's just pure state and then there's a customer aggregate that's sort of a type class instance that's got all of the behavior so there's a couple of traits that define the tight couple of tight classes that sort of capture this behavior right so we've got the aggregate type class that's got the new instance behavior corresponding to the seed and there's an L so the apply event logic as well and that's used by the event store to reconstitute the state of an aggregate when it when it's loaded from from the database so we've got that bit and and it's got an abstract type member that's the aggregate event that's currently constrained to be a subtype of event though that might not be necessary and then there's the command processing type class that represents the concept of processing commands and that's got an abstract member type member representing the type of the command and unlike the Haskell version I've hardwired in a try just so I just didn't want to make go down the route of adding complexity of making that extensible but yeah that would be a poor request if anyone's interested so here's the customer you know behavior customer sort of class right case class its knows about credit limits and credit reservations there's a bunch of functions that act on it reserve credit you know returns a failure if you try and exceed the credit limit so that that seems pretty reasonable and then here is the actual concrete height class object so it's defining the actual types of the command and events so it's a customer command and a customer event you know new instance is actually just creating a blank customer then process command is sort of doing the obvious thing pattern matching against the command and returning the appropriate event and in the case of reserving credit it actually tries to reserve the credit if it succeeds it actually forgets about the updated customer and just returns the event and if it fails because the credit limit was exceeded it returns the credit limit exceeded event so pretty straightforward and I think some of this could be abstracted away and then apply events pretty simple right it's just the X the actual business logic to do it it already exists so it's just calling reserved event that's returning a try and then just getting the value out of it because we know that that will X succeed at this that by we get by the time we get to this stage I've also been experimenting with using lenses as well just know that was just for fun really so apply event is defined to cool this modifier for with modifier function that's this actually gives you a partial function from an event that then when you give it in to a function that when you give it the aggregate will return the updated aggregate so it's sort of kind of a couple of different levels of function there but then that lets you use like lens code in in to implement apply event and so this is using the monocle library and so you can just implement all this functionality using modify or source set and that was kind of kind of interesting if it you know if you like that kind of thing and then the event store interface is actually a bit simpler it's all parameterize by the type of the aggregate just yeah with a with a constraint to say this must be an aggregate so it's quite nice I feel kind of an improvement over the hybrid version so a couple minutes remaining just kind of wanted to look at a bigger picture of design right so if you actually look at a micro service it hosts one or more different types of aggregates there are these various adapt ters that are consuming requests from the outside world and also events from the events store and then invoking in one case a server service and in another case event handler which actually then figures out an aggregate to send the command to and then that that command will actually result in events that get persisted in the event store so if you know we consider the actual place order story right so as a customer I want to place an order and then you can think of a scenario that's got various preconditions and postconditions so the old-school way of doing this of course was to begin a transaction reserve the credit create the order commit the transaction right but in this new world this ends up being and if in an eventually consistent event-driven workflow so so the service when it gets a request in this case to create an order it actually has to create what's called a saga which is the little state machine that that coordinates the process of kind of approving the order right you can imagine in a more complicated scenario not only do you have to check credit but you also have to reserve in this inventory as well and do five other other things right so there's this whole kind of workflow that has to happen before the order can be properly accepted and in some scenarios you would create a synthetic set saga but in this case I actually using the order as its own saga so it's coordinating its own approval rights publishing the events and then gathering the events back well in this case just from the customer but it could also come back from from the inventory service as well so the create order service you know method creates an order which then publishes an event which then gets notified the credits been approved and and then it can say I pipe and approve right so you know that's that's the big picture flow but then you also have to think about compen seedings transactions right you know whenever you have eventual consistency something when it's time to actually perform a step the preconditions for that step might actually be false now right so the credit check might fail so you have to cancel the order the credit check might succeed but by the tight and then used to tell the order that but the customer might have already cancelled the order so you have to undo that credit reservation and and so you you know and actually you have to think quite carefully about all of these compensating transactions but in this case it's fairly simple so yeah creating an order here's what the code looks like and it's actually implemented in terms of this DSL that's just sort of saying what we're creating an order aggregate and we're sending it a create order command right and under the covers is actually instantiating the aggregate object processing the commands applying the event and then persisting them in the event store so it's actually picking up this implicit event store under the covers and using it so that that's quite nice and then so that's resulting in an order created event being persisted in the event store and then being published and here's the event handler for it so it actually we're creating a durable name subscription to the event store so if the consumer is not actually up and running that it's gonna get the events when it starts up again and then here's an event handler for order created that's saying well I'm gonna update this customer aggregate with this particular ID so it's coming the customer attribute of the event and I want to reserve credit for the entity that part the order entity that published the event you for this amount of money the total coming out of the event and that under the covers that's actually loading the customer aggregate with that ID processing the command applying the events to the aggregate am persisting it in the event store so yeah the nice thing about scholar is that it lets you write all of this in a fairly concise style which I really appreciate it anyway so I'm out of time unfortunately unfortunately I'm out of my slides as well so you know I hope that you found this useful and I'd love to get feedback yeah so shoot me an email about suggestions on how to improve this talk or what you thought about it etc so yeah so in summary write events sourcing solves a variety of problems with modern micro service no sequel application architectures you get a nice flexibility in terms of how you deploy your app whether it's monolith or micro services you can defer making that choice and then Scala is a great language for implementing all of this stuff right case class is pattern matching you know the functional the fact that current state is a fold over events implicit sand tight classes and and so on but I will say that Haskell is super intriguing and I would encourage everybody to bang their head ache bang their heads against the wall and try and learn it anyway so thank you here's my contact info and please if you're interested check out my event sourcing platform so thanks