scala.bythebay.io: Chris Richardson, Developing microservices with aggregates
Recording: scala.bythebay.io: Chris Richardson, Developing microservices with aggregates
I you yeah it's funny I'm not sure how extraordinary I really am but for the next 40 minutes I'm going to try and stand in one spot and you and just speak into a microphone because I'm really bad at speaking and holding a clicker and holding a microphone at the same time so this talk is sort of basically about Scala flavored microservices so a large part of the you know microservice architecture is really I mean it really is just architecture and it's very much language neutral but at some level of course you've got to get down into codons so you know this in this talk I'm going to sort of talk about how you can how how and why you should build microservices using domain driven design aggregates and then I'm going to show how you can implement those aggregates in scarlet because sort of scala adds this sort of functional flavor to things and so the aggregates are implemented using scarlet type classes so for those of you who don't know me I'm chris richardson i got my start in programming back in the 80s and spent the first seven years of my career building Lisp systems Lisp of course being an early functional and then object-oriented language and then eventually ended up programming in Scala wrote the book pojos in action which is all about spring and hibernate so that was ten years ago nine years ago I started tinkering with this obscure service known as Amazon ec2 back when it was in beta we were shocked at the Oakland jug to find that they had a P is for provisioning service right I mean take it for granted now but back then it was mind-blowing so much so that i created a startup called cloud foundry which was then acquired by springsource back in 2009 right before spring sauce was acquired by vmware so I was part of spring source VMware and then pivotal for four and a half years and then I left about through our three years ago so now I just sort of do everything related to mike receivers so just sort of you know living the microservice dream not really so I do consulting and training and i also have a startup called eventuate and we're building a platform to simplify the development of transactional business applications using the micro service architecture and if you want to know more go to learn microservices i oh and there's just links to articles presentations example code and whole bunch of good stuff okay so I want to first sort of set up the problem that I'm going to solve by talking about how there's really a mismatch between the micro service architecture and how we normally develop business applications right so the micro service architecture in a nutshell you know is a way of tackling complexity by modularizing the application and in this case the modules are services which have these impermeable boundaries you can't break encapsulation and so you know you architect a system like this where there's a whole bunch of services each rep is implementing a particular business capability and this is distinctive a characteristic of the micro service architecture namely each service has its own database it's data is private to it and can only be accessed through its API and that's really really important because it enables loose coupling between the services so teams can work autonomously on their services without having to coordinate schema changes with other teams but it actually creates some real problems so sort of the micro service architecture as a whole career has a bunch of it sort of challenges and then that functionally decomposed data based architecture creates problems as well so number one if you look at a typical domain model you know like here we've got orders customers and products the objects just reference one another right that's the essence of the domain model so patent so orders have a reference to a customer and order line items have a reference to a product and that's perfectly normal the trouble is is when you want to put orders in the order service and customers in the customer service and products in the product service you've got these things that are sort of object references and they and by definition object references cannot go across process boundaries so there's sort of this mismatch between a the domain model and microservices another problem that you run into is we rely traditional business logic relies heavily on acid transactions right so in this case we want to verify that a newt where we want to create an order but before we do that we want to verify that this new order will not exceed the customers credit limit and so that's trivial right you begin a transaction you find the existing orders you find the customer credit limit make sure there's enough credit and then you insert an order and you commit the transaction and the acid properties of transactions guarantee that even if there are simultaneous attempts to create an order for the same customer that that business rule will never be involved will never be violated you know acid transactions basically is serialized at some at some level so it's really simple and we just sort of take it for granted but there's a problem orders belong in the order service customers cantrell belonging that belong to the customer service so you can't write sort of transactions that access data in this way and if you did it would require a distributed transaction that would span order the order service and the customer service and for a whole bunch of reasons just you know modern applications do not use distributed transactions it's just sort of not the done thing anymore and there's sort of a whole but I mean obviously it would work subject to a whole bunch of fine print but there's a whole bunch of problems that you run into both sort of practical in the sense of how distributed transactions work and sort of issues around performance and then of course there's the fact that many modern technologies don't what distributed transactions right now pick your favorite no sequel database and then there's the cap theorem that says you basically have to pick choose between consistency and availability and people prefer available systems / consistent systems so it's sort of like isn't isn't an option in modern applications so it's like our domain models kind of don't our domain models resist decomposition and transactions also resist decomposition as well so what can you do about it so one part of the solution is to use a concept from domain driven design known as an aggregate so this really is a must read book even though it's sort of a relatively ancient texts being written back in 2003 2004 and it has nothing you know was you know microservices sort of obviously didn't work sort of in a sense didn't exist back then it's packed full of useful dessert or design techniques that are incredibly useful when building microservice architectures and it's sort of I mean it's a whole bunch of different techniques but one of them is this aggregate concept what's interesting is an aggregate is part of these sort of the building blocks of domain models so there's sort of role ways of structuring your domain models and some of those building blocks like entity value objects and repositories of pretty much entered everyday develop her vocabulary right I sort of read that you know 10 plus years 15 years ago and said yeah sure I'm going to build systems with with those putt those components and then there was this aggregate thing which had very sort of arguably these philosophical motivations for using it and I just ignored it like actually all of the useful stuff in dddi actually turns out I ignored and now I'm paying attention to but it turns out aggregates are really an incredibly useful tool when writing domain domain logic for micro service application so domain an aggregate is a cluster of objects that can be treated as a unit so typical business objects like or business entities like an order doesn't just consist of an order there's some other objects in there as well like water line item and and say address and payment information and collectively they form what it's called an aggregate so it's a way of basically modularizing your domain model instead of it just being at a massive interconnected classes it really is a set of interconnected aggregates so that's kind of good in a sense and sort of just says well this is the scope of an order but aggregates go beyond that and that's because they have to conform to a couple of different rules so one of those rules is that references between aggregates can only be in terms of primary key value so in other words an order does not have an object reference to a customer it has a customer ID likewise a product or so an order line item doesn't have a reference to a product it just has the product ID so and so within an aggregate you can use object references but between aggregates you can only use primary keys and if you're used to just traditional object oriented design this would be considered a sort of design smell it's just something that you never ever did but indeed edu do and it turns out that in a micro service architecture this is really really important because now your domain model is made up of aggregates that have very weak connections between them no longer object references but just primary key values so then that makes it really easy to put an order in the order service a customer and a customer service and the product and the product service so it enables you to decompose your domain model so it's great so you go read this ancient text on aggregates and you can apply it to modern application architecture but wait there's more another rule is that a transaction can only update a single can only create or update a single aggregate and when I read this it just like made no sense whatsoever right like I was using a monolithic well I was building monolithic systems i had a relational database and so I could begin a transaction update whatever I want and commit the transaction and then there was this crazy book saying no you can only update one thing and then once again there were these sort of philosophical motivations for this but if you think about it this is great in a micro service architecture because if you have an aggregate within a service then a transaction can will by definition be local to that service so we've sort of solved this problem of a sort of true where we've eliminated transactions that span service boundaries which is really really useful so it's sort of like yes there's this book that had philosophical motivations for saying this is now solving a kind of addressing a very deep technical constraint in a modern architecture and not only that it actually fits with how no sequel databases work because they don't have acid transactions as well so we've sort of it kind of kills two birds with one stone in a sense it you know fits with the micro service architecture and it fits with no sequel databases as well so that's aggregates right it's it's they give you the tool to design business logic that fits within the Micra service architecture and later on our show some Scala code that implements some aggregates so that's one so that's one problem taken care of the other problem is is that yes you know we've broken things up but we still need a way of maintaining consistency across services or in other words between aggregates right so because earlier I showed this the example of how to and you know we need to enforce this business rule that says that with its credit limit business rule and if if your system is decomposed and how we going to do this without two-phase commit you know how did how do we do this between aggregates and the solution is to use basically an event-driven approach which you could call us what goes by the name of a saga so instead of a distributed transaction that spanning multiple aggregates and multiple services instead you're going to use a saga that is a sequence of local transactions and for the credit limit case is actually a transaction in the order service and then a transaction in the customer service and the third transaction back in the order service and each transaction emits an event that triggers the next transaction in the sequence so that that's the architecture so distributed transactions get replaced by event-driven sagas so the way this so the way the order checking the credit limit checking would work is an order gets created in sort of this pending state it publishes an event saying the order has been created that gets consumed by the customer service which then goes and reserves credit for that water and then it will publish an event indicating the outcome of that credit check namely either the the credit was reserved or the credit check failed that event gets consumed by the order service which will then either approve or reject that order so we've got this event-driven eventually consistent workflow that spanning whether that's spanning multiple services and maintaining data consistency across them so that that's the new model so we've got aggregate and we have these event-driven sagas and it's not in you know it's a little bit more complex right so one example is if you think of it think about acid transactions when in the middle of an acid transaction you find out that a business rule is going to be validated you just roll back the transaction and those updates are basically undone it's as if they never happen whereas in this model you have to write code to explicitly undo what was done earlier in the saga so for instance in when this case write an order was created but the credit check was failed so we have to cancel the order and more another sort of more elaborate example is when you're transferring money between bank accounts you've debited one bank account and then you try and credit the two account you actually have to undo the debit and if that if the two account is closed you have to undo the debit by re crediting the from account so there's not you have to explicitly rollback so that's a design challenge and then another challenge that it's sort of a more technical challenge that you run into is in order for this saga mechanism to work you have you you need the underlying infrastructure needs to be able to atomically update the database and publish an event both of those things have to happen in order to to ensure correct working of your system right because for instance if you created the order and then crashed between before publishing the event that order would just be left hanging righted that the next step would never happen so those two things have to have to happen together and what's ironic about this is that the normal way of achieving this is to use the distributed transaction involving your message broker and your database but we can't use that so we end up how the sort of the underlying mechanism has to work and quite in a different way and there's a few different mechanisms that you can use so one of them is actually it's like in use that some companies like LinkedIn they actually tell the database transaction log and commit log and then when looking for changes to tables and then those changes get published as events which can then get consent rigger the next next step in in the saga to a few different options there but my personal favorite is one that's cool event sourcing and that's an event centric way of both persisting your aggregates but also an event centric way of sort of structuring your aggregates as well and it basically uses events for everything so rather than having to update state and publish events all you're actually doing is basically saving events in an event store which also publishes them as well so we just we get rid of the state now is consisting entirely of event and the way that works yeah is for every aggregate you identify the state changing events define event classes so you sort of model them as first-class citizens so for an order it would be order created canceled approved rejected and so on but then you actually persist the events in the database debate you could imagine instead of an order table and a customer table and a product table and in a very sort of simplified form all you know have in your database schema is an event table and so every whenever the state of a domain object changes you simply save an event into this table so when an order is created you insert an order created event when it's approved you insert an order approved event and then when it shipped you insert an order shipped event and that this is how you now store the orders so there's no separate or to order table and then when you need the state when you need to reconstruct the state of an order you just query the event table load its events and we play them which in a functional sense is doing a functional fold over that stream of event to reconstruct the current state so it's a very different way of sort of doing persistence and there's actually a whole bunch of benefits to doing things this way that I don't have time to go into but you can basically think that this event store is B hate is basically a hybrid of a database because you can insert aggregate events and retrieve aggregate event and it also acts as a message broker because you can subscribe to the events as well and there's a few different implementations and cluding that you know like light Bend have like I'm sort of pronounced that right and also my startup is building one as well and there's one in.net land as well so that saw the models so underlying so we've got aggregates that are persisted using event sourcing which generates the events that can be used to trigger the next step in a saga and then maintain data consistency so very very different model to this sort of the traditional acid programming model but it's you have to do something like this if you want to build transactional business applications using the micro servers architecture so that that's so that's sort of the theory and now it now where you can actually look at some code so one of the examples is this is the customer and order example so there's a customer microservice that hosts the customer aggregate whoops there's the order service that hosts the order and there's some other pieces in there that I will explain in a minute and then the the third service on the right is actually the order history service that that is subscribing to events that are coming out of the customer service and the order service and updating a materialized view that sort of got water what kind of customer order history and that it's using a technique known as seek urs and then there's this mysterious thing kafka command bus because Kafka's cool so you have to have it in there of course but that is actually a good reason so it's a little bit more elaborate than the model I've described so far and event in a minute I'm going to talk about why that it's a bit more elaborate and why I'm using Kafka and why in here are these other things there's not just an order ok so the there's a whole bunch of different ways of implementing aggregates slow so the Java way would be using mutable sort of traditional mutable sort of object-oriented classes scholar of course is all about function sort of functional style code and and so the aggregate model here is based on tight classes so like an order you know that is actually going to see this is a case class that's got the state and then the business logic of processing commands which yields events and applying events to update state is actually implemented by an order aggregate type class so in the framework they're there for instance is there's an aggregate trait that captures the essence of representing state and as a sequence of events so there's a new instance member which creates a new instance of the state and an apply event method that takes to stay and an event and gives you an updated state and then there's an apply event method that's doing a functional fold so this aggregates representing the concept of all this aggregate traders representing the concept of reconstructing reconstructing state by folding events and that's actually used by the event store when you load an aggregate from the event store oh and there's this abstract member called aggregate event which says for this aggregate this is the particular class type of events that it supports and and the implementation of this aggregate will you know define the concrete type so the event store has like a simple kind of crud API that's parameterised by the aggregate type that is T in this case and then there's this cut then there's command processing aggregate that builds on aggregate and adds in the concept of processing a command so a command is a request to update create or update and aggregate and so there's an abstract type member aggregate command that is the type of commands that this aggregate supports so those are the at those are the kind of the abstract framework level traits and then here's the order aggregate there's a whole bunch of stuff in here so there's like an enumeration here's the order state do I have highlighting yeah so here's the case class for an order here are the commands that the order responds to and then then here is the order aggregate type class and it says for that for this order the at the type is order command the event is order event and here's the implementation of the methods so new instance just creates a blank order and then we'll have a look at process and apply so here's the process command method which is taking an order and a command and it's just using pattern matching to generate a sequence of events so the business logic here is really trivial so it's that there aren't any business rules in here so it's sort of not doing anything particular interesting but other than saying you know for create order command return to create an order created event so it's kind of swapping the you know you got to do this command and it generates that done this event basically an apply is look similar so it's taking an order and an event pattern matching against them to return an updated copy of the order so once again that the business logic is kind of trivial so there's not much complexity here if you go look at some of the other classes it will that you actually will actually see at least three lines of business logic in this application because it actually has to the class that's keeping track of the available credit actually will say if there's enough credit then approve the order otherwise rejected that that's the you know there's only one business rule in this sample application so that's that part and then if you look at a service that say invoked from the webbed here it's written in terms of this dsl well me so we've got this sort of small amount of code thats basically says is a shorthand for creating an order aggregate so we want to create an order aggregate by sending it a create order command and under the covers is actually instantiating an order aggregate by calling new instance processing the command that gives you the event applying those events to update the state in memory and then it's interacting with the event store to save those events in the event store so that that's this really really simple aggregate and you can imagine if there's business rules the pattern matching would actually look a bit more computational logic in the pattern matching and so on that would be more interesting okay so that that's an example of the order aggregate and I now want to talk about this saga mechanism so the way I drew it before it was like this right the order service published events like order created which got consumed by the customer service which would then emit in this case like a credit reserved event or a credit check failed of em on the one hand that's kind of nice but it does mean though that the customer service has to know about all of the events that can affect of the credit like so what an order is create when an order is cancelled it would actually have to get that event and free up the credit that was allocated to that order so that there is an essence tight coupling between these two services which is bad and it so it's actually cyclic coupling which might be okay but it might not it just sort of kind of is a slightly smelly scenario so one way of solving this is actually to move the functionality for maintaining available credit into the order service so what that means is when a customer is created we actually create an available credit tracker for that customer in the order service so we've sort of kind of denormalized the data so the customer service so a little bit of data about the customer now resides in the order service and that actually is quite good because then the events basically bounce back and forth between the order and the available credit tracker and all of the knowledge about the various order events that can impact the available credit now just reside and tie with inside the order service so we've sort of localized the knowledge which is good and the only thing we might the only dependency that the customer service now has on the avail in the order service is that it would have to subscribe to the available credit changed event so it could update keep this filled up today so we've dramatically eliminated a lot of the coupling between these two services which is quite good you know it's better but there still is the cyclic dependency this is still slightly you know it doesn't smell as bad as the last slide but it's still a little bit smelly so yeah that's an interesting question where could you introduce a credit service yeah so imagine you'd like draw a line here between the two that that you could do that except that the arguments that I used on this slide would now apply to the credit service instead of the customer service so you've sort of you've kind of moved the problem somewhere else rather than rather rather than solving it in its entirety so this kind of I feel better about this I mean there's always different you know different ways of slicing and dicing it up but if you think about these approaches right you could you know like when it comes to sort of system integration there's two patterns or can choreography and orchestration this is sort of a choreography based approach yeah this one when you were saying that you were in omagh lodging they're available credit tracker into the auto service if you now had some other something besides borders that affected the available credit our customer would also be normalized e yeah I mean it all it depends I i mean i'll give you an example where where you couldn't really do this like so imagine that in addition to reserving credit you had to also reserve inventory it will probably be bad to have a have a replica of the inventory data inside the order service right so it's sort of you know what what what which approach to use kind of depends on the specific on tact specific but it turns out that on the next slide i'm going to show a different design that actually solves these problems but big by eliminating basically instead of this service having to subscribe to events from this from the order service this service will actually send commands to the other service as well so we're replacing in many cases events by commands which sort of inverts the dependency and is cleaner that that is curved there's some interesting kind of consistency issues that then arise so so yeah so so part of it is when the credit limit changes here an event would be published which would then trigger the update of the available credit the credit limit here and but please don't ask about the issues of when the how to deal with like the outstanding credit is now greater than the the new avail of the new credit limit because that's bit complicated but let me talk about the next that get to get to the next slide so this this approach where services are listening to one another's events is you could basically describe that as choreography where weather is sort of the flow is distributed around all these different services but it turns out that another way of implementing sagas is basically that is to use orchestration and so you have this basically an orchestration object that that is a that's basically a state machine that comes into existence in response to an initial event like an order being created it's like oh and orders being created so we need to go and validate that that order is is is allowed and the state machine actually receives events from aggregates and then sends commands via Kafka hence the kafka command bus so sends commands to other aggregates or in other words other services so instead of a service having so instead of the order service publishing an event that is consumed by the customer service the order service actually sends a command or in other words basically invokes an API that the customer service provides so we're actually inverting dependency the cust so the customer service doesn't actually know about the order service anymore the order service which makes sense just knows about the customer service so you've got this state machine that's actually a purser that itself is actually an aggregate so it's persisted but it sort of interacts in a different way it receives events from aggregates and sends commands to other aggregates so the whole so the flow works a little differently so we've got our available credit tracker in the order service than the customer services over there an order is created that cause Mia publishes an event which triggers the creation of this order creation saga so this little state machine that's responsible for implementing that this sequence of updates it will then send a command so the triangles or events these the squares of commands so it sends a command to the available credit tracker telling it to reserve credit for an order and the the available credit tracker publishes an event saying the credits been reserved and then the order creation saga consumes that event sends a command to the order service saying you've bit to the order rather saying you've been approved so that change that will trigger the state that state change of the order and then it will also send an update available credit command to the customer service so this as as I've been saying this is how the dependencies get inverted in this system you know this is basically invoking this is really invoking an API on the customer service so it's a one-way dependency at that point so we now have this sort of a site click set of dependencies in the system so it's a slight I mean I and then if you cool this more complicated but what's interesting is that now the workflow is centralized in one place as a close to being scattered around the system which in some cases is kind is simpler so so to me this is a very prominent promising approach so I'm going to sort of walk through the Scala code which at which currently is not is where it's very much work in progress it doesn't the code doesn't really reflect the state machine nature of what a saga really is and so it's a little bit more complicated but basically a saga is is actually an aggregate so it adds in more behavior so specifically well the critical behavior is this which is what's the specification of the state machine so ultimately I think it's going to look more like the won't look the same because it's quite different because the goals are different but you know I want to take inspiration from like the a curve state machine implementation so that's what the code is evolving towards so yeah it extends aggregate has the the concrete class has to supply a specification of the state machine and then here's the order state saga class and the key thing here there's a whole bunch of stuff here but the key thing is the state machine specification and then because it's because it sell it is an aggregate it has to know how to apply an event to update it to restore its state as well so that's that other method so and then here is the state machine which is not quite right I'm still working on it but basically it's saying there's a starting when so this so that says create an instance of this saga when an order created event occurs so we got that part and then the logic here is saying update the order customer telling it to reserve credit actually that's the credit tracker now and then there's this event which is representing the state of the saga as I said I want to make this more state machine like and then there's other clauses which says well when the credit limit exceeded event occurs then update the order in that way and then here's the event representing the state change so hopefully that's kind of giving you a flavor of what it does so the goal is when an event occurs update some aggregate which is actually asynchronously sending a command via via kafka and then it's generating event which is used in to preserve the state and so as I said under the covers it's sending these commands asynchronously using kafka so for every aggregate that you want to send a command to there is a Kafka topic and then for every saga there is a or there's a failure topic that gets a method that will contain a message if the execution of commands fail so like the order saga sends a reserve credit to the credit tracking aggregate if for instance the customer doesn't exist it will send back a customer not found message and the order saga can react appropriately to that like cancel the order because the customer ID was it was invalid they went perfectly I given you a sense of of what what kind of the flavor of how sagas work and as I said over the next few days the code should should look a lot cleaner so that's my talk which is good because I'm out of time so yeah in summary right aggregates to the building blocks of microservices event sourcing is a really good technique for persisting aggregates because it actually fits in really really nicely with this with the event-driven sag or approach that you're going to have to use in order to maintain consistency between aggregates and between services so thank you for listening hope that you found this useful and here's some con to info so thanks [Applause] you