SBTB 2014, Chris Richardson: Building microservices with Scala, functional models & Spring Boot
so for the next um 40 minutes i'm gonna sort of sort of talk about my experiences building an application that's got which which is for me has quite a different architecture so i'm using scala which is actually something i've been using for a while but i'm using microservices along with a functional domain model and then some really interesting architectural patterns event sourcing and command query responsibility separation cqrs and along with spring boots so it's sort of a mixture of new stuff and old stuff and it's been quite interesting and i i for me and i hope that you will find it interesting as well so a little bit about me so i actually started software development way back in the 80s actually programming in my sort of real first real job was programming in lisp building compilers runtimes and ides and so on lisp of course being a very early functional language but by the time i started using it it was object oriented as well and then of course eventually i started um using java and about eight years ago i wrote the book pojo's in action which was all about building apps with spring and hibernate which back then were these pretty radical technologies some of us still like them and then back in 2007 i started tinkering with this obscure thing called ec2 which much to my surprise sort of turned into this kind of whole cloud computing industry and along the way i created a startup cloud foundry which then got acquired by spring source right before spring source was acquired by vmware and and i like to joke that in fact cloud foundry was sort of much more of a plankton than a fish otherwise i would be fabulously wealthy and have my have my own island by now and but i i still work and i'm working on a startup right now and i'm also doing consulting so here's the agenda so i'm going to talk about why we want to build event driven microservices and then how i'm going to talk about how this technique known as event sourcing fits in really really nicely and solves some of the key data consistency issues that you get in a in in a micro service architecture and then i'm going to talk about some other sort of different aspects and i'm going to finish up by talking about how how you can build and deploy microservices using spring boot which is sort of one of the newer technologies within this inside the spring ecosystem so you know so why microservices so let's imagine that you're building a banking application and i want to apologize to anyone who actually is working in banking because my example is really really naive and probably wrong in a lot of different ways but you can imagine you've got a domain model like this where you've got an account that has a balance and operations like open debit and credit and so on and then you have transactions like a transfer transaction that transfers money from one account to another so you know you could scale this up some and you know if i was to implement this sort of using the way i've been doing for most of the time that i've been building enterprise applications and i'd have a ui component i'd have various backend modules like one dealing with accounts another one dealing with transactions perhaps another one dealing with customers and so i'd have this nice modular architecture and i would have used spring mvc and spring and hibernate as sort of the infrastructure frameworks to kind of deal with persistence and transaction management and so on so but i take my nice logical architecture and then i just package it up into a single monolithic application that i would then run on top of tomcat and you know there were pretty good reasons for doing it that way apps like this were pretty simple to develop pretty simple to test and pretty simple to deploy you had one file that you would just drop onto an app server and it would just run and if you wanted to scale it you just ran multiple copies behind the load balancer and for the longest time that was the way i built applications and probably a lot of you you did as well but imagine as you'll become you know we're sort of becoming increasingly aware monolithic applications sort of basically don't scale they don't really deal with complexity and you know there's probably some of us in the room who have who are now living with a million line monolith that they're just absolutely afraid of touching because it's sort of it's intimidating you just don't know how to make changes it's got complex build process and so on tends to be an obstacle to frequent deployments ideally we want to deploy to production many times a day but if we've got this large monolithic application that's really difficult a large code base also overloads your ide you know it just slows things down makes you less productive it also slows down the application server as well so if you're going through the make a change start up the server try it out and your server takes minutes to start up that's going to make you very unproductive and it tends to be an obstacle scaling development you want to sort of functionally decompose your team you know if you've got a large engineering team you want to have one team working on accounts another team working on transactions and you know but then at the end of the day they've all got to package up their code inside this single war file that requires a lot of communication and coordination and so on and then also a monolithic application requires sort of a long-term commitment to it to a particular technology stack it's very difficult to upgrade to newer technologies because you have to rewrite large amounts of code you know even today i still run into people who are using ejb2 and struts which are you know from the dark ages of enterprise java development so obviously the solution here is to functionally decompose your application into a set of standalone services or micro services you know so each service has one one very tightly sort of focused set of responsibilities typically just one function each one has its own database and so on and that really addresses many of the problems on the previous slide then another problem with the traditional architecture is that you you we've all sort of got very used to using a relational database the trouble is that sooner or later we're going to run into issues around scalability or maybe it's difficult we want to have a distributed database infrastructure schema updates on large tables can be a problem and of course there's the traditional sort of object relational impedance mismatch where we've got a nice object model and we have to map it into a relational schema and then also relational databases are not very good with handling semi-structured data and here now of course the solution is to use one of these trendy nosql databases so if i want to use text search i could use solar or cloud search if i want to store social data i could use a graph database like neo4j and if i wanted a highly distributed database i could use cassandra so you know sort of the model we're moving towards is microservices plus no sql databases or polyglot persistence where we're using a mixture of different types of database but then we actually you know we've solved some problems by doing that but now we've sort of kind of entered a whole we've encountered some other problems basically around data consistency we've sort of now got we've sort of partitioned up their data but we still need to keep things in sync so that's deeply problematic if we don't have two-phase commit um both with microservices and also with with nosql databases as well where we've sort of got lots of denormalized copies of our data that need to be synchronized so the solution of course is to have an event event-based architecture where services publish events when they're state changes that other microservices can subscribe to and then they can synchronize their rep their replica of the data so we end up with an architecture looks like this where you've got you know in the banking example we've got transactions over here and accounts over there and they're sort of publishing and subscribing to each other's events and keeping them both in sync the trouble is is that if you dig into it reliably generating events is quite tricky without two-phase commit because we need to atomically update a database and publish events as well there's a really good article by one of the guys who worked at ebay about their their approach and it sort of quite just ends up being a very tricky problem once you get into it if you want to be get you know commit your data to the database and be sure of publishing an event and that's where this technique known as event sourcing comes in and the big idea is that you're sort of taking a very event-centric approach to your domain model so you for each aggregate you identify the the state changing events and you defect define explicit event classes so for an account you'd have events like account was opened event account debited event account credited event and likewise for a shopping cart you'd have events corresponding to items being added or removed and and the order being placed and then what this is where it gets really interesting instead of storing the current state of an aggregate you actually store the sequence of events that led up to its current state so instead of having an account and storing data in an account table you have an event table that stores the sequence of events that occurred to that that that account like it was opened it was credited it was debited and so on and then to reconstitute the current state you you replay those events so you actually load the events from the event store replay them to reconstruct this out to reconstruct the current state of the account so it's a very different style of programming and you end up with something where the domain model looks like this i've got a few little bits of scarlet code because all of this is kind of written in scala of course so i've got a couple of base traits um that my domain classes in extend so the first one is aggregate and that defines an apply event method that captures the process of applying an event and returning an updated aggregate so it actually the apply event is a function that returns a partial function from event to t so that's you apply an event you get back an updated aggregate because it's a the domain domain objects are immutable in my model and then the command processing aggregate extends aggregate and adds this process command method that take plays the role of taking a command that represents an incoming request giving and applying that to the aggregate and getting back a sequence of events um then you know here's the account class you see the process command method makes use of makes really nice use of scala so it's a case class which has some benefits that i'll talk about on the next slide you can see that process event process command rather is doing pattern matching against the various commands that are flowing into the aggregate and each one correct returns a sequence of events usually it's just one event but there are scenarios where you might return multiple events so it's sort of splitting up updating into two parts generating the events and then applying them and so here's the apply event so it's a partial function that matches against events and then uses the copy method that's conveniently generated because it's a case class to actually return an updated copy of the of the account so you can see that debit just gives you gives you a new account where the balance has been decremented by the amount that you're debiting by and credits to going in the other direction so that that's what your domain objects look like which are kind of different and slightly weird but it's all very very event-centric and in more elaborate aggregates instead of just having say a single balance field that's a big decimal you can imagine much more complex much more elaborate structure and you could also imagine using techniques like lenses to do the update and so on but in this architecture you know requests are handled sort of like this you've got some kind of adapter that's handling http requests so an http request comes in the events for an aggregate get loaded from the events store a new aggregate in this case it's an account is just instantiated the events are replayed to create reconstitute the current state and then a commander's process which generates some new events which are then persisted in the database so that's sort of the flow so we're no longer preserving the current state where everything is the persistence is done in terms of these events and in one han you know that's you know there's definite just benefits to doing that but in the broader context of a micro service application why this is really useful is for the simple fact that other that the events store can actually publish these events when they're persisted so it's kind of acting as a combination of a database and a message broker and sort of transactionally publishing these events and then other microservices can subscribe to those events and then they get notified when the events are getting saved which could then enables that other microservice to update other aggregates so it means that we can change an aggregate over here in one microservice and another microservice can then update its aggregate to keep it in sync so we're sort of solving some critical problems there or it could update just some other data in some nosql database or in a relational database that's sort of a denormalized copy of that data for some reason so it's kind of letting us synchronize um a view of data with the system of records so we're sort of killing two birds with one stone and the actual event store api is a bunch of sort of looks like someone sat on the keyboard because of all the type constraints and so on but you can see there's like a save method which saves a new aggregate in the data database so it takes the aggregate and the and the events update saves an updated aggregate and then there's some fine methods and then down here there's actually a subscribe method as well which is how us how an event consumer can subscribe to those events that are coming out coming out of the event store and that's actually right now it's implemented using rx java to represent the stream of events that are flowing out though there's sort of some work that needs to be done and everything is asynchronous because you know that's the way to build apps these days and i actually plan to open source this at some point and you know some of you are probably wondering why am i using aqua persistence um and there's sort of a bunch of reasons around that um but one of them is i find acker persistence is much too sort of echo-centric um you know not not everything should be your needs to be an actor among other reasons um so there's a bunch of benefits to doing things this way um so number one from sort of a business sort of domain modeling point of view we get this built-in audit log so this stream of events which are representing the state changes to our system just happens to be an audit log of the changes so we know what was changed when and if we're including user information in those events we know who who made that change as well so we've got this nice auditable system that's quite important in a lot of domains and then another really interesting thing is because we're not we're no longer storing the current state but a sequence of time stamped events we can actually we can reconstruct the state of the system at any point in time so we can basically sort of do these historical or temporal queries like what was the state of this account on on this particular day which is quite useful and then to go back to solving some technical concerns you know which was the sort of motivation for using this approach it solves some date some really key data consistency issues in a micro service slash nosql based architecture so this event store is atomically saving and publishing events um other services can subscribe to those events and update aggregates so we get basically like this workflow mechanism so we can debit debit one account over here in one sort of step and then an event will trigger the other aggregate to get the other account to get credited eventually and then as i mentioned earlier it enables views stored in either sql and those sql databases to um be kept up to date and it also solves the or mapping problem because effectively in this system there's simply an event table so rather than tables specific to each entity within our system and which we then have to think oh we've got this object model here how do we map it to the database and instead it's all going through these events which sort of inherently have a much simpler sort of simpler design so that's quite nice it's really nice not to have to deal with or mapping anymore you know whether you're doing it with an or mapping framework or by hand you know you still have to deal with a mapping problem um you know the downside of course is that well this is kind of a weird and unfamiliar way to do things right our domain objects look quite different how was persisting things look quite different so it's sort of a bit of getting used to another really interesting thing is our events really end up being like because you're storing effectively storing events from the beginning of time and they that and they're immutable you make mistakes at the beginning beginning of the project and it's you know it's kind of tricky to sort of recover from them in a way um i've actually you know in the app that i've been building i've sort of been been been encountering that it's like oh my god i can't believe i did that but unlike with a sort of a system where you're storing the current state only and you can just do a schema migration to fix things you know the events are frozen so that's kind of tricky and then of course in any event-based system that the you end up having to sometimes deal with duplicate events because sort of implementing a message or an event store or a message broker with exactly one semantics is really really tricky so it's usually at least once which means under some circumstances you might see the the same event twice and sometimes that's not a problem because you've got item potent commands but other times you end up having to do duplicate detection like keep track of the most recently seen event and if you see one before that then you know that you've processed it so there's a little bit of work but on the whole it's it's sort of been an interesting and useful approach so kind of going up a level and looking at well how you know how does this what are microservices look like when you're using this approach you know the sort of the design pattern i'm really following is that now a microservice really implements one aggregate or possibly you could bundle aggregates together and package up the code for multiple aggregates within site one service but there's sort of this fundamental chunk of design and then you actually have flexibility as to how you package it into services so you have an aggregate then there's various adapters that are handling requests using a variety of different protocols from the outside world http or they could be plugged into sort of more enterprisey message buses and so on and these these adapters are turning those incoming requests into commands that get processed by the aggregate which then saves the its events in the event store and then on the flip side an aggregate subscribes to events being published elsewhere any and there's an event adapter that takes an event turns it into a command and then has it processed by the aggregate which generates events which get persisted so that that's kind of the model um sort of the sort of standard design pattern that i'm following and then you know what it's if we to take a slice through one of these what does it look like so you know so here's a spring an asynchronous spring mvc controller um so that's layered on top of the async servlet api and so it's taking an http request the post request to create a transactions the spring mvc is nicely turning the json body into a transfer details that's got the from account the to account and the and the amount to transfer and then it's calling a transaction service over here to transfer the money and it's all wrapped up in a for comprehension because th this is all sort of reactive code and then at the end it's turning the future into a deferred result which is spring mvc's version of of of a future um and then here's the service and what sort of the interesting thing about this is that i've got this sort of what i think's kind of a nice little dsl that expresses the semantics of creating a command and having it processed by um an aggregate and here what it's saying is so here's we're creating we're creating a create transfer transfer transaction command um and sending it to a new entity of type transfer transaction and that arrow under the covers is actually sort of taking care of instantiating a brand new transfer transaction processing the command applying the generated events and then persisting the events in the events store but all of that sort of hidden from you with just this sort of mini mini little dsl and the transfer transfer transaction aggregate looks kind of like the account um and it's got a process command method and an apply command method and it publishes various events like i've been created i've recorded the fact that the debit of the from account has occurred i've recorded the fact that the um credit of the two account has has occurred so it's generating these events and then at the same time it's subscribing to events that are being published by the accounts so these if you remember the picture at the beginning the the transfer transaction is publishing events that cause the account to be debited or credited and when and that debit or credit publishes events that the transfer transaction listens to to so that it can update the state of the um transfer so in this particular case it's using another sort of version of the dsl to say for this particular event for this debit event for um go find the existing transfer transaction with that with the id that's stored in the event and then go tell it to process the command to record the fact that the debit occurred um and under the covers it's doing the load from the event store processing the command applying the events and then persisting the events again so that's kind of like what all these sort of the adapters both at the http level and at the event handling level look like and so it's kind of an interesting style of design but you know there's one interesting thing about this is that the event store as a database has a very primitive api it's basically load by primary key yet you know if you look at a screen on a typical application you're getting data from multiple aggregates possibly and you're doing joins and quite you know relatively complex things so let's imagine you know in this banking example you know when you log on to bfa.com or wells fargo you see you look at your account and it's displaying a whole list of you see the balance and a whole list of recent transactions and with the design that i've been talking about to do that you actually need to do a join between the accounts and all of the transfer transactions that have impacted that account either transferred money in or transferred money out so it's sort of if we were doing this in sql we'd do some kind of in or out of join between those two tables but you can't do that with an event store because it's primary key based and so that's where this other architectural pattern command query responsibility separation comes in and when you apply that you define separate views or separate database tables that store in a denormalized form the data that's need to support the query side of your application so the the event store is supporting the event the command processing side of the application is publishing events that these views are now consuming so that they can um build up these sort of materialized or denormalized views of of the application's data so basically for the kite to support the queries you define some number of views and those those views use the database technology that's best suited to the queries that you need to execute so you know if it could be cloud search or solar for or elastic search for text search neo4j if you need to kind of do searches of social graphs or mongodb it's really really useful because you can have quite large json documents stored in that you can just access with a prime you know with a query or even a primary key look at and you just get a blob of json that you can then hand back to the browser which can then render all of this data so you have these views and then in terms of microservices you have enough data that subscribes to the events coming out of the event store and updates the and updates the view and then there's a query service that responds to http get requests and queries the data and then in one of the example apps that i built i represent accounts as a manga an account and its recent transactions as a as a single document inside mo in in mongodb so you know we've got the account got the balance then he got the the transactions that have impacted that account in some way caused debits or credits and to occur and then i actually store the list of of the recent debits and credits so it's very sort of denormalized representation but with one primary key lookup you can grab that mongodb document and pass it back to the browser which can then render it so that's kind of a really nice pattern and got a bit of code here but you know so you've got an events handler that's subscribing to to various events and then just doing in-place updates of a mongodb document so key thing is it's doing just pushing changes onto that list of changes inside the document so we don't even need to read it from the database can just do an in-place update and that's all done quite nicely with spring data for yet another spring project and the queries are also quite nice as well you don't even actually have to write any query code those get auto-generated by the spring data for dynamic repository mechanism so that that's sort of the view side of the app so you've got the event store that's still acting as the system of record and then we're storing denormalized copies of that data inside to support sort of the ui so final part of the talk is just about how you've got these apps or they've got all this code how do you build and build it turn it into an executable application and then actually you know deploy it into production so the technology that i've sort of been got been using recently and i've sort of been very excited about is spring boot which is yet another project inside the spring ecosystem and you can think of it as a framework or and tooling that sits on top of a whole bunch of other spring projects that makes it super easy to package up your application as in as an executable jar file so it's ready to run and it it uses convention over configuration to configure your app so you have to write very little configuration yourself it also makes it easy to externalize configuration like the host and the port and the user id and passwords of any databases or other infrastructure services that you need to talk to and you can actually supply it run at launch time a separate config file or it can grab it from environment variables or from the command line which is really really nice you really do not have to worry write any code to do that and then there are plugins for the id for various um for like well gradle and maven that spit out executable jar files so you know probably some of you have used spring and you know that when you use when you're using spring you write your application components but then you have to supply configuration metadata that tells spring how to assemble and configure your application components so way way far you know a long time ago in the distant past you actually had to write xml um but since about 2007 you haven't had to do that so in the beginning that around about then the annotation based configuration was introduced and then more recently there's java configuration where you're writing java code to do the configuration that uses some annotations as well and that gives you some nice type safe configuration that you know statically typed which is really quite good and the neat thing is is that you know and so it actually got to the point where you had in the beginning you had to write tons and tons and tons of xml and then with those two things you ended up having to write very little config code and now with spring boot you end up writing even less because it just it does things like look at the jar files on your class path and say well you've included a database driver therefore you must be accessing a relational database and so i automatically defined all of the the components like the jdbc transaction manager and the and the jdbc data source so you don't have to write those yourself and you end up writing very little config so this is basically the configuration for the java service so it's a configuration class and there's sort of an app configuration annotation on it and that that's basically it and then you also have to write a main program and all that's doing is calling springapplication.run passing in that configuration class and then also any of the command line arguments and you're good to go and then building um your app is quite straightforward so because there's um plugins for gradle and maven um so you just you know um gradle build will give you an executable jar file that's got command our command line arg processing that's really quite nice so you can tweak the port that the server listens to and it also gets health checks built in as well so if you ask the application need on the on the slash health url it actually tells you about the status of the things that your application is talking to and that that's an extensible mechanism so you get all this stuff for free which is really really handy um so i'm making use of that so each micro service is now a spring boot application and then the other part of it that i just sort of throw in there because docker is cool so i have a jenkins-based deployment pipeline that's building and testing the application and then it's taking the executable jar file packaging that up in inside a docker image and doing a smoke test of that so it launches the container and then pings the health url to make sure that it's kind of functioning and then it uploads it to a docker repository and from there it gets pushed into production and it's one of these build pipelines per per micro service so anyway so that's sort of kind of a whirlwind tour of of sort of this sort of way i've been building apps recently and i hope that you found it useful so you know the key thing is is that event sourcing solves some really critical data consistency problems that you have when you're using microservices and also that you have when you're using either nosql databases or multiple relational databases and then i also think you know spring and scala play nicely together you know i've been building apps with spring and scala for a number of years now and then spring boot kind of makes it really really easy to take those apps and package them up as sort of executable ready to go production ready applications so anyway that's my talk um hopefully you can follow me on twitter shoot me an email and you'll you can find more information slides and whatever on my blog plain old plain oldobjects.com so thank you and there may or may not be time for questions actually could you speak up just a little uh so the question is how do i how do i deal with large transaction lock i'm not sure i i don't um so the well the event store is currently sitting on top of other actually right now it's sitting on top of dynamodb though it could sit on top of other databases as well um but so there's that but i i mean it's your question referring to the fact that the every time there's a new transaction i'm pushing onto the list well the thing to do is you could have a separate task that then trims that list right like drops old entries off the back like you know just preserves the last like you go to your you know the last n days of transactions you can do it that way um yeah you're closer so i can hear you okay would you suggest actually in that same case a strategy to take collapsing records into snapshots oh okay yeah so the question is snapshots are really interesting so yeah so if you if you have some long-lived entity then there's all that's being updated regularly then obviously there's a long you could end up with a lot of events that when you load them from the event store get have to get replayed and that could get expensive so yeah in event sourcing there's the approach of periodically taking a snapshot and then and which means that you then only need to load the snapshot and then the events that occurred after that snapshot like it's like the um memo what's it pattern right where you just say ask the entity can you just sort of give me a serializable representation of yourself i'll save you in the event as a snapshot event and then just grab that in any events that that happen afterwards to reconstitute the current state um yeah um do you find that it's a problem if you want to change the behavior that is yeah so sort of sort of a combination of things so i mean part of it i think actually designing events is a little tricky um but like so the simplest form of events really it just capturing it's almost like you're capturing the data that came in as part of the request and you're effectively replaying that and yes it might that might evolve over time so you have sort of version one event then you might have version two events and you you actually have to support both of them so that that that sort of that that can be kind of an inconvenience um and yeah and in general um not only sometimes you actually have to there's like if you go read a domain driven design book there's the notion of like having rich events that have extra data in them so the aggregate doesn't need that data to actually update itself but then consume but then you include extra data in the event so that consumers have that data readily accessible so they don't need to go and get it from elsewhere and actually making choices around what data goes in events can be tricky at times uh yeah so it sounded like you were using event sourcing to in part and solve the problem of having to persist something and then notify about it what could work in the other could fail but it doesn't your event store itself on that problem if it's gonna receive the data and then notify yeah that so that that's a really interesting point i mean if you think about what kind of atomically committing means right is that you you know at some level you have to like perform an operation that truly is atomic right and since the event store is simultaneously persisting and publishing kind of like you can imagine it having a transaction log that's where you're in where you insert those events which and the act of doing that simultaneously persists those events and then makes them available for publishing so you sort of you've you've got that atomic step whereas if it was a database and a message broker then you know you'd have to use two-phase commit to do that atomically or you know one of the things that the ebay guys do is they or did they actually use the the the relational database that you're storing stayed in as also a message queue and so you could have a local basically a local transaction that committed and kind of sort of published at the same time the downside what i didn't like about that is that it required a lot of extra code that you'd explicitly have to write whereas sort of this event-sourced approach just kind of is a is a nice uniform mechanism that just takes care of everything in one go and without any special casing at any point yeah i think my question you may have just answered but in banking or creating script ordering matters a lot and how do you enforce time and ordering to distribute something like that yes that's interesting um so yeah so right now sort of the event store is just one a single server um and so ordering at that point is easy um and then but then across the distributed system i mean ordering is is slightly fuzzy anyway right um but there are the but the notion that there are these i mean one of the things i'm doing is event ids that are a combination of like system time in milliseconds plus a node id for for uniqueness and so that's giving you sort of a a rough watering um uh you know the event ids are monotonic are roughly monotonically increasing though because of delays from one i mean from once uh you know they might get slightly out of water um because of network delays so there's sort of some interesting issues around that well i mean this this system is eventually consistent right what's that well that's why i i began this talk with an apology um but but the interesting thing if you think about i mean once again if you work from a bank please do not listen to anything i'm about to say but if you work for a bank there's oh sorry if you look at how banks work there is some kind of weird eventually consistent notion right like so if you transfer money from your account to someone else's the bank will immediately take it out of your account right and then eventually it ends up in someone else's account and and i'm sure that under the covers you know they're actually putting it in their account or whatever but there is this sort of um you know it strikes me that it's not as acid as you might think it as it should be and then i mean i also at one time i worked i i had a brokerage account at like ubs and if you look to if i logged on to this account after about one nine o'clock pacific the numbers in that account did not make any sense right i mean they were they were doing some updates and things running batch jobs and things were seriously out of whack like massive negative balances and stuff like that um so yeah i don't know maybe if you need sort of global consistency this is perhaps not the right approach but then once again i go back to the original ebay article on this in cacmq magazine you know they were dealing with money right you know it's it's a commerce platform and yet they were using and eventually you know eventually consistent strategy because it all it it's almost like it did absolute consistency was you know um very difficult to achieve inside a distributed system anyway thank you