chris rechiarson revised 3 17 16

chris rechiarson revised 3 17 16

Recording: chris rechiarson revised 3 17 16

my talk on some micro services in general with some exit with some example Scala code and some example Java code so a little bit about me if you don't know me I'm Chris Richards and I live across the bay in Oakland I got my start in programming back in the late 80s building Lisp systems so runtimes garbage collectors all the way up the stack to quite sophisticated IDEs and then back in 2006 my book pojos in action came out maybe I'll stand over here came out and that was all about building applications with spring and hibernate and then back in 2007 I started tinkering with Amazon ec2 which back then was an extremely obscure service like it had just got started but then but the concept of being able to provision 20 servers and pay just ten cents per hour per server just blew my mind so I ended up creating an open source project which evolved into a platform as a service for java called cloud foundry which was then acquired by springsource shortly before spring source was acquired by vmware so told this joke on tuesday so apologies to the people who header before but i'd like to say that this this is not drawn to scale in terms of market cap otherwise i would be multi billionaire by now cloud foundry was much more of a plankton than a than a fish anyway so i was there for four plus years these days i do everything related to mike receivers is consulting and training i'm actually teaching a public class in oakland on one day class April twenty-eighth if anyone's interested in having an in-depth look at microservices and I'm also the founder of a start-up that's a platform that makes it easy to write applica to write event-driven microservices so that's called eventuate dot io not to be confused with a github project in the Scarlet community with with the same name and if you want to have more information than just fill in the form here and i'll send you a bunch of links that are relevant to this talk so event march events March seventeenth ok so the agenda I want to begin with just the sort of philosophical discussion about why one of the projects they've been working on is a pattern language for microservices talk very briefly about micro service architecture versus monolithic architecture I don't want to sort of rehash the intro argument too much and then just dig into event driven applications with event sourcing and seek urs and look at some code examples and then if there's time I'll showed and I am a short video of an interesting application that we recently built okay so why a patent language for micro services so now 30 years ago fred brooks famous computer scientist well known for the mythical man month said that there are no silver bullets in software engineering so in other words there's no technology that if you adopted will give you a 10 x boost in productivity I mean there's actually a whole bunch of things that if you do will slow you down by that order of magnitude but they really aren't these silver bullets yep 30 years later you know developers are passionately arguing about their favorite technology absolutely convinced that if you adopt it everything will be rain you know unicorns and rainbows and you often see arguments like this especially on Twitter it's like if you do X a puppy will die or a kitten therefore you must just to get a sense I've sort of created this little quiz where you can match up things on the left with things on the right you know I'll leave that as an exercise for for later but there's a tremendous amount of discussions like this where this is really bad and you have to do this and the underlying reason for a lot of this just sort of hype within our community is that we as human beings are basically emotional beings right so cognitive psychologists like to use this metaphor of an elephant and a writer so the elephant represents the emotional part of our brain that's how we make decisions basically using emotions and the writer represents the this sort of the rational part of our brain which tries to influence the elephant but really the elephant does exactly what it wants and the the writer basically comes up with logical sounding justifications for why you've come to some decision emotionally so if you take this model of human behavior it explains an awful lot about technology adoption and decision-making also on a broader scale why Donald Trump is doing well really it does so the actual a and also some of the other candidates as well but yeah there's actually this book righteous minds why good people are divided by religion and politics by cognitive psychologists and it explains reality remarkably well but you know as engineers we need to battle we need a better way of making decisions right to something to overcome our sort of emotional infrastructure and for me one of the one of the one really useful tool that you can use to describe technology is the idea of a pattern or a software pattern design pattern you know an idea that itself is 20 plus years old and went through its own hype cycle but the actual structure of a pattern provides a really really good for word format for discussing technology so you have you know obviously a pattern has a name that describes that describes it it has a context which is really really relevant right so solving a problem using one solution in one context makes sense but perhaps in a different context maybe a different solution would be more appropriate so you think about today many of that much of the thought leadership is by companies who are operating an extremely high volume netflix Facebook Twitter and so on and maybe if you're building a lower scale application and perhaps some of their stuff is not the best approach just to throw out an example is the problem that you're trying to solve there's forces which are the issues that you're trying to address and then obviously there's the solution right you know that's the whole point of the pattern and then there are two really interesting parts to the pattern one of which is the resulting context what happens when you apply this pattern so that that says well here are the benefits here are the forces that have been resolved but also here are the drawbacks here are the forces that have been left unresolved by the solution plus there are issues what are the new problems that applying this pattern have actually introduced so you know just to pick an example right like reactive programming is all the rage yet code understandability and debug ability suffers quite quite a lot in a reactive architecture right so that even though it's high performance there's death distinctly trade-offs and then there's also related patterns so these are either patterns that provide an alternative solution to the same problem or these are patterns that solve problems that have been introduced by applying this pattern so the pattern just kind of points you through the solution space and helps you helps you apply it helps you solve a problem or apply a particular technology so that's sort of philosophical discussion they want to one of the things they've been working on for the past 18 months or so is a pattern language for microservices so if you go to microservices i oh you will see this and it's a collection of patterns that can help you apply microservices well first off decide whether you should be applying microservices and then if you are how to apply them so there are the core patterns monolith versus microservice architecture there are infrastructure them and they were various infrastructure pattern such as how do you deploy your micro services do you run many of them on a single machine do you use a micro service per container or a micro service per vm then because you're building a distributed system there's a whole host of communication related patterns different styles of interprocess communication service discovery sort of the API gateway how does how the components clients of the system interact with the system which is now made up of micro services oh there's the micro service chassis a framework for upon which you build your micro services and then which is really what I'm going to talk to talk about today is a set of application patterns which are all concerned with data and in particular maintaining data consistency in a micro service architecture and this is very much what very much work in progress every so often I add new patents so that's so that's the pattern language go to microservices dot IO and to learn more so I'm going to talk a little bit about the monolithic of architecture versus Mike receivers architecture I mean I think you know we all pretty much have heard kind of why microservices are fantastic we've also heard why microservices are not so great namely the complexity but I kind of wanted to make a couple of quick points so in theory right you could build a modular monolithic application which gives you many of the benefits of a micro service architecture but in practice I think most monolithic applications end up as a big ball of mud right I mean it just study this is certain inevitability that you write some code or write an app that application is just going to get bigger that the quality of the design is going to degrade it's just going to be a lot of tangled dependencies and you're going to end up in a very painful place where any notion of agile development and deployment is next to impossible in theory you can but I think in practice is extremely difficult so that means that for most large complex applications you know microservices which for many reasons are not a silver bullet attend to be the right choice for those large complex applications I don't know what large or complex actually means you know for some people that could be a million lines of code but I did hear a talk about some one company around here who had 75 thousand lines of C sharp 15 developers so it was kind of a tiny application but their whole development and release cycle was sped up by an order of magnitude by migrating to a micro service architecture so it is very context specific but in general the big benefit is that it lets you build better software faster right or build and deliver so you as a developer could come program more productively the tempo of your releases goes from quarterly down to and hopefully many times a day and then also I think another thing that's really worth pointing out I think a key benefit of the mic for service architecture is that it enables you to try out new technology now I know like you know I very much live in the Java world well the JVM wild I should say and you know this is a scholar meet up so I'm sure everyone here is a big fan of scholar despite that the slow compilation times but there are other technologies out there right you know node is a good candidate for some microservices golang is another alternative closure and so on even Python possibly so one of the nice things about the micro service architecture is it really does let you experiment with new technologies and fail safely so you know you can sort of you going forward we're going to be in in increasingly a polyglot persistence a polyglot world where we're using mixture of languages I guess we're using a mixture of databases as well there's very similar arguments that apply there so that's my brief brief commentary on basic microservices and I just want to talk about event sourcing an event-driven architecture so there's a whole set of patterns starting off with database architecture and going into event-driven architecture and then various ways of implementing an event-driven architecture so one of the key patterns in the micro service architecture is a database / service which is a slightly inaccurate term what it means is that the data each microservice owns data that data is private to it it doesn't necessarily mean that it's an entire database server / microservice it could just be private tables a private schema or possibly a private server there's lots of options and in a way the shared database pattern because of the coupling that it introduces between services can be considered an anti-pattern rather than a sort of legitimate pattern so you want to really go with the database / service approach right so if you're dealing with orders and customers that water service has its order database the customer service has its customer database they could very well be implemented using different technologies one could be relational another one could be a third could be neo4j so it's sort of this Polly got persistence functionally decomposed database world but one of the problems that introduces is how to maintain data consistency across across these multiple databases so if you think about that's what we use acid transactions for right so in this example customers have a credit limit so that means that when we create a new order we need to verify that that new order will not exceed a credit limit if you if both orders and customers were within the same day base you could accomplish that fairly straightforwardly by using a transaction with the right isolation level and that so it can enforce this invariant but in a micro service architecture is partitioned so orders in one database and customers in the other so it's sort of like how do we maintain those invariants now we cut you know from a whole bunch of reasons we cannot use two-phase commit and so that leaves you in the direction of whoops having an event-driven architecture so the idea is that when state changes in one service it publishes an event an order has been created that would get consumed by the customer service which would perform the credit check and then publish an event indicating the outcome of that credit check and the order service would receive that event and update the state of the order to approved if the credit check was approved cancelled if the credit check was rejected so you know first read about this approach I think 10 plus years ago in an article by ebay so they had functionally decomposed their databases and even though they were relational they still used an event driven approach as opposed to distributed transactions but that integer introduces a problem how do you atomically update state and publish an event the traditional solution to that of course is two-phase commit right where both the message broker in the database participate in that same transaction but that's exactly the kind of approach that we're trying to avoid so there's several different ways of reliably sort of atomically and reliably updating state and publishing an event so the application can can actually write two concerta vents into an event table as part of that local transaction you could use database triggers that could work as well you could tell the transaction log because that is the source of truth in the system then the approach that I'm particular and there's actually various trade-offs with each one of those approaches that I most some of which I wrote up in an article on the engine X blog and then there's a vent sourcing which is sort of my preferred approach and the big idea behind event sourcing I'm sure some of you are familiar with that in the context of arc of persistence but the idea with event sourcing and is instead of say taking an order mapping it to an order table or a row within the order table instead what you do is store the sequence of state changing event so when an order is created you insert you save an event order created of em when it's approved you insert another event when it's shipped and so on so you're actually storing in the database the sequence of state changes and then when it's time the one you need to reconstruct the current state of the order you just load those events and basically do a functional fold to reconstruct the current state of the order and in order to optimize it you might use the notion of snapshots as well so you periodically snapshot and then you only have to you find the most recent event most recent snapshot and played and replay the events that have occurred since then so that's kind of the essence of event sourcing at a super duper high level but one question you have is well what about queries so if you suppose for a minute that the events are stored in an event table inside post gross some people actually do that you could write queries to say find all of the bank accounts whose balance is greater than X right and theory you could write a query to do that which would somehow in sequel have to do the functional fold of the events each aggregate you probably really don't want to do that it could I mean it gets complicated and inefficient very quickly so that's where another pattern comes into play command query responsibility segregation and the idea here is is that you split your system into two parts you have the command side which handles updates or creates and updates and deletes so in other words HTTP POST puts and deletes you also you're there and then you have the query side that handles queries in other words HTTP GET and those two parts of the application which have separate data stores and men are kept in sync by the query side subscribing to events that are published by the command side so as the name suggests it's really separation of responsibilities so the query side gets these events it can update a materialized view so in terms of microservices it could look like this so you have a view which could be MongoDB right because that's like jason's good jason storage for supporting a REST API in a web-scale way it could be neo4j because you need to do graph queries on a social graph or it could be elastic search because you want to do text queries so you pick the kind of view to support the queries that you want to execute you then have a service that subscribes to the events and is responsible for updating the view you have another service that handles the HTTP GET and queries the database queries the view and retrieves the data so that's sort of the piece so putting them together you end up with an architecture that looks like this the fonts a little small so you have microservices that play the role of for the command side the host domain driven design aggregates that are persisted using event sourcing you have a view quick quill view up work view query maje of that view module that's updating the data store and another component that's handling the get you might also have other services that are subscribing to the event store and sending out notifications email twilio text messages and so on and so the product I'm building is basically an event store that supports this model and then it's got one or more client site frameworks that are both that a language and framework specific so today I was going to show the Java flavor in the Scott and one of the Scarlet flavors of those client frameworks so yeah there's actually different flavors there's sort of a traditional Java model that's got mutable domain objects so just fields that are just set in the traditional Java way and then I also have a couple of different functional scala models one that sort of hybrid oo FP and then the other one that's more hardcore FP using tight classes I guess like if you're at my scholar by the bay talks like last year in the year before I might have talked about i think i did talk about both both these models so let's quickly look at the java version the one that mutates state and kills kittens so going to look at the customer aggregate so customer in this simplified model has a credit limit which is money and credit reservations that's a map from order ID to the order total so saying this order is reserving this much of the credit the actual business logic is implemented by process methods that take a command so that's a command to perform an update and they return a list of events then there are apply event methods that take an event and do an in place state mutation so just modify the customer state in some way so that that's sort of the model and if you think about it right so business logic is has this very stylized form of process methods that take commands give you events and apply methods that do the state update return of voice and what's basically having it happened here is that an old sort of traditional style business logic method like reserve credit which is doing two things sort of performing validation and mutating state has really just been split into two parts the command processing that's doing the verification the validation and it returns an event which then when it's applied does the state update so it's sort of the same kind of business logic but just split into two parts and the reason you need the apply method obviously is so that you can reinitialize the current state when you load the events from the database so you know normal kind of processing is you instantiate an object using its default constructor take the list of events basically fold it cooling apply over and over again with each event so here's some code so it actually it's not a it's not a complete pojo just yet kind of working on it so it actually implements reflective mutable command processing aggregate which is makes sense right it's an aggregate so it's got a got an apply event method it processes commands right it's mutable because it's doing an in-place state update as opposed to returning a copy and it's reflective because it's using reflection to look up the appropriate process and apply methods based on the type of the command or the event that you're processing so it's quite logical which reminded of the whole joke about spring right abstract factory decorator doodad commands class right but anyway so like here's the process method so the first one is really you know processing a create customer command it just returns back a customer created event so nothing much going on that the reserve credit command the method the process method the process is a reserved credit command is actually has some business logic right it's saying if the available credit is greater than or equal to the order total then we have successfully reserved credit or we can successfully reserve credits so we're going to return a customer credit reserved event otherwise we've exceeded the credit limit and we're going to return a customer credit credit limit exceeded event so it's conditional logic right so then there's supply methods that they're fairly straightforward they're just taking data out of the event an updating state so the one that's interesting is the customer credit reserved event where it's inserting an entry into the map to say this order has reserved this much credit the other one doesn't represent a state change it's just sort of a signaling that something important has happened which is not a state change so that that's so that's what an aggregate looks like and if this causes you distress just hold on a little while and I'll show you some nice functional code in a little bit here's a service that's creating an order and it's written in terms of this helper class aggregate repository and that has a save method that takes that says save it just says here's a command and under the covers what it's doing is actually instantiating a whole new order processing the command which yields events applying those events to do the state change and then saving those events in the event store and Ashley returns back the the updated aggregate i know i'm using RX java because Java except Java futures mostly suck so it's sort of react written in a somewhat reactive style so that that's what a service method looks up looks like this one's creating a new order and here's an example of an event handler it handles the order created event and finds the appropriate customer and Reserve and attempts to reserve the credit and it's actually gets past this event handler context that's sort of like an envelope that contains the event and it also provides an update method that says go find this customer with this ID and process this command and under the covers it's actually interacting with the event store loading the customer processing the command applying the events and saving the events back into the the event store oh and it's written using oh my gods spring so the at events subscriber annotation is actually picked up by a spring bean post processor which automatically registers that automatically subscribes it to the event store using the under using the event stores stomp API so you know it's this this this eight the style of API is using a bunch of reflection and one could easily imagine the non spring version sort of being more sort of you know you would write code rather than rely on this sort of reflective capability oh yeah and in the query side you know you really need to support many different kinds of views and so this particular view I'm storing a customer and their recent orders as a document in MongoDB so in order to display this that's just one primary key look in and you get a blob of Jason that can be returned through the rest api to the to the browser client so that's quite nice and you know here's an event handler that's processed that subscribing to all of the customer and order event and updating the corresponding document it's a bunch of details here I mean the actual code itself is implemented using spring data for mongodb which provides a nice high level abstraction over the database but you could imagine doing exactly the same thing with cassandra with neo4j with Redis whatever database makes sense for your particular kind of queries I've also heard the postgres outperforms when it comes to Jason so that might be another candidate as well just really funny ok so that's that let's look at the the nice immutable Scala version so it's functional in style so there's means there's a separation of concerns there's the immutable state representing the customer the credit limit and the credit reservations and then there's the customer aggregate that's got these process command methods that a process method that takes in a que ok she should take in a customer and a command and gives you back events and then an apply event method that takes a customer and gives you back a customer I don't know how many times I've stared at that slide and not notice that typo so there's some so it's implemented using type classes and there's a couple of sort of AB that which are represented by a couple of traits so there's a concept of an aggregate that knows how to you can create an instance of the the type the is also that's actually used by the event store to reconstruct the current state of an aggregate most note one of the interesting things is that a aggregate event is just defined as an abstract type member at this point and then it's got apply event methods and apply events that sort of that's doing the fold and then there's a an inter eight that extends this and adds command processing functionality the process command method and that once again that's got an abstract type member for that for the type of the command so I was not able to express that cleanly in the Java version or at least I didn't try hard enough to express it cleanly but some of these sort of more advanced features in scarlet mean that you can express the typing a lot better oh and the actual if the command processing fails i'm using a try so it's either an exception or it's a sequence of events so here's the actual module you know customer aggregate module so there's some business logic in here like for calculating credit and so on so just some basically functions in there that manipulate the customer state and here's the concrete type class so it's a command processing aggregate of tight customer defined the concrete types for the commands and the events you know concrete implementation of you know the new instance method that creates a blank customer and then the process method just relies on scarless lovely pattern matching capabilities so I don't need to sort of use reflection to look up to dispatch the command so that's quite nice and then likewise the apply event method relies on pattern matching as well to actually you know dispatch on the on the event type as well and because the state is a case class can just copy and modify it using the built-in copy method so it's actually kind of a lot cleaner than the Java version but it like important to remember that these are just different flavors of framework that are sitting on top of the same underlying platform so you get to pick whatever makes sort of make sense to you alright so that's the Jarvis Carla version remarkably I am almost on time just rarely happens so I want to finish up with just a really simple talk about another demo I'm excited about because it's visual a lot of this demos I've written just have a rest api and you actually have to interact them using coal this one has a fairly decent you i though this is not the best screenshot but the idea is it's a Kanban board and so you can create boards and then when you're in a board you can create tasks which you can then drag and drop between columns and what's particularly nice about this is that it's collaborative so multiple people could be viewing the same board and the application is actually subscribing to events that are coming out of the event store translating them into web sockets and pushing them down to the browser which is just kind of cool really so architect really looks like this you've got an angularjs application in the browser the server has a REST API for you know crud operations on boards and tasks basically and actually retrieving them as well and what's actually interesting that the server has a micro service architecture so even though here I've drawn it as a monolith is actually like five different services fronted by an API gateway so if you go go to the github repo you will see you all of this code and it's wonderful complexity it uses web sockets to push events from the events store in down to the browser it's using the eventuate events store to store the boards and the tasks and it's also using MongoDB as a materialized view is kind of a lot going on but the end result is looks actually quite decent so this is a little video I think I recorded this in my hotel room in Sweden so it's actually you know running on my laptop talking to an event store that's running in the AWS Northern California region and the performance was not bad I actually tried to do this on the flight home as well and United Wi-Fi has really high latency it actually uncovered a couple of bucks in the angular code that you would never ever have seen on a high performance network so let's see if we make this play so I'm just going to create a board whoa whoa this is cool you can hear me one if you can hear me cursed when I make a typo as well okay so I i created a board and it actually updated in two places and because this was on my laptop the screen wasn't that big enough so you can only see the top of the other board and then now that i'm going to drag it it's like hey presto the the other you can see the task is updating in the other browser as well and that's actually all these round trips between my hotel room and stockholm and this so and the server running in u.s. west one okay and then one other thing right yeah because you know nach one really natural consequence of using event sourcing is that you have the task history or any business entity you've got its complete history and so it makes it trivial to just show the history of each task in other words who did what and when because you just basically grab the events and then and instead of folding them to reconstruct the current state you can just display them like that so you get this sort of built-in audit logging which is super cool anyhow so that that's my talk so kind of in summary which may or may not represent entirely what I said today so microservices aren't a silver bullet but I think if you're building a large complex application they're generally your best choice for a bunch of reasons that I only kind of touched on you know you should really use you end up having to use an event-driven architecture in order to maintain data consistency across your services event sourcing is a really good way to implement an event-driven architecture and you you almost always have to use seek urs in conjunction with event sourcing because the event store itself is not very queryable so events plus C QRS and it's in a way is actually really really good because it gives you these highly highly tuned views that are designed to support queries very very efficiently and then just use a programming model that sort of fits your language and your framework so just you know pick the flavor of API that sort of suits the style in which in the language and the framework that you're used to programming it so that's my talk thanks for listening hope that you found it useful and you got my Twitter email and then if you want to know more there's that form you can fill in so thank we have questions can you review why is beneficially not your database and us it's worth painful surprise this should be the question oh yeah so this seems like an awful lot of complexity to solve the problem where the underlying root cause of the complexity is not sharing a database so you know I think one of the primary benefits of using micro servers is to go fast right so in other words you have your application consists of n microservices and ideally you want to be able to develop deploy and scale each of those microservices independently and in parallel right so that's the goal now the trouble that if they end up sharing a database so one immediate problem that you run into is that there's coupling right so imagine that say the customer let's say the order service access the customer table because it needed the credit limit that that would mean that in or if any time the customers that the team that was working on the customer service wanted to change their database schema they then have to go talk to the team that all of the teams that were accessing their tables right so that and that kind of communication and coordination that would result from that will slow you down quite quite a bit so that that's a so basically coupling is slows you down I mean you could argue you could say that this is continuum right but I mean I visited companies where they have lost track of who has been accessing each table right and then you can never change anything because you're afraid it's going to break right and then then on top of that you could you can easily end up in a situation where different services have very different data requirements right you know we've sort of moving away from the relational model to you know the one ring that rules them all to a polyglot persistence architecture where you know different database access requirements lead you to pick a different database so that's sort of the poly that that's the polyglot persistence argument the right tool for the right job yeah cool I'm available for consulting and training by the way the world one night was kind of late blooming something behind all the thing wasn't your ganization never to see the road one way oh yeah was it my intention to never say the word monolith no I mean like you know in another version of this talk I go into great depth about white a lot of the time the monolithic architecture is really good right I mean so much about it is really really simple the trouble is it just gets it's almost like its inevitability that you will get to a point where your monolith is a big ball of mud and your life as a developer will suck right but in the interest of time I skipped over that whole argument but it you know it really monolithic architecture is not an anti-pattern it is a perfectly legitimate choice but whether it's a good choice in your situation really does depend on the D on the specifics yeah so the first question talking about kind of decoupling and the complexity of decoupling how much code reuse on an average project do you look at in terms of your data models themselves like i'm sure there's like a normal pattern is to have data models as a library that depended on little things but like how do you wear you know to cut off yeah i got to remember the question um ya know it sort of like whatever what a what's the level of reuse the of data models that you could expect between services yeah that's really interesting I mean they sort of at a higher level I mean I another topic I didn't go into right but a real another really key thing to think about is the concept of a domain driven design bounded context right so what that effectively means is that you know in practice well we like to think about having an enterprise-wide model right where there is a universal view of a customer now if you try that your customer table will have hundreds of attributes right and we're and it's really complex and in practice different parts of you organized if it's even just take something as simple as a customer different parts of the organization will have a very different view of what a customer is right like you know like accounting or has had some financial view of a customer so that's all to do with money whereas customer support has a sort of you know a support related view of a customer there you know basically a collection of tickets right and then marketing will have a another view and sales will have another view and yeah they're going to have some attributes in common like the primary key and the the name but they're likely to be wildly divergent anyway so that you know that those are all examples of like bounded context and so you know that which is a really important kind of mac tool for designing your services my light I mean you know the book domain-driven design it came out in 2003 that book is fantastic 13 years later it's full of advice on how to design basically microservice architectures right he would have thought because it's not only like strategic ddd like bounded context but also the tactical patterns like aggregate are absolutely asset which I kind of threw sprinkled throughout my talk but aggregates are a really essential part of designing business logic for in a micro service architecture I mean those are the things that are persisted in the event store so go buy and read ports that just read it as well not just buy it domain driven design and then the implementing domain driven design book as well I can't see you behind a pillar I Alliance Event like record store yeah Oh what do you think I'm sorry but what do you think about the possibility on the new side from having persistent references to previous versions of other resources or sick available if that is a yeah so so the sort of the question is what about on the view side actually sort of storing historical state and exposing that that kind of yeah so I mean I guess the answer is yes if if it makes sense in your application and that can be a valuable benefit right because the view side is from its perspective it's seeing a stream of events so for a given aggregate at seeing a stream of a stream of state changes and so it could very well store some representation of the currents of of historical state as well as say the current state as well and you know and thatthat's that's a perfectly legitimate use case I see it back when success things yeah more or less oh so yeah so what do you do to deal with performance and fault tolerance and an event sourcing they are my rough approximation to feel question so I think one of the key performance things is we take something like a order right that has a very sort of limited life cycle you know the let's say there's just ten events during the lifetime of an order because it gets created and then it shipped and maybe it's returned but that's about it whereas if you have a bank account and each debit and credit is an event you could very well have a bank account for 20 years which would be an awful lot of debits and credits and so loading all of those events and replaying them would be quite expensive and so the solution there is is to periodically snapshot so let's say every and events or every n seconds you take a snapshot of the current state of the business object then when it's time to reload it you just find the most recent snapshot and find the events that occurred after that snapshot and basically fold them together so you never need to go back and process 20 years worth of events oh hell yeah let me just finish up by it we can talk offline so yeah the reliability is yeah you want to have a super reliable messaging infrastructure so that the look that look guarantees you know where has to guarantee at least once delivery so that's my answer in a nut and I and I have to get off this virtual stage now