reactive James Roper 4 21 16

okay so thank you for having me yes my name is James I work as the tech lead of both play framework and logon and logon is a new micro services framework they were going to see a bit of tonight but actually tonight is not just about logon it's it's really about architecting microservices so it applies to a much broader scope of things that you might do not just one technology of course vlog on is a very good way to do it which is why we wrote it so let's get on to what we're going to look at tonight we're going to first look at tonight what is asynchronous communication asynchronous communication is a very broad term which can mean a lot of different things but I have a specific definition of it that I want to talk about we're going to compare asynchronous and synchronous architectures how they have a form together how they might be better than one might be better than the other and finally we're going to look at asynchronous communication in practice this is where you get to see some of the concepts that I'm talking about tonight applied in log on and it can be applied to you any application so asynchronous communication what is it so the first thing that I want to say about it is that I'm not talking about asynchronous i/o I think in a silo sure is is a form of asynchronous communication if you brought that define it very broadly but I'm more referring to communication between services so i think in Sao is good okay i'm not just because i'm not talking about asynchronous i/o doesn't mean it's bad in fact lagaan uses asynchronous i/o at a very fundamental level but it's not what this presentation is about so just by way of contrast asynchronous i/o it means don't consume threads while waiting for things to happen so his code in Java right we have here something we're making a request that request returns a future of response or a completion stage for those of you that haven't come across completion stage before it's java's in a way that only Java can do very long way of say in the future or promise you might have come across futures in scholar or promises in JavaScript the it's it's a different thing from Java util concurrent future but it represents a value that is not available yet it's going to be available sometime in the future so if we want to do something to this response it's not available yet we can't we can't do it instead what we do is we attach a call back to it and so you can see when I want to decode this response that I've got into you know some sort of model object my own model I attach this callback using the then apply method which is another Java way of saying map and and that callback won't be executed immediately because the value is not there after attaching that callback the thread that's doing this will go and return to the thread pool or go and do something else and then whether when the value of the response of the future is is redeemed a thread will be pulled out of a thread pool somewhere and will execute that decode function so asynchronous i/o is about not blocking threads asynchronous communication in contrast is about not blocking requests and what do I mean by that I mean a service if a service is processing your request it should not be blocked on another service having to process another request in order to return its response so let's have a look at what this means in practice so we've got a user users happy scoop when the user starts off happy user makes a request to service a i'm using the bi directional arrow there to say that it's synchronous communication so you have a request going one way and a response go on the other now service a actually doesn't have all the data it needs to to be able to to fulfill this request that has been made on it so it has to call out to it might have to a call out to service be making a synchronous request so the request goes into a and then another request goes to be it processes something and sends a response back to a and then i can send the response we might have more be might need to talk to see during its processing they might need to talk to d i might have to go to multiple systems and he might have to go to d as well and as long as everything is is happy here as long as all these systems are running and performant and and working then our user will remain happy they'll get their their response but what happens if service d blows up spontaneous server combustion well service see an a there which depend on it they're still running okay these are these are different servers they have their different different services probably running on different physical machines there's no reason why just because service d has failed that they should also fail now if they're using asynchronous i/o then there probably any requests that are building up on them they're probably be going to be able to manage fine because they're not going to run out of threads however the thing is is that service see needed to make a request on service t in order to return its own response so it can no longer do that so it's going to return an arrow to so service C and E they may as well be down they just appear to be done as well similarly bna they're effectively down I mean sure they're actually running they can receive a request but they're not going to be able to send the responses because they depend on these systems downstream that are also unable to send responses and so consequently our user is unhappy because they get an error so synchronous communication failure Babel bubbles back up the synchronous communication pathways so we saw that as you as you communicate synchronously along one direction back the other direction the failure failure goes not only that the slow response times bubble back up if service see for example was slow at processing the response maybe it was a bit overloaded its database was slow whatever reason that's going to impact how the maximum speed that I can return its response because it's dependent on that service if you really embrace synchronous communication you may end up in a situation where some services will be needed by everything a typical example of this is let's say you've got an authorization service so in a social network you might have a service that managers friendship relationships and I can only view your post if if i'ma friends with you I can only comment on a post if I'm a friend I macol only like a post if I'm a friend and and so it turns out every single action that happens with the system it needs to go to this friend service to check this friendship relationship and so what that means first of all though the friendship service there would be a bottleneck to load if it can't if the whole system is only going to be able to respond and handle as much load as it can as it can handle and it's also going to be a single point of failure if it goes down everything's going to go again I can no longer like things post on things even view things because it's always being used to check these authorization assertions and often in a situation where you really in both embrace synchronous communication you'll have a situation where the whole system needs to be up simply to reserve to serve any requests and when you've got a system that looks like this what you've really got is something that looks no different and behaves no differently to a monolith so let's say you've taken your monolith and you're broken up into micro services or you might have started a new project and so we're going to go microservices not a monolith if you use synchronous communication everywhere sure your your code bases might be independent they might be able to be deployed independently they might be able to use different technologies but when it comes to how they behave they're just looking no different from a monolith so how does asynchronous communication help this we've got a user user makes a request on server say and so the sale returns the response and that's all that's needed doesn't matter if anything else is down because ever say is there but hang on didn't we say that server side doesn't have everything that it needs to to to serve the request it needs so we might have a service over here so this day and some things happen it has its responding to some sort of external events coming in and when it receives those events it's going to push them asynchronously to services see any and likewise they're going to push to be and beat away and and so a is going to be presented with all the data it needs from from service DCAM be via these asynchronous pathways so it can always have everything it needs in order to respond to the user but what so what happens now when service d goes down well services see any they don't they don't actually ever need to call directly onto service d there they they're still up okay there if they are handling synchronous request themselves they're fine because they have everything they need likewise service p is still up and service a is still able to respond to that request so with asynchronous communication failure stops at the failed component slow response times like ways to stop at the slow component I mean if service see was slow responding to the events that it was receiving well sure might take a while then for its events to get through but but service be isn't going to be slowed down and services is is not going to be slowed down either it's still going to be able to respond just as fast to the requests that it gets we removed the bottlenecks and single points of failure so that service D doesn't matter if it goes down to the other services consistency may lag and this is the big problem because now if likes let's say service see goes down the events pushed by service d are not going to go through then eventually make it to the front but we can use some eventual consistency patterns to be able to address that and that's we're going to spend a lot of time looking at that this evening but the big point here is the components are isolated the reason why we do microservices is to isolate the components from each other we want to create boundaries between them boundaries for technology boundaries for deployment and organizational boundaries but also boundaries for scaling boundaries for load boundaries for failure so that so that our system can can behave and and be resilient to these things happen so what does this have to do with rest ok so the talks title Brie thinking rest I should first point out that when i say rest there there are certain people and and I don't come across them that often but they certainly do exist that that went very strongly to point out that that rest is bigger than just HTTP and that's true in the original paper that was written about rest but when you say rest to people today ninety-nine percent of developers here HTTP and Jason or maybe XML so when I say rester I'm referring to what ninety-nine percent of people understand it to mean which is HTTP so rest in that context is a synchronous messaging protocol so the client sends a request and the server sends a response it can be used to do asynchronous messaging ok any synchronous messaging protocol can still be used to do asynchronous messaging and HTTP actually has built into the protocol a way to express that semantically the 204 no content response so if you send a message to it to a server via HTTP it can specifically say hey I've got nothing back for you and that's asynchronous messaging if we were in the when we did our code before where we made that wrist and we've got our completion stage of a response back if we were doing asynchronous messaging they're using HTTP we would simply do nothing with that with that completion stage we just let it do whatever let it be redeemed or not redeemed or whatever so rest is not really the problem this is not a presentation about how you all need to stop using rest there are plenty of good places where rest can and should be used especially at the entry points into the system where where browsers going to into the system but even internally between services there are plenty good places where rest can still be used but so I've written a social network it's fairly small at the moment only has two users but it is one going one day going to overtake Twitter trust me on that now in the first version of this of this social network it was a monolith okay I just wanted to get things done as quickly as possible that the social network by the way is called chipper and so so I had this model live called chipper and then I went to a conference and I went to a presentation and the presentation said that monoliths are bad and and so on oh I've got this bad monolith and so I had to go and convert it to microservices so I just want to say now this talk isn't about why you should move to microservices there are plenty of resources out there that you can talk so that you can go to there are plenty of blog posts I'm not really talking about why to go to microservices I'm just talking more about how this evening so we're just going to assume that whenever we have a model if we want to go to microservices so this is a rough diagram of what my monolith looked like and you can see I had three components in there was it was a spring app they were wired together by spring so I had this friend service component ship service like service chip service needs the friend service because when you go and get a list of chips for the users that you've follow you need to get the hypno to get the list of friends at the chip so this needs like service because when it returns a chirp at once to return in that shirt a always account of the people who would like it so that that could be kind of rendered on the screen and the like service needed the friend service because it needs to check that I'm friends were if someone before they can like it so the first step to to converting this to microservices is get rid of the monolith so I did that but then we've got these war components out there that need to live somewhere so let's turn them into their own micro services so we've got friends service microservers Chipmunks service like micro service and then I've got all these things that used to be method calls so these were all in the same jvm before but now they're running on different servers they can't just be method calls what am I going to do with these communication pathways well let's just replace them with rest easy and so I've converted my system to to micro services and micro servers starts delivering everything that it promises and chirp it becomes really popular and so after a little while the the number of users increases and the friends service starts getting a bit overwhelmed and it crashes so so friend servers stopped working but it's micra services so so okay that part of the application doesn't work but I still got the chip service and like service but because they're synchronously calling the friend servers they're effectively down and so effectively my whole system is down and the users not happy and if your user can't access their their social network well we know what happens so when moving from a mic in front of mana left to micro services the problem is not rest the problem is that rest is our go-to it's the way we use rest the way we say well here we've we've got some method calls and and with this system needs something from that system so let's let's just use rest to get it so when we move to microservices we need to make sure that we don't just do a one to one service interface replacement we really need to rethink our architecture and design for an asynchronous of architecture so what does this look like in practice distributed systems and during a synchronous communication is is a communication is a huge topic there's no way that I could cover every single technique and practice that you might use in order to replace synchronous communication pathways with with asynchronous communication but i will look at two methods this evening so the first third that i'm going to look at is called de normalizing push data and that's that's the method that we that I we saw well that would be what you do if we were doing it the way with a diagram before with a talking to be talking to sea dog in today another approach that oh I'll cover just a little bit is communicating through the through the client this actually isn't a synchronous it's more not synchronous Saudi normal eyes and push if you've if you've ever used a relational database you'll probably know a bit about normalization a normalized data schema has a number of properties and one of those properties is that data only appears like any piece of data will only appear once somewhere in your database you won't have it repeated multiple times you'll have you'll have references between the data that are enforced to ensure that you don't have stale references and things like that as well when you d normalizing push the idea is that services will store all the data that they need and what this might mean is duplicating some of that data so this is D normalizing it because a normalized system would have the data only once but if you're replicating it then if you're duplicating it then you've just been normalized because of this consistency needs to be addressed so the data can get out of sync especially in that in the happy case things might work pretty well but with something goes down if something gets over loaded then this data is then you will have problems where data is going to get out of sync it will ask 11 service a question ask another service the same question and they give you different answers one approach that we're going to look at quite in detail this evening to solve this is to use at least once messaging and Adam potency and if you don't know what item ponen see is the idea is is you can have multiple the same operation can be performed multiple times on your data and the state will stay the same regardless of its whether it happens once or multiple times so you know I mentioned before that the the church service had to call in to the like service to get the count of likes so this is currently implemented like this so the user goes and gets the chips and then the chip service goes and gets the likes for the trips and then returns the chips with the likes to the user and so we know if that means if the light service goes down the church service isn't going to be able to return the like spoke to the chirps back to the user so let's get rid of that synchronous communication pathway there and we'll replace it with an asynchronous communication what we're actually going to do is every time the the likes somebody likes something we'll send a count of likes for that trip to the church service and it can respond to that and store that count of likes for the for the chirp so now when the like service goes down the church service is still going to be able to respond to the request is coming into it so now we'll get into the demo chipper has been implemented using log on it's actually a sample app that I'm that you can get from the logon repository so you can go and try this out yourself logon is is a micro services framework for the for the jvm now the word logon is a Swedish word don't listen to my pronunciation of it because I'm not Swedish you'll get it very wrong if you try to replicate my pronunciation but what it means is just right or sufficient and it's more than just a word meaning just write a sufficient of them and civ to the to the Swedes it's more of a cultural thing a way of life for a way of just having things that are just the right size not too big not too small and so when when we're coming up with a name for log on when one problem with the word microservices is that it's kind of like services with micros and micros the feature word there which is means you know very small and people have often taken this too far as they say you know you should make sure that your services as small as possible and the smaller the better the system that you're writing but of course that's not what Mike services about much services are about having the right boundaries between the services so they can implement that the requirements that they have whether it's transactional requirements or scaling requirements or for resiliency requirements the the service should be sized to be the right size to meet you two to meet your isolation needs so logo and fit that really well it has first-class support for asynchronous messaging and that's obviously going to help us a lot this evening it also provides a high productivity and development environment so if you if you've used to play before you'll know that when you change some code in play and you hit refresh on the browser it will play will pick up that that change and reload your app and you'll see it immediately with log on it's doing that not just for one service but it will run all of your services and you make a change and it will and will reload all you all of your services that depend on that that code change and finally it guides you all the way to production so starting with them so I mean one of the complex things about microservices is that you're now not just deploying one system you're deploying many systems and you have to to manage the configuration of those systems how they talk to each other how they know about each other where their databases is is how communication from the outside world gets in and Logan gives you that through the use of luck and conductor currently there's only a java api we we found when talking to our customers that the questions that our java customers were were asking us were more were very often about well how do i do microservices how do i get stuff that gets started one of the best practices where's our scholar customers didn't tend to be asking those questions so much and so we decided well for the first iteration will will just implement the Java API will get some feedback will understand whether it's me in people's requirements or not and then we'll release a scholar API we've started working on the Scarlet API already and if you go to issue number one in the log on in the leg on github repository you'll see its add a scholar API and so it's the first thing we're pretty much doing after releasing the after we make a general availability release so let's get into this demo so here i have my terminal I'm um I've started sbts so like our muses SBT is a build tool and development environment and I'm going to run this command run all so when I run and run all the first thing that you see happening there is it's going to start the Cassandra server so low Tom actually runs your database for you you don't need to set that up and so I so running a service locator and gateway the service locator is how services find each other the Gateway is how requests from the outside for example from the browser will get to the services and then it's starting each of my services you can see they're all running on different ports I don't need to worry about what port they're running on because the Gateway knows what port they're running on and we'll we'll wrap the requests accordingly so let's try it out so here I am inside chipper I can write a post I can view likes I'm logged in as Bob at the moment I can like something i can unlike something you can see that i'm you can see that summary of lights there that the church service has to get from the lack service and so it's all working it's all good I won't show you that many other features because you'll very quickly see that it's really not a very feature full social network so going to beat Twitter Sunday so what happens if our like service goes down so you can see here I'm not going to go into this putt in too much details but this is where we actually get the chip stream as using our streams here we've got some recent trips which we've loaded from the database and then we're concatenating them with all the chips that they come through the system and a published using using acha distributed pub/sub so this is this forms the stream of of our chips that we're going to load and then here for each chirp we're doing this map async call and that's going to invoke this update chat blacks rich chip you can see here for each job it's going to go to the like service it's going to invoke it to get the bikes for the trip and then that returns the list of likers and then we can update the trip to to have the right number of lichens there so let's say somebody accidentally in the configuration file sets kill equals true and legume which is watching the file system automatically picks it up reloads and so then if we hit refresh here you can't see us power feed anymore so the like service went down but because it went down we can't see our chips even though the trip service is still up you can see exceptions in the logs here because it was unable to access hope we can see it says that was killed okay so how do we fix that well century cause a killer cause false is probably a good start but more long-term we want to ensure that killing it doesn't doesn't cause that problem so I've done some work already here just in preparation here's our like service interface this is what a service interface looks like in larem you can define multiple service calls and then you define a descriptor down here which will describe how those service calls are addressed and invoked so here I've got accounts service call service call takes three parameters the first parameter is the is an ID and this is what this is something that can be extracted from the you are right typically so we're going to pass in a uuid in the ER I it's optional and that's going to be the ID of the last a bit of of the starting event that we want to the counts front and we'll learn a bit more about that later then the request is not used because in this getting the stream we don't need to send a request message and then the third parameter is the response message and we're saying the response message is actually at an extreme sauce so it's a stream of messages and each of those messages will be this likes message if we have a look that's got three fields first a chirp IDs so that's the chap being liked then likes which is the count of likes and finally the ID of the event that that this is and that idea is is also going to be used in combination with this year whitey so at this point I just like to talk a bit more about why I'm doing this and what I'm doing here there would be a much easier way to to implement this and that's to you something like a patchy Kafka kapatic Africa gives at least once messaging and and so I can use that to achieve that least ones messaging part of of my requirements I'm not using that for two reasons one is that our integration with Apache cough gurren lagann isn't there yet it's possible to use if you drop down to lower level api's but um we're going to provide a really nice API to use that but that gives me an opportunity to to actually show you how at least once messaging gets implemented and so we're going to have a look this evening basically what we do is exactly the same thing that Kafka itself does so I think that's important to understand because it's important to understand why your system scales how is it that Apache Kafka is able to to scare out to too many nodes and and how is it able to do what it does so what you'll see me implement this evening is a lot more complex than it needs to be but it will give you a good understanding of the underlying concepts behind what we're doing so let's go to the implementation of this caps service call and you can see it hasn't been implemented yet service calls in logon generally get implemented using Landers so we take this is a lambda that takes that ID the offset and the request which we're not using so we're not going to use that parameter but on this offset is going to be what the the service that consumes this is call this stream will keep a track of the ID of the last event that it successfully processed and so when it elles to what so when it needs to recap connect for whatever reason it can send that that Heidi the as the offset and so then it will only get the events that happened since then so the first thing we want to do we're going to return something immediately so we just returned a completed feature then we're going to get the event stream for all the like events now logon uses event sourcing out of the box in contrast to say a traditional relational database system where you store your current state in the database when you use event sourcing you don't store the current state you store all the events that occurred and then by reading those events and have an event handlers for those you can compute the current state and this has many properties such as been able to do temporal queries being able to use a really scalable database because it clearly has to support one operation append but in this case it's very useful because it means that we can just do a query on the stream of events and be able to present those events to other services so we're going to get that stream of events you can see we were asking for all their beds from the offset and then for each of it we're going to have to map it now each event that the like services is storing is something like on this user liked this chip and this user unlike this trip and this fuser like this chip but what we want to send to the service is the current counts of luxe which obviously from those events can't be determined so when this when we get these events we're going to have to look up the current counts and so that's why we're doing this map the two there just so that says we want to do to maximum to to look up at a time and so for that prevent where we'll go and get the the like entity like entity is the the list of users that liked the event for for the chip so the event itself that the event that's been sent here that the class is actually a pair and the first element in the pair is the like event which we can get the chair by D from the second element in put in the pair is the event ID and that's used for our offset handling so we're going to ask it for the likes and then we're going to create a new likes object with a chirp ID the number of likers and that event ID and so we've implemented the the like server side of things now the chubb service need to consume that so in the chip service I've done a little bit of work ahead of time I've created I've added this field the likes field to the chip table this is actually Cassandra code I've added this light-filled to the chips table and so whenever we get these this likes event we're going to do this update will the update the number of likes now this point I hope you can see that that this is an idempotent handler here because if we get an event saying that 10 users liked this chip then we'll set it to 10 if we get the same event again we'll set it to turn again or if we get an event saying that 10 users and then for some reason we process a whole stack of events again so we get 8 users and as long as they come again in order 910 will end up eventually in the in the rec'd state so that's unimportant event handling and the reason why that's needed is you can see here let's say we execute this right on it and it succeeds and then afterwards here we insert the offset into the table so this is the idea of event that we last successfully processed but if that fails or if the database goes down between those two things then we're going to have process of the in event but not storage the fact that we processed it and so when we go and and load the event streams will end up getting that same event if again and so will end up processing it twice and that's why the item potency is really important so I've also added that something here to go and get the the current like subset which we need to to stand our stream so in setting up this stream I'm going to use an actor to do that this actor when it starts is going to make the connection and it's going to start doing the updates so the first thing I need to do is get the likes offset and then I'm going to invoke that count stream on the like service passing in in the offset now when it's complete when when I get the the stream back I could actually have actually gotten narrow it may have failed to get the stream at all like maybe the service wasn't even there and so what I'm going to do when when that happens is I'm going to fail actors when something goes wrong the best practice is to simply fail and crash and let your supervisor handle how to how to deal with that so we're going to do the supervisor a bit later but for now we're just going to to tell ourselves this failure message and you can see where we handle this failure message we're going to with simply throw an exception which will cause the actor to crash but assuming it's successful then we're going to take the the subscription the count stream and for each item we're going to invoke that update trip likes method that we just implemented and so now they're handled and so we can run our stream but with each message sorry not that one each message we're actually going to feed it into a sink that simply points to ourselves so by doing that when the stream itself fails it will also send this valley a message to us and we'll crash and so that means if the stream fails we can then let our supervisor handle that and this message here is saying this parameter hughes is what gets sent when the stream finishes gracefully and again we're seeing the failure to ourselves so that we crash so all this crushing how are we going to handle it logon is built on Play and Play uses juice to wire everything together so if we want to set something up that happens at the start of the application we have to do that with a juice module and so here we're binding our like count subscriber actor as an eagle's egg at singleton using juice so that's going to start it when the application starts up and we're using a provider to create the actor when you create an actor you don't immediately you don't you it's not you that creates the actor you have to let the actor system create them after because the actor system is responsible for hanging lifecycle so what you do is you create these prop subjects which describe how to create the actor and then you can give that to the active system and the actor system will do that will handle that for you so we're going to create something that just describes how to create create a like accounts subscriber and then we're going to wrap that into in a supervisor that's going to do an exponential back-off and this is where our failure handling happens so whenever our actor crushes the back off supervisor is going to after three seconds try and restart it if that fails it will do it after six seconds and after 12 seconds uses this zero point to this random jitter so that means the feed have multiple actors during this they're not all trying to reconnect at the same time and it it limits it to to a maximum of 30 seconds so once it reaches a 30 second reconnect time it just stays at 30 every 30 seconds and the final thing we're going to do we haven't implemented any sort of partitioning of our events if you use apache kafka it does implement partitioning but we could have actually implemented it ourselves where you divide the events up into inter partitions and H petition get some okay then has its own offset and this would mean that you could subscribe to the stream from multiple nodes each node subscribes to a different partition but because we haven't done that we've just kept a single we only want to subscribe to these events from one node because there's no point in having multiple nodes process these events so consequently we're using something called the cluster singleton pattern in in Schuyler clustering it's anarcho clustering and it's going to ensure that our active only ones on one node and so there we have it the next thing that we need to do is is just remove that synchronous call that we had here because that's now no longer needed and so if we wait for that to do all the reloading we can then go back and the like service is running now so this should be working so there's our whole axe and we'll see if it works so I've just liked that I'm going to hit refresh you can see it still says 0 likes now as it happens the way that I'm events awesome workers another is it actually for Cassandra puts in a 10 seconds delay before it actually propagates the events and the reason for that is because Cassandra itself may not be consistent so in order to ensure that all the events coming order it it and for the for consented Cassandra to reach a consistent state it waits 10 seconds we probably actually don't need that because of the way that I'm we don't care about two different entities their events getting out of order because we're only handling the events on on a per entity basis but I left you can actually configure that delay to be short or even zero but I left it in there to demonstrate that we actually doing this asynchronously because now if I hit refresh you can see that's now one likes so it's definitely happening asynchronously now and so of course we didn't just do it so that it happened asynchronously we actually want to implement to see that resilience Lee so let's kill our like service for a load and now if we hit refresh you can see that our stream is still working we can still see our our chips of course alack service is down we quick here we can't actually see the users that liked it but try and click like has no impact so our like service is down but our chip service is still able to respond to our requests to get the chips ever actually have a look here you can see it's that killed message and our backoff factor is actually try to recap reconnect quite a number of times you can see stream power of restarting their stream very stunning so just to to reiterate again about the about using this what we've we've done there we've just gone and implemented our own at least once messaging you can see because we're keeping track in the consuming service it's it's really quite a simple mechanism of Kevin track of that offset and so we could have used Apache Kafka there and then we all we'd have to do then is on the trips publish those events into Kafka from the like service and the church service set up that that that's subscription and it would and when we provide this in logo metal will be literally a one line of code changed to do that but what we've seen this evening is is exactly how these that's basically how Kafka works underneath the client store was an offset to say what it's processed so let's move to communicate through the client now we're going to look at really trivial example the client can cache data it might catch up for just a small amount of time just to reduce some of the load or could cash it for a long time might cash it at the start of the session so that might be minutes or could be like a mobile client if you've ever so with Twitter caches advertised for example and on the mobile client because they don't change that often the browser's can also catch things local storage is a particularly nice way to do that you can also use the cash as a fallback so you could try and make the request to the to this service but if it fails or if it doesn't return within a given time then you just fall back to you your cash and one case where we use in synchronous communication where we could replace it with with communicating through the client is in the activity feed so currently we go synchronously to the Activity Feed that's fine but in order to return the chirps the activity feed has to go to the friend service to know which chips to which friends beds to get and then it goes and gets those feeds and returns them to us so friends service goes down the cavity feed can't can't fulfill its request and we want to get rid of that so let's get rid of that call to the activity stratum and instead just go directly to the church service so now if a friend's service goes down tivity servers can't feel the request but that's alright because we're not using it so just a quick demo of of what this looks like if we kill our friend service actually I just start and kill the like service first all rights that look okay so we can see the feed still working so if I go now and kill the French service if I go back to the feed we don't get our chips anymore so let's uncurl that first we can see in our JavaScript here we're using react there are two different calls to get the stream one is to go to the trip service can get the stream passing a list of user IDs we can't we just pass one user ID when we want to view their stream for one user and the other is to go to the activity stream for for a given user and that get the chips for all the friends of that user we're going to create a new one that goes to the church service and passes all our friends so we create on your method that takes all the friends I DS we do that service call and then we send the list of user IDs that we want to get the their chirps for and now we can use that in our tivity stream page so we're loading that user and the list of friends for the user from local storage which we've cached and then create friends stream with the friend no DS so let's make sure it lets reloaded and recompile that looks good so if we hit refresh here looks like it's working let's now try and kill that friend service again wait for a reload go to add friend go back to feed and we can see offering out we've still got our feed we're unable to actually load the actual names of the users were just getting the user IDs there because the friend service is down but at least was still was still able to read our chips so that was a really trivial example what about something that's a bit more complex and I didn't implement this because it would be a fair but it'd probably be a bit too much to do in a presentation but another example is client authorization so I mentioned earlier that requests can go that if you have like let's say an authorization service with it with it that we need to go to the Friends Service and that can cause a real problem so what if we communicate the authorization through the client but how can that be possible I mean it's the client that we want to kind of protect our servers from how could the client can't just go hey I'm allowed to do this let me do it well the answer is to use authorization tokens and specifically for something like JSON web tokens so Jason web token is a bit of Jason that then has a signature associated with it and what that signature that's that signature will be signed maybe with a shared secret maybe with some sort of those asymmetric encryption and this and so when when the user loads when the browser loads an entity for example when it gets the current user and it'll get the list of the use of friends that friend service can then send tokens for each friend and say this user is friends with this with this user and he a signature to prove it so then when the client needs to do some sort of action it needs to like a chirp or needs to post a comment or something like that it's going to send that token with the action perhaps in a header and then the service that receives that can say okay well first of all I verify the signature so I know that the that this was that this token was produced by the Friends Service and not just made up by the client and then this action this person needs to be friends with this person and this token is saying that this person who's friends with that person so all out and so by doing that we've now reduced a huge amount of load on our friend service and we've ensured that our friend service is not going to implement is not going to affect whether our our whether it's up or down is not going to affect whether it be say the like service is able to accept alike or whether the trip service is able to return a chirp so this evening we've looked at at a few different things first of all we've learnt that rest although it's not a bad thing if we if we use it just to replace method calls without really thinking that it can cause some problems we don't really realize the full value that microservices can bring us so and we've we've looked at two different ways to to remove the use of the reliance and synchronous communication one ben de normalizing your data and pushing it and the other being avoiding synchronous communication by sending things through the client so as a next step you could check out log on so like been calm / logon is the current website it has all the information there's four screencasts on there as well as a very nice looking video to it to introduce you to log on another thing that i really recommend doing is yonas bernie is one of the founders of light Bend and the author of vodka has written a mini book called reactive services architecture it's only about 30 pages long I think and and you can read it in an hour it's free if you go to the light Bend come website / reactive dash mark services that data architecture you can get a free copy of this mini book and it explains everything a lot of the concepts that I've talked about tonight plus a lot more so I really recommend reading that and if you want to check out this presentation github.com / gyro / / rethinking dash rest you'll find the slides you'll also fly into the code branches for the initial state before the presentation ND and the eventual state and you'll also find the IntelliJ live templates for for doing the live coding so thank you for for having me here and I'll take questions yep yep now Cassandra is our data stores it's not been used as a cash it's the database and that's what we're encouraging people to use or that's the default database that we support in laganas Cassandra you could do that but it you know in the case of of the I mean it's all the trips have been stored in there although all of our like entities herb and store stored in there and it's not being used as a cash-in in this this application it's been used as as the database the data store no actually we've got some t-shirts does and we're going to give them out to people that ask questions um I was a life in person that i can send to fetch them John they're just on the bench there yep join a t-shirt now they're all excel that's all we've got left unfortunately but who else wanted to ask put carefully the data cache is going to be what was asked for in case of push back the first one how do you are you pushing everything every life okay so a question is in the push in the pushing scenario are we pushing everything so yes we're pushing every like i think i'm guessing the the more underlying question is isn't that kind of mean that you're pushing a lot of data everywhere and having to store a lot of data everywhere is that everything yeah yes so I i mentioned that that there are many different techniques that can be used to get rid of synchronous communication and their d normalizing push is is one that works in some situations and not not others so in this case it worked really well because all we had to do was store it and an extra integer on on each chip if you wanted to store for example if if you want to store the if you wanted to not need to go to the friend service in order to get user information for each chip storing say the current user user's name and linked to the avatar whatever on its trip that would be a bit ridiculous because every time they updated you'd have to go through an update their entire history of chips so you wouldn't use that there there are other another approach is to it may most in that case what you might do the activity feed is something with a limited number of chips so this is I don't work at Twitter or Facebook but I'm pretty sure this is what they do to implement the feeds is they have a materialized view of the of of everything that for the past say week maybe your core two weeks or up to a certain count of of tweets or posts and so there's only a limited amount of data in there so when so that can receive their events and it does make sense sometimes they're too to put a bit more data there because you don't it doesn't mean that you're going to have to update years of data so that's another thing is to create new services that's in front that hold a view that's just specific to to certain use cases at the front and sometimes it may make sense to to have a service that goes and another way would be to actually use caching so if you had a service that's responsibility was was receiving the tweets and getting the user data and cashing it and maybe it might subscribe to updates for the cash two years of data there might be some partitioning of things so but yeah there's there's there's lots of different ways of of implementing these things and that denormalize in the push is one interesting way they were working many use cases but not for all yep yeah so like Ben has enough conductor right look here's that we're just conduct our conductor to rift what sort of tools ways it is a life ended all so so conductor has integration with a few different container management systems I understand we're currently adding meets us for support we have integration for Joker and AWS and so conduct is able to manage a few different things there so it i guess provides the higher level side of things and builds on top of some of those other tools i'm probably i think Ed's here is ed Ed's waving his hand he he's on the conductor team so if you've got one a better answer asking yep so compared to the monolith limitation how much more complex is this sort of implementation terms of just simple Atlanta lines of code and then all the things you have to think about all the different yeah and then as the number of services grows is it like linear yeah so the question is compared to an monolith how much more complex is is this kind of implementation where the micro services and does it grow linearly or or or as the system grows it is definitely more complex than a monolith microservices and distributed systems in general are more complex now one of the one of the points in one of your nurses presentation study that he makes is that um databases like relational databases actually present you a lie they they want you to they present your review of the world where everything is consistent which is fine when when you don't need to scale and when you don't need resiliency but as soon as you start to do things across multiple nodes that that becomes a lie and and either you end up with a system that is is that that is going to crash all the time we end up with a system that can't scale or both and and you went up in with you you basically can't do the things that you would usually do in a model if you want to scale so um so yes it's more complex but the complex complexity is is necessary and the key to it is managing the complexity now as far as as your system grows how does that complexity increase and I'd say when you are using these these patterns will the complexity I guess just compresses linearly as your system increases so of course every new component adds that much more complexity but because you all your components are late 'add from each other they don't really add they you know you're just passing events between them you're not putting new interfaces where things need to be up things and you don't know what's talking to what this this I think as long as you don't right we'll we'll it will help you ensure that that complexity stays the same where contrast our model if as things start to grow you start need to tend to do a lot more caching is starting to do clusterings don't need to worry about things being consistent between those and and things become very complex that it very complex very quickly without actually gaining the the scalability that you need next reims so so sorry was a question not do I any connection between la DOM and specifically events or seen in darkest dreams so so when in the implementation of the like server so i asked for the stream of events and it was a nucca stream so any streaming that's done in my gham is is done with with darkest dreams so you can get and like a stream of events directly from from the event source database in lid on gives you that so it's I mean this there's a very tight connection there and it's um quite a useful use case of vaca streams there I did was that with a question or yep yes sir I mean on that same topic we log on provides provides the event sourcing but to query that the data there's it uses see QRS so II then define read source processes and these are completely separate from your your right side of it persistence and so whether that's a red side process of restoring some sort of business data that needs to be returned to the client or whether it's going on out to some sort of analytics you know when it put in a data store somewhere maybe been to be processed by spark is it doesn't really matter that's it is completely disconnected from the events coming in and being assisted in that state being updated so but um that um that side of it you don't really need to see yucca streams there because Logan currently provides that I guess a fairly opinionated API which is more specifically designed for ctors than the next Ramses but of course you could do exactly what I did there with publishing those those likes to to expose that advances the stream if you wanted to process them in another service for sure yep first one is so everybody used to do raise or shoot these service versioning as database changes you need to change our data model and change our endpoints assuming up version advance as well so like one of the practice up around event versioning or we wake up activities at this kind of thing because resin points you do you can keep a community so so you go so you the text eventually learn how to use the new version what is it max because uh is a movement to arrive so you may be the case where the event processor or congestion or to be the wrong so so the question is in a rest system you often used versioning because things might change but in a push system because you're just receiving the vents you're not requesting a specific version and so how do you handle when things change so just back to using versioning and rest the first rule of oh well this is we've had a number of discussions at this in light band and how this should should pan out especially in micro services where you're going to have many services that they all have the main lead to deal with multiple versions the first rule of versioning api's is don't version api's so first of all don't break things ensure that all your changes are backwards compatible okay but sometimes that's not enough so the second rule of versioning api's is don't version api's so sometimes and you can you can prepare for that by ensuring that that you design your schema to be four words compatible and so for example of that is if you have a list of things and and let's say it's a list of strings don't do that do a list of objects and even if they just got a single string property at some point in the future it's a very common thing when you've got a list of strings to sometime later turn it into you need more information for a item in that list if it's a list of objects then you can always add more properties later where the strings you can't do that but of course sometimes you don't foresee these things and and so you really do need to to change a schema so the third rule of versioning API sirs don't version NIV is how what you really want to do is if something if the protocol changes you create a new service to handle that new protocol and so and so then office services that want to speak the old protocol speak to the old service services that want to speak to the new protocols speak to the new service the old service you may have it mapping you may have one of them just been a facade over the other one and mapping in each other like that but they do need to be separate services now in Lagaan we haven't implemented this yet but we will be providing support where we'll do some of that map in automatically so so you can define you know migrations I guess for from the old protocol to the new protocol when that when that's possible but conceptually it's still important to see these are two different services they speak a different protocol when it comes to push it's essentially the same thing so if in the case we we were doing a peer-to-peer push them what we were doing is we're connecting to this service when we connected to it we could connect to a different version of the protocol if you're doing a publish-subscribe sort of thing what you would do is you'd publish to a different topic so there's one topic which is for the new protocol and one topic which is for the old protocol and you might in order to implement that have have a subscriber to the new protocol that translates it to the old protocol and gets it and so the the old the the old the older consumers can still consume that that are that old topic the topic for the old protocol but I'm conceptually I think that the best way to view it is we don't actually have versions we have we have a server specs of protocol and it ensures that all the changes are backwards compatible and when we have version new version it becomes a whole new protocol we've conceptually a new service yep typically they would have so the question is how can two services with the pack and the new service coexist with the old especially if the old is a legacy one which hasn't been versioned what what you would have is is different service names so put the version in your service name and if the old one doesn't have the version number in it that doesn't matter because the new one does so in my gum when two services talk to each other they look up in the service locator for the service game that will give them the youth that IP address or the URI for that service and they'll be able to talk to each other it gets bit more complex if you're going through a gateway because then you might have to do some versioning via headers or some things where something's put a version in there you are I and others don't but again if you're for the outside world it's then even more important that you don't break things because you have no control over over the outside world and who's talking into you sir really that back to rule one don't don't version things yep I'm not using my gum okay sorry I thoroughly asking a different question so that so so for locom itself there's actually something that are put into the logon documentation which was a little controversial when a few people first saw it and I said if you have a big system with many services and they're all like gollum services then you possibly should be rethinking how you're making your technology choices because one of the big advantages of micro services is that you can choose the right tool for the job and legume is a great pop general purpose tool but there are going to be some jobs that you have where this specific tool over here that isn't logon is going to fit the purpose better so in in log on there are a few things we are doing to ensure that that a player that it enables you to have a polyglot stack one thing that we're doing is we ensure that we only use their protocols everywhere so when we do rest in my gham it is idiomatic rest it is rest the way that everyone does rest not not logons brand of rest when we do the streaming we're using WebSockets again that's a that's a standard protocol and we're also going to be introducing with support for generating swagger specs and generating clients from swagger specs so swagger is a is a specification for for a remote interface that that um has a fair bit of support in different stacks and on the deployment side of things and then development the development environments other things we have in ways of if you've got a system outside of lag on that's not deployed to say conductor I've been able to tell conductor how is this system so when this client needs to talk about it you go out over there outside of logo and talk about it to talk to it so yeah they've been able to integrate with with legacy systems with non logon solar systems with systems that just don't fit in for whatever reason is is a very important thing in micro services and something that we will are providing support for anyone else well thank you