Reactive Systems and Microservices: Anand Patel, "Migrating Monolith to Microservices"

Reactive Systems and Microservices: Anand Patel, "Migrating Monolith to Microservices"

Recording: Reactive Systems and Microservices: Anand Patel, "Migrating Monolith to Microservices"

so before we get started raise your hand if you guys are currently deploying a mana list to production a few people okay now keep your hands raised if you want to take that monolith and move into a micro service cool that's good and now raise your hand if you guys are currently deploying microservices and production today oh now keep your hands raised if you want to move back to a monolith okay cool well today we're going to be talking about how to move from a monolith to reactive microservices I'll go over how people typically move from a monolith to microservices and I'll propose an event-driven architecture and the winds and woes of this kind of system that will end with a nice demo a bit about myself my name is Anand Patel I'm the lead engineer here at runnable and i graduated from Georgia Tech with my master's degree in Computer Engineering and I started my career at Nvidia riding low level software Colonel drivers and from there I moved to large-scale distributed systems now surprisingly they do have a lot of things in common mostly programmers but for the for the past four years that i've been working at runnable building and designing reactive system now I have to put this morning out there this talk contains grossly oversimplified code and is not intended for youth in production now it's a warning out of the way today we want to make a micro service we want our we want to make a service we want to make the best music calm now on this website what it should be able to do is you should be able to listen to music and also comment on it and lucky for me I have all these smart guys here in the room to help me build it so let's get started so we spent some days and we build this thing as a music player comment and Jin email or video hosting database all in one gigantic monolith now how would this look at some example code from this sort of tech system honor let's take a look at what happens when / when a user comments we have a parser which normalizes that data we have a database connector which you know saves that comment in the database and we have an email or which sends a thank you email pretty straightforward and I think this relationship to production so we go ahead and ship it to production and you know it no hitches besides the fact that it takes 10 gigabytes of memory on boot but you know whatever we can ignore that for now and this is running first fine until one day we release the rough remix by the d-o-double-g now on this day our site went viral and we had to start scaling like crazy and it became pretty expensive because you remember our motto that takes up 10 gigabytes of memory every time for every instance of it and we needed we needed to do something better because along with all these new users that came in they were asking for more and more features people wanted to add emojis in the comments they wanted to support markdown in their comment or they forgot their password and wanted password recovery and so we started adding all these features to a monolith and it actually became more and more dangerous because one small change could cause the entire system to go down and that we couldn't accept that was because our competitor overplayed music com was always up and they were shipping features like almost every day so what do we do we heard about micro services and so we decided it will help summer palm so we go back to our mana list and we tear it down into small functional parts the database the parser the emailer they all get their separate microservices and for everything that we can't functionally break out we stuff it into our macs are master service AKA the micro lift now what would toads like in this type of system well now instead of calling the calls directly we have restful clients which interact with each of these micro services its pre identical to the modest code right however we've introduced a few problems for example what if the email service goes down now when a user tries to comment he's greeted with an ugly 500 error and he's angry because he can be the first one to comment on this new music and typically in these kinds of systems when one service goes down the rest of the services that are dependent on it go down with it or we bubble up some air all the way back to the front what about another example say the email was just slow because it's bogged down now when a user tries to comment he's stuck looking at a spinner just because we couldn't send him an email and he gets really frustrated now of course I'm sure this will never happen to any of us because we write perfect code we have a hundred percent test coverage we catch and handle all of our errors and have seen timeouts for all of our HTTP clients but what about in this scenario we realize that in this app you only really eat emails anymore we need to do something more hip so we decided we wanted to send thank-you selfies on snapchat now nobody here knows how snapchat works so we needed a hire a developer specifically for this task so we hired a a snapchat developer and since we have microservices all they had to do was write this snapchat microservice and add one client line to the common functionality easy right well it turns out our snapchat developer lives the Yolo lifestyle and test his changes directly on production an authentic prod cowboy and now we've actually broken our critical path nobody can comment on the site anymore now microservices promises fault tolerant resilient systems but without significant code investment we actually never get to see these benefits now we got to this kind of system thinking in a linear way when X happens we want a B and C to happen but what if we think about this problem in a different way enter reactive architecture one service should not tell another service what to do instead these services should be smart enough to know when something's happened in the system and react appropriately now what do I mean by that remember example the michaelis should not be telling the email or send an email or it should not be telling the parser to normalize the data instead these services should know when a user has emitted a comment and then respond appropriately now what would the architecture like this look like something like this as you can see our Michael tipps has shrunk considerably now that it does not have to handle talking to the other services but the major change here is our message bus we now have a message bus that connects everything and its job it's a pass messages around it knows who subscribe to what events and those whose publishing those events and it routes it correctly this becomes the heart of your application now there are few services that do this already amazon SQS Kafka and the one we use here runnable is I'm cute now let's go back to our code example now when I micro list now when the my girl is gets a comment it emits an event called comment received and it passes to the message bus now the message bus knows who subscribe to this and sends a message accordingly so the email i guess this message it sends a thank you email the parser gets this message it normalizes the text and it also emits event comment normalized which the database picks up and then we finally save the comment now remember our snapchat example how about that well the snapchat developer just has to make this service and it doesn't actually have to change the model is it just listens for a comment receipt and does what it needs to do it sends a selfie now one thing i want to note here is the event name comment receipt is very important that you never describe the intent in your event name event name should always describe what has happened in the system and if you do this you can truly d couple these services because they don't know what's going to happen when they aim at the event let's talk about what makes a system really great as you saw from the example the services don't know that the other exists which makes these services to lead a couple because they're not describing intense in their events their only saying hey this happen take do with it what you will and with a snapchat example you saw that you just have to make your own service you did not have to change code in any of the other micro services which makes it which lowers the risk of adding new changes to our system now since these services can go in and out you know they can crash as much as they want they won't bring down the rest of the system each microservice is independent which allows for the system to you know gracefully degrade and when these services do come back up we can actually send them the events that they missed allowing them to self heal themselves and get themselves back in the right state and now I think you've noticed that these services are basically just consuming events and emitting evidence what we can do is we take out of the spec give it to any developer and they can implement it and they don't have to know how the rest of the system works at all and this promotes independent team now of course with all architectures there are some trade-offs so it's time to talk about some of the whoa first things first the message bus becomes the single point of failure if this goes down your entire system comes to a grinding halt so we want to make sure that you know our message bus is highly available and we're following all the best practices for this and we also want to make sure that we're persisting these messages whether in a database or on disk somewhere in case of catastrophic failure now that we have all these messages flying around it becomes extremely difficult to the above these kind of situations there's not one place in the code where we can look to see flow we can't see you know a B and C have any sequence so we need to make sure we have really good logging practices we want to log every message with a transaction ID who's published the event and who is consuming the event we also want to make sure we have up-to-date documentation of these events and the flows if you've done number one is relatively easy to make this documentation dynamically update itself now consistency also becomes a big problem this kind of system because we're dealing with a sink messages if you can never guarantee at what point in time these events are actually going to be processed however we can ensure eventual consistency by doing two things you want to make sure that first your message bus guarantees at least once delivery and second your event consumers are item potent meaning if you send the message twice it will still be in the correct state now it's cool to talk about these newfangled architectures there's another thing to actually use them one of those actually built on this event driven system and we've seen a lot of benefits we were able to implement new features a little risk before we had a lot of regressions which cause you know significant portions of our application to go down but when we move to this new stacks we were able to add features more rapidly and reducing the risk of bringing down the rest of the system and some of these services don't have to have Eric catching inside of the client and they don't know about the rest of the applications we've significantly reduce the complexity and the code required in each of these services and it's also actually helped us with unstable endpoints we've actually been working with doctors since the point three days and we've been through all the API changes all the econ resets and socket hang-ups and we started when we have the original microservice architecture we just had an HTTP server sitting in this guy and when dr. went down or held two sockets you know this service would handle it would you know still return to 500 area and we realized when we had 10,000 requests happening at the same time this guy just choked up and that was because it turns out ports are actually a scarce resource on servers and when you're running 30 40 micro services on the same server you'll find you'll quickly run out of ports if you're holding them open and since the message bus now takes care of most of this traffic we significantly reduce the amount of TCP connections that were holding open and one unsuspected benefit that we got out of this was for new hires before when we got a new hire they were fixing boring bugs writing tests or reading outdated documentation today we can actually give them a service to write themselves and they can push that to production and they'll feel accomplished and we know that it won't bring down production so I talked a lot about the service but so you want an action I'm just going to go here i'm using dr. composed for this example so we're going to bring it up all right it should be good now this is actually an interactive example whoever has a laptop can log in and if you visit reactive chronicle com you have the best music calm and here you can write your comments hey do it and now what this is doing it is sending an event to the back end and this back-end servers publishing comment received inside of our random q over here and you know just like an example we have an email service and we have we have an email service and we have our parser our parsers actually are normalized it just normalizes the content down to 100 characters it was over 100 characters it will just truncate it down to 100 characters very simple and so if you want to see how fault-tolerant works let's so let's kill the Nativity let's kill the database right right Oh sick stepid let's chill oh my god all right now nice its kill so let's see alright as you can see you know database is down so it's not saving anything and if we just bring it back up wait a few seconds live demo refresh the page it came back this is a vest news nice now what's now like I said you can actually start adding services to this to this rabbitmq I had the link here if you clone this repo you can actually start the service and now you connect it to our to our rabbit meaning you'll also get the message so if you start typing something this service you know immediately gets that event and this can crash that many times it like and it's not going to affect you know this message messaging system okay cool right so you guys can play around that if you guys would like but you guys have any questions yes what is it yes yeah Mike yeah right right mm-hmm great okay so there's there's like two ways you can solve that you can solve it you know with traditional HTTP clients you don't have to fully go you know fully reactive you can still have the clients where when you get a request you can you know communicate with the with your all server you held up whatever you want and ensure that you have the correct authentication or you can do is sort of a proxy message of message where you would say that you send this message to an authenticator and that Authenticator would authenticate the message add whatever data it would need and then we send the message to the actual target that it was intended to intended to go to sort of like middleware for event great yeah correct right so yeah you would you would have to wait it that's one of the downfalls of the system it is eventually consistent it's not consistent right away so that's one of the trade I think have we run into any race conditions because our consumers are item potent so how we deal with race conditions is even though sort of these are a sick messages we usually if we need something you happen in sequence we omit it those events so as you saw in our example if we need to be to be in sequence we would actually listen for like a new day event that that would emit so in this case comment normalized and that would actually save the comment it would actually listen for the actual comment received so if you need something to happen in sequence you'd have just a sequence of events that you want to listen for will it be a warning for protective of the girls with online version no we went out for the results Tennessee's Commission each other a design review or ever will even help our reproduction or any production Network iva-mos the most unreliable they are aware or energy thing through your city or they are unreliable mundo Network only company people while people were more confident abilities are both all his immunities happens my promise we do the first musician or by our attack you which is our more petrol reliable however our people no quick so his question was the networks and reliable and other methods are you know the trained to handle these situations how does it how does this basically system reacts when networking issues happen so one of the cool things about random q let's go back to our architecture one of the cool things about random q is they actually have two days to you know get around the networking issues they have two things whenever you publish American methods it has guarantees as a message was published and that means it's persisted on two discs and it's the process is only valid only wants to save it onto disk so this process will continuously retry until it's known that the message is on disk and on the other side subscribing to guarantee at least once delivery which confirms rabuck you won't release that message from disk until it's guaranteed that the server has profited and you know acknowledged and it received the process so basically we deal with the networking issues we're having exponential back-off we try processes in both in both the consumer and the publisher Arthur welcome additionally he balances on education so that if of one of your network who was appointed course be partying network because settl Hillary down in which I will fail and locked and helping them are to eat on the metal darling mean we tried everything oh right so you can actually have more than one of the same work at server so you can have like 20 of these all on different notes and they can communicate with the same rather than Q or you can have a replicated random q across you know multiple data centers and so this this side goes down the next you know parser will parse the data so these aren't like single these aren't single it looks like single but this can be scaled and scale web scale any other questions all right oh thank you very much [Applause] you [Applause]