Scale By The Bay 2020: Muktesh Mishra, The art of being resilient. How to handle failures...
[Music] today we are going to talk about resilient system and how to design or practice some best practices in uh distributed systems so that we can detect uh failures gracefully and handle failures gracefully uh so about me as swapnil mentioned i am a senior software engineer working for adobe in their cloud technology and sensei group and i am a contributor to 20 plus open source project including historic servo contour and spinnaker open tracing lately a lot in and easter as well i run java user group also so if you have any topics to present uh you can reach out to me i am based out of sunnyvale and i am i usually you can find me in a lot of conferences i have shared my github and twitter as well so if you have any questions which we are not able to answer in this session you can always reach out to me on twitter and i'll be happy to answer any questions so that's about me let's start so uh distributed architecture a distributed system so um like we all uh so what happened is uh in the distributed architecture you are distributing your uh system and components uh over the internet uh to to do the resource sharing to do uh computation because what happened was earlier um you had all these systems which were monolith and now with the evolution of cloud with the evolution of distributed computing people have started distributing their load into different servers to in order to do the computing faster in order to share the resource in order to use the resources efficiently so uh basically these diagrams represent your microservice architecture where these are some of the traits for your microservices microservices world is again an example of distributed computing because essentially you break down a bigger system into separate individual component and then each component is responsible for a particular functionality and now you deploy those components separately and then you perform the computing and things like that so each of the component it becomes a micro service and it is responsible for a particular functionality uh take an example of uber so in uber say you have components like payment you have components like ride sharing you have components like right booking customer management so all these components become micro services and essentially you can break them further if they are too heavy but the core idea behind that is uh you are breaking your component into a different system and each of these systems have a separate life cycle so that is why uh we saw the evolution of ci cd continuous integration and deployment and things like that so you want to manage their life cycle now one of the philosophy behind the micro service architecture was they are heterogeneous in nature that is you are free to choose your own technology so for example microservice you can use java as a programming language then in another you can use go as a programming language so you have some freedom on technology technology selection and then the resources are distributed and essentially multipass involves where in order to work all these micro services or distributed systems they have to have some kind of coordination so coordination it means they have to talk to each other and that is where the different design patterns come into picture so since they are talking as a separate resource so you need to have some kind of control over there what kind of authentication and authorizations are there so then essentially because you have distributed your load so it your one micro service can decide anywhere your another component is residing anywhere but they are talking over the internet or the network so uh that is why they are uh that computation computation is distributed but again this is all fine and dandy this is all good but what happens uh here when any of the component fails now um suppose because these components are distributed and if you have to debug which component is behaving badly what is what is the reasoning behind that then it becomes a challenge and that is why you can't see so that is why you see this guy this is saying this guy's saying i don't see the problem because there are certain traits you need to follow there are certain best practices you need to follow if you want to debug the problem quickly because problem can't be anywhere network is a fallacy so network can go wrong or any component can behave badly or there could be things like uh load of one component is choking others is another component so there could be n number of problems go wrong in the distributed systems so this is the other aspect of distributed system architecture where if you are not careful then bad behavior of one system can go wrong and take down the entire system which goes against the philosophy of a distributed system as as well as microservices as well where uh essentially they were meant to be designed india to be operating independently so it means you while designing your system you should be careful about certain traits where the dependency they are loosely coupled they are not strongly coupled so that you can replace them you can manage their life cycle as well as bad behavior of one cycle uh once component does not affect the another component so and then this is what happens uh if system starts behaving badly you don't have any idea where the things are going wrong this is what happens uh this is happening from avengers your spiderman is saying i don't feel good and i don't know what is happening so essentially you need to figure out which component is behaving badly so this is the problem uh a statement like how we can see how what all practices we should do so that if any problem arises in the distributed system architecture we can detect it quickly we can isolate it and we can resolve it because in order to resolve the problem you need to detect it first which component is behaving badly can you replace it can you replace it from the system so or can you stop the flow can you stop can you isolate that component so that it does not affect the entire system right so let's talk about that so uh here uh uh when um when we think from the problem perspective what problem can uh we see so first of all uh we need enough information enough information as in where the things are going wrong what things are going bad so in order to know that you need to have enough contextual information you need to have enough information about through logs through different kind of components you need to have the context you need visibility visibility as in if these microservices are or distributed systems they are talking to each other you need to know where the integration is failing where the component is failing so you need to have that kind of visibility so that you know you can quickly debug which system it needs to uh which system you needs to focus on you need to be able to stop the bad component or isolate the bad component as soon as possible because uh in the essence in the with the evaluation of cloud computing you have the ability to scale your components up and down horizontally or vertically right so uh if you can isolate that component you can quickly spin up another component which can take place of the bad bad component but you need to first identify which component is bad and then uh you need to find it out and you need to have some sort of control like how you should be dealing with that bad guy so uh around these four categories uh let's talk about what we can do so because we talked about we need enough information so enough information means you need to have the logger now uh traditionally the logging has been um what we do is we generate the log logging statement we print it in a file or somewhere and then we use some sort of log retrieval or log parsing tool like splunk to generate some meaning and then if in case of any failure we rely on our logging so traditionally these uh this logging is in the plain text format now what what we can do is and what we have what i have done is uh there is this concept of structured logging now what structured logging is with your logger you are appending the contextual metadata with the login so you can what does this that mean is in case of any problem you can attach extra information with your logger in a form of json payload as you can see in the example here which tells you which you can associate like user which you can uh through which you can associate uh metadata information from your logger for example is it a customer flow is it a golden flow is it a what kind of matrix is it is and where this exception is coming so that way you can add the contextual information for example in this snapshot if you see this is saying action is repair and thing is car so now if if we interpret this if we if the logging system interpret this and if a user sees this they can quickly know like apart from the log name this is what this is where the additional system information is think from a perspective in uh distributed system your logs are pouring into from different systems into a central logging place and and anytime something happens you need to find like hey where where the things are going wrong so that is why uh that is why um you need to have associate the additional contextual information here now this is a relatively new concept but what happens is um because uh right now not all logging uh systems are supporting this kind of logging but then you have uh ability to write your own log parser especially in the fluency or in splunk they started supporting it so you need to give the schema because you can implement a standard schema or a particular schema custom schema but you need to tell your log parsing tool like how to pass this information how to process this information so that way you can write it down that way you can see it you can parse it in the relevant format and that way you can quickly detect you can associate metadata information also with your log common format uh this is another problem uh we see because as the organization or teams or components grow we have a disjoint set of logging format and things like that so what happens is you need to have some kind of common logging format some kind of governance there for example uh you can generate all the uh say in splunk where you can generate the index as commonly formatted like application name dot something name and then in the logging also you can print out like method name and all those so that way there is a governance and uniformity and this especially brings down the if you try to think uh in terms of asari terminology so this significantly brings down the mttr mean time to repair because every time uh when you try to debug submit something into say splunk and now you have to find the appropriate logger and if you are not aware you cannot find the index name you cannot find the log name and you cannot debug the problem so that way you need to find um you need to if we have a common formatted loggers you have a uniformity there and then you can quickly find the logger name and then you can quickly uh debug the information so that is how you it this all focuses on the efficient logging so now uh you uh we talked about logging information so we have the information but now we need to find out how the flow works how the things are working in so uh in this case you need to get some visibility onto uh where your calls are going if if you have different components talking how are they talking so if you have this kind of information then you can quickly detect like hey fro at this level your call is failing so for this phenomenon we have this concept of observability so observability is essentially uh talking about two things so one is uh distributed tracing so if it is especially in the microservice world what you need what this concept tells you you propagate a common set of headers across your different calls now what happens is uh you have service a calling to service b calling to service c in a user flow now you are propagating a common set of headers across these calls and then if you have to retrieve the information for a particular user you can retrieve by those headers you can pass the customer id or user id or whatever information you want to pass up but there also goes some kind of decision making because you need to identify how you want to group your flows how you want to uh how you want to see so for example in this snapshot if you can see there is this hello client is calling the formatter and then uh publisher echo echo tracer so this is showing you the spans and traces like how much each call is taking and how much time each call is taking and then what they essentially when you build your timeline like in a timeline uh how it is rolling up to the entire call stack second concept is the call graph tracing so call graph is again a concept of um tracing like when you are talking in in a microservice world when one service uh is talking to another and things like that so call graph tracing comes into picture when you have to trace the flow so you have to uh trace for example in uber's world again if you go back to the uber uh you are booking a ride but behind the scenes there are a lot of things happening it is finding the available number of rides it is when you are trying to move it's calculating the distance it is calculating the price so there are different microservices or different functionalities working together now uh if you have to trace it like uh when user clicks on book a ride what all things happening there so that is where the call graphs come into picture it is a way to visualize uh how your micro services are talking to each other so that is how call graph works so again in order to trace this in order to visualize this you have different mechanisms you can leverage distributed tracing again to pass that header and you can quickly visualize there are tools like uh where you can visualize these call graphs or even in this case this snapshot is from jager jaeger is again a distributed tracing library from uber so that gives you this kind of instrumentation which you can instrument in your code and then you can quickly generate your call graphs so you can identify where your call is going so that is how you identify the patterns uh like where your call is going so logger gives you the contextual information and then the call graphs give you information about the communication uh control switches so um we uh talked about um so we have information like uh what kind of uh contextual information uh we have the visibility through our tracing and then once we identify the component how do we stop the flow so that is where um that is where the control switches and bounded architectural contexts come into picture because uh using control switches and bounded architectural context you can stop the flow now what does this mean so there are two kind of architectural context you need to decide so in a distributed system like we talked about if two systems are talking and one system is behaving badly you should always have a means to disconnect that system now uh what does that mean you need to be aware about your participating systems participant systems in uh what all thing what all systems are working what all components are working a system knowing the limits so when you are integrating with any any system when you are talking to any system you should be knowing what their system boundaries are whether they can scale to a particular limit or not and then uh if they can't then you should always have a throttling kind of mechanism so throttling mechanism is like you can always stop the flow from going there you can hold the request until the till the time other system recovers so usually for throttling in different people uses a queuing kind of mechanism where they hold on the request and then they apply the leaky bucket algorithm they only pass the number of requests which can be handled by downstream systems but again this is the best practice you should always implement where you can hold the request and process it as per the downstream systems capacity so that you don't end up choking or even if the downstream system is behaving badly you can stop the flow there control blast radius now this uh essentially these uh blast radius terminology is used in uh in terms of distributed system when any system goes wrong how quickly you can stop the domino effect so that it does not propagate to other components in the system so uh that is how when you need to control the blast radius you need to isolate that component again the faulty component again and now in this case uh feature flags are one of the methodologies adapted by a different companies and teams so whenever you are integrating with um any other dependency or any other system whenever you are talking to any other system you can always enable it or disable it based on some flagging mechanism so that is how the feature flagging come into picture and feature flagging is also used for uh gradually controlling the interaction as well so same concept if you want to apply some kind of you want to release some kind of feature for certain set of users or things like that there are also feature flex being used but again if you want to apply uh x uh in terms of infrastructure you can also use there to quickly stop the flow or quickly isolate a particular component circuit breaker so circuit breaker is a design pattern which gives you the ability to define the fallback behavior at the runtime and now circuit there are different libraries where you can do this on the code side and now there are infrastructure component as well which gives you this kind of ability so essentially what happens is if you are talking to any system and that system starts behaving badly then you can actually uh define your fallback behavior and you can define a chain of fallbacks for example um in a real world system if say you are you are developing an application where you are trying to fetch the codes of the part uh like in a if you are in a repair kind of system where you your customer comes in submits the device and then you quote their repair cost and things like that so in that case you need to fetch the inventory from some system and then uh you need to get the pricing information quota information things like that now these things are critical for the application but what uh if say you are fetching this information from a third party site and that site goes down what are your options because you cannot stop the customer flow so you need to find a way where you can handle this failure gracefully so maybe you can develop a system which syncs with the third party and then you can at least give a quote to customer so now um how do you handle this so circuit breaker design pattern helps in these kind of failure scenarios where if the third party goes down it automatically enables a fallback behavior and then fallback behavior you can define what how the alternate flow look like and it also gives you the ability to define it also gives you the ability to define a chain of all fallbacks where if one component goes down you can you can rely on the second flow if second goes down you can rely on third and so on so that is how the circuit breaker helps now we talked about circuit breaker design pattern uh so there are two places where you can do implement the circuit breaker design pattern one is via code and another is on the infrastructure side so in order to do that in the code there are different kind of mechanisms you can do histix was one of the very popular library which is ul now end of life but two of the most important libraries in java world are resilience 4j and then sentinel uh so they give you the ability to define your application's alternate behavior in case of a failure in case of exception and from there you can easily configure like what kind of flow you want to do in this case uh what happens is you need to embed these libraries into your code so there are different kind of library libraries available into different languages but again you need to in order to leverage that you need to instrument that into your code itself you need to inject those libraries into your code so that that is a that is an extra hop you need to do so that is why there is a second kind of uh circuit breaker implementation where uh these proxy pattern comes into picture along with circuit breaker and steel and envoy gives you the very good ability to do that that's the same thing essentially in the sd over they inject a side car and then side car keeps track of in and out inbound and egress and ingress traffic for your container so from there also you can apply this kind of behavior where if you need to turn off the traffic to a particular container and then you need to redirect that to some other container so history also gives you that so the leverage of that is uh you don't need to do anything in any instrumentation with your code you can handle totally it at the infrastructure side so that is where the circuit breaker comes into picture now uh well-defined sl axis so that slx uh essentially talks about service level objectives and service level indicators in your distributed system so anytime you are integrating with any system uh and you have a dependency you should always have the indicators and objectives defined like what what how do you measure if a system is behaving correctly and what kind of indicators or what kind of kpis you need to measure in order to determine whether the system is behaving correctly or not so uh for example if you are integrating uh with a particular system you should essentially ideally in the micro service world there is this notion of red matrix where you measure requested error rate and durability and things like that which gives you the some sense like how your system is performing if you have if you are sending a request response kind of system then you can measure the error rate and things like that in order to determine whether your system is behaving correctly or not so you should uh as a practice you should always do that and uh in this case uh if you have defined these sls uh like well then what happens is you can tweak your alerts so in my experience i have seen a like tweaking the alert definition dynamically so essentially you have basic indicators defined essentially you have basic indicators defined and then you are measuring those and you are tweaking your alert definition based on to that and then you need to always validate your alerts as well because if your alert if your sls are tweaking then you need to validate whether they your alerts are incorporating the slx's or not and uh smart and actionable alerts so all alerts should be actionable if you have any kind of indicators there then you can you can tweak your alert definition based on to that so what did we learn so essentially we learn a couple of important concepts and i have a demo also which i'll be sharing at the end of this talk um essentially i would like to run that demo but we ran out of time so feel free to run that and if you have any questions reach out to me so essentially what we learn is logging information is important and then architecture consists of components and any component can go wrong at any time call graphs and integration and communication patterns should be traceable at any given point of time you should be having you should be aware about what is going on in your system and then service level notation should be part of the design that is when you are designing a integration pattern at that time only you should be thinking about your service level objective and service level indicators that is how you will define that correct dependency in the accurate dependency whether the dependent system is behaving correctly or not so appropriate indicators needs to be defined i have included a couple of resources uh you have uh i mean essentially these resources are talking about the structured logging how to do that and then micro services how to debug it and i have included this demo also um feel free to clone it and run it uh this has all the information and concepts which we have talked about so yeah that's all from my side thank you so much again if you have any questions reach out to me on twitter or linkedin as well so i'll be happy to answer thank you so much you