Scale By The Bay 2021 : Shiv Toolsidass, Roy Williams, What the heck is gevent?

Scale By The Bay 2021 : Shiv Toolsidass, Roy Williams, What the heck is gevent?

Recording: Scale By The Bay 2021 : Shiv Toolsidass, Roy Williams, What the heck is gevent?

hey how's it going uh thanks so much for coming to our talk uh which is what the heck is g event uh next slide so uh again my name is roy um i'm a principal tech lead manager here at lyft i support the developer platform team as mentioned we own a lot of the tools that engineers use to build apis and otherwise build lyft and shiv if you want to go introduce yourself yeah my name is chef tosidas i work as a software engineer at lyft and i work on the bikes and scooters team over here cool um yeah so today we're gonna be talking about gmat uh you may have heard about g event before uh it's as shiv will talk about more it's kind of magical pixie dust you sprinkle on your code to automatically make it have much higher throughput but with that great power comes some downside um there can be some carcass problems you run into there can be some performance problems around too uh and then shiv's gonna talk through how we took some of these learnings about g event as a whole and applied them to improve the business here at lyft uh and then we'll talk about some takeaways and uh end with the q a uh but with that we'll hand over to ship for intro to gven uh take it away awesome thanks rory so so let's get into what gman is g n is a core routine based cooperative multitasking framework that's a mouthful but don't worry we'll explain what each of these terms means in a bit so why does g event matter so much to lyft and why are we giving this talk to you today the bulk of lyft's python microservices are backed by g event when you open up your app and request a ride our python web servers are able to schedule requests thanks to g event understanding its intricacies has allowed us to scale python really well in production and why should given matter to you if you leverage an async framework or language that uses an event loop like node.js these learnings are going to be really useful to you and if you're encountering challenges scaling your services they're especially relevant like roy said g1 works like magical pixie dust you write these two lines of code and your service is able to 10x's throughput and cooperatively multitask its workloads however this power comes at a cost and we're going to explore that during this talk to understand how gment works let's look at a scaling problem that lyft might encounter while matching passengers and drivers let's hop on over to a jupiter notebook to look at some code examples awesome so let's say that lift matches passengers and drive passengers and drivers by matching passengers with the driver closest to them so let's say we have a couple of classes that model etas user drivers and passengers now let's say we have two drivers ship and roy and we have two passengers alice and bob and we're trying to match alice and bob with drivers in the lyft platform these requests to match alice and bob are separate web requests and let's say we we you know match them one after another synchronously this is going to this takes about two seconds now this is fine when live small but let's say we start experiencing hyper growth and lots of drivers and passengers on our platform this doesn't scale very well and our users are unhappy because it takes so long to be matched with the driver so how do we fix this let's let's add some concurrency we import this library called g event which we just talked about and don't worry about these two lines they'll make sense in a bit we also have to spin up a separate sub shell because g event messes up our jupyter notebook and import our classes again so you can see that we have the exact same code except that we modified this one line and added a g dot join all and spawned up several green lights to to handle this workload and make it concurrent and amazingly it's cut down our wall time in half so just to recap very few code changes really neat now let's compare that with another popular framework called async io um in in this example you can see we had to change our code a fair bit we had to add in these async definitions we had to add in a weight and overall a couple of more changes than just using g event and when starting up this example we actually you know fumbled quite a bit and it took us a while to come up with it so it does add a bit of complexity to your code it's not as elegant as g event so let's get back to digging into what g event is all about so earlier we told you that g event is a core routine based cooperative multitasking framework let's break down what that means in the word core routine co stands for cooperative and routine stands for function core routines are functions that cooperate with each other let's first take a look at some functions that are not cooperative in the example here when print numbers cause them and get numbers print numbers can't continue until get numbers finishes and returns to where it was called the control remains with get numbers until it executes completely and only then can the control be returned back to print numbers back to core routines so what are core routines core routines are functions but the where the control is transferred from one function to the other in a way that the exit point from the first function and the entry point to the second function are remembered and this happens without changing the context so each function remembers where it left off execution and where it should resume from in the example in the slide control is transferred from print number print numbers to get numbers and when the yield function is encountered in get numbers the current state of the function is saved and control is returned back to print numbers on the second iteration of the for loop and print numbers controllers return to get numbers so as you can see we can leverage this concept to pass around values between functions in the middle of their execution without breaking away from the context of each function this notion of yielding control from one function to the other is the building block of concurrency in g of n so in g event a core routine is called a green lit or also known as a green thread but keep in mind that a core routine is not a thread earlier we said that gun is a cooperative multitasking framework what does that mean cooperative multitasking is a style of multitasking where the operating system never initiates a context switch from one running process to another that seller of multitasking where the os initiates a context switch is called preemptive multitasking the operating system understands priorities and how much time each task is run for and it can interrupt processes to allow another one to run incorporative multitasking on the other hand processes voluntarily yield control to each other when idle or blocked in this form of multitasking programs must cooperate for scheduling to work the image on the right over here is windows 3.1 which used to rely on cooperative multitasking to allow multiple applications to be run at the same time on a computer with one core the other thing to know about g event is that is that it uses this concept called monkey patching to modify the runtime behavior of standard and third-party libraries so that they can be cooperative in the example in this slide the monkey batch time sleep function when we run this program it prints out should have slept 10 seconds instead of actually sleeping 10 seconds in g event monkey patching is the first thing a process needs to do otherwise libraries are going to perform real blocking implementations which won't be cooperative monkey patching is generally considered to be a bad practice because it can lead to bugs that are hard to understand for example if the underlying library changes the monkey patch needs to be updated as well and one might forget to do this so let's tie everything together we've talked about so far with an example monkey patching in this example makes time dot sleep that's invoked in the send stat function cooperative instead of blocking the cpu it yields control for at least the sleep time all the functions in this example are cooperative otherwise known as core routines control has yielded from do thing to send stat back to do thing which yields control back to do things i'm going to hand it back to roy to talk about correctness in g event cool thanks shiv really appreciate your overview uh and so with that we're going to dive into some more code and talk about some of the nuances of gmat uh so as she mentioned jiven is a really useful library for changing code that was written in the synchronous manner and automatically making it asynchronous but this can come with some of the traditional problems that come along with thread-based programming um so if you've ever worked with java or things like that uh so we will go through some examples here and some of the problems and how we're gonna fix them so for this first example we're gonna imagine that we're building a bank like no fancy cryptocurrency just a traditional in-memory bank database um so we have a database here that just maps uh strings to integers which is just the name of the account to the balance we can list the accounts we can send money between accounts um we have some useful debugging code here that we'll use later uh and we have this function here compute new balances which is what we use to decide what the new balances should be it takes in the old balances the amount transfer and creates the new balances so you can see when we send money we just compute the new balances and then we assign those values in the database pretty pretty straightforward so i have my code here great it's it's running you can write some unit tests for this code if i send 10 from roy to jack where we should have ninety dollars and jack should have 110 at the end of the day assuming everyone starts with a hundred i run my test which is great um now we introduced a new function here called send money around i know this function is just gonna send money in a loop um so of the three people royal send money to jack jack will send money to nick and nickel some money back to roy uh and so at the end of the day everyone should have the same exact money same exact amount of money in their bank account uh and so again we can write this code and write another unit test and our trespass um and because we are and to simulate web traffic with a g event based system i'm gonna be spawning a bunch of greenlits uh they're gonna start at random times so call send around money um and we run this and it's great with those hundred requests that are sending around money at the end of the day everyone winds up with an equal amount of money in their bank account so i'm feeling i'm feeling pretty good about this um now let's say we're getting ready to go to production we have this given based system and i send this for code review to shiv uh shiv being the stickler for correctness that he is and reliability he says roy you gotta add some logins in this code how are you know how much money is being sent around uh and so i say you know what ship you're right we're gonna add a function here called uh stats anchor amount which is going to just increment some counter for the total amount of money and we figure compute new balances is the right place to put this code in and this feels pretty low risk right um i'm not going to try to look at discord at the same time but you know put in chat if you think this is risky or this is a safe change to make it feels pretty safe to me like what's what's wrong with his observability nothing nothing can go wrong here uh so we're gonna add this code in you know think of us of what what could be breaking here um we're gonna run our unit test though just to make sure everything's okay and great our our unit tests still pass uh so let's go ahead and put this in production and let's see what happens all right so something's gone wrong here um jack has 120 at the end of this nick has 130 event of this uh let's let's just try it again oh wow now nick is really angry because he only has 40 left i'm super happy because i've got triple the amount of money um what could be happening here though like this doesn't make sense how could just adding a stat call create all this mayhem inside of our code um so luckily we did add that debugging code in earlier so we're going to go ahead and use that so and all this code by the way we wrote a blog post about g event and all this code isn't there so if you want to download it and play with it and work with the problems there you can you can try it for yourself okay so this um logging code will spit out changed if the amount of money changed between when the transaction started and when the transaction finished and so let's look at this example here of thread93 sending money from nick to roy and so at the start before we compute the new balances roy is 110 jack has a 110 and nick is 90. but now afterwards roy has 110 jackets 110 and nick is whereas 100 jackets 100 hundred so what has happened the balances have changed out from underneath me as i was starting to run this code and you can see where this happened right here so because i added in the opportunity to context switch what has happened is that we context switched to another thread when i started to compute the new balances i switched to 57 and finished that up and i started 63 to compute those new balances there but 57 changed the balances out from underneath me before there was no opportunity to contact switch but now now there is one and that's what's creating this mayhem that's what's creating this problem and so traditionally you think of this as something called a critical section or a piece of code where you can't have any context switching and so we're going to talk about two ways you can potentially fix this inside of g event uh the first is to avoid the context switch all to avoid the critical section altogether so instead we may do what banks usually do banks very rarely update the actual account balances they keep everything a ledger one of my first mentors told me that accountants don't use erasers if they accidentally send so much too much money they don't update the account balance they do a reconciliation transaction and so that's what we're going to do here instead we're going to accrue transactions to a ledger and then when we read the database we will replay all of those transactions to get the current state of the database this is obviously much more expensive but should be a lot safer great and so now i've got my concurrent system running and i don't have any of this nonsense of tripling my money um another way this is also one place where async i o actually can add some value um so if i were instead to try to solve this problem with async io i'm just going to go back to the old state of the code so with async io one of the nice one of the problems with this code here is that i have no indication that this function stat sync or amount is going to make a network call even if i want to try to avoid context switches and keep this keep this all running at the same time i can't really enforce that in async io because you're adding in these new keywords you can actually enforce that and the caller gets some indication that the code is going to context switch so if i were to use async i o instead i'm going to get an error when i try to await from a regular function so i have to mark that function as async and then when i wait here i also have to i also use the same error so usefully i get more signal as someone using this code that hey this code is going to content switch and so now actually i know this is good signal to me as an engineer that hey actually i can yields here when i'm calling the stat synchro amount and so actually it could be wind up with incorrect balances here um going back to the old code though another potential solution to avoiding context switches is actually g event has some support for this so g event built in has a function called the search switches which is a way inside of test cases to assert that a chunk of code will context switch we obviously want the opposite of that but we can massage the test case into being what we want so if we take our old bank so we assert switches and then we fail if the exception doesn't throw so we take our old test case and we fail because we were context switching if i go back to the code and i remove the stats call i'm gonna pass or if i change to running a safe bank my test is gonna pass again and so now i have a few options for how we can think about solving some of these concurrency problems uh switching gears to performance um again g event like you're using g event to improve the performance of your code and so it may be surprising to hear they can actually cause a lot of performance problems uh so as shiv had mentioned you know gfn is great for paralyzing code um we can take some code that's fetching a bunch of resources serially and now automatically make them fetch in parallel for this example though we're going to be using something akin to a web server now most web requests are do a mix of cpu work and i o work so uh we have a function here that's at the bottom um we're going to be running a series of benchmarks i'm not gonna actually run the bm it's gonna show the output for the in the sake of time um but we can take in a work time the network time and the number of green threads that are running and so you can imagine this setting here being roughly equivalent to this where we're going to do five milliseconds of uh compute work 55 milliseconds of network work and then we're going to be off to the races um and then we're gonna output a bunch of statistics here so we're gonna run this benchmark here for five milliseconds and 55 five milliseconds of cpu work 55 milliseconds of network with one green thread uh and we get what we would expect we have a really tight response time everything's centered around 60 milliseconds um p50 p99 are really tight we have histogram here uh but the challenge that we have is this number here the cpu utilization is at eight percent and this is as high as we can go uh and we're only at 16 uh requests per second so let's see if we can do any better um so now we are going to quintuple the number of green threads lo and behold we're actually still doing pretty good we did successfully quintuple almost quintupled our rps and we've vastly improved our cpu utilization uh also simultaneously we haven't really negatively impacted our um our request time distribution we're still having that pretty height 61 milliseconds to 63 millisecond response time uh so this seems pretty good so let's think about what's happening here so what what's effectively happening is that you know request seven is doing some cpu work going to the network but then when it goes to network request 33 comes in and it uses the cpu and then when it goes to the network request press 2 comes in and by the time request 7 is done it's ready to ready to execute and so this will be my next question to the chat feel free to add some thoughts in here uh if 55 green threads is good 50 green threads has to be better right does everyone agree with me uh so let's see what happens here we can actually run this one this one should only take 10 milliseconds so take this 10 seconds to think about what should happen here is this going to be better or worse than our previous results and so the good news is we did dramatically increase our cpu utilization we're at 95 percent we did increase our throughput we're at almost 12x the throughput improvements but this looks real bad our response time distribution our p50 has gone to 180 milliseconds our p99 has gone to 229 milliseconds like this is our histogram you notice there's some chunking here um we can talk about that in q a so why that happens but how could this be how could we have a 60 millisecond request take 180 milliseconds um luckily we do have some instrumentation in this benchmark as well that we can go through so we have an option here to print out the long to print some details and so we're going to print the longest request there's some different settings here but it's the rough same idea so we have request three here as our longest running request it starts running it does this 25 milliseconds of cpu work and then it schedules its network work that's supposed to take 150 milliseconds then request four runs and request zero ones then request five then one then two then six and seven and then finally we're back to three and this is the really interesting part here that 150 millisecond network request that spent 150 milliseconds on the wire actually took 225 milliseconds um and what's happening here is we have that same pattern of you know we do some cpu work and switch to network but now about midway through request eight request seven is ready but it can't run because it hasn't been scheduled yet so 11 runs 33 runs six runs and then we're back to seven and so now all this additional time is adding to my latency uh really harming the user experience um of course because this is a jupiter notebook we can use some pattern we can run this benchmark over and over again and see some patterns uh and so we see that for this benchmark for these settings of five milliseconds of cpu work 55 of network work we see we asymptote to about 12 that we are never able to get more than 12x throughput increase um and if you're familiar with any operational research this probably reminds of amdahl's law ondel's law roughly states that the maximum speed up you can expect is one over the percentage of the work that's paralyzable so if you have something that's 50 paralyzable you can only ever get a 2x speed up if you have something that's 95 percent paralyzable you can only get a 20 xp up now this can be applied to g event because this can allow us to think about what is that maximum amount of concurrency that we want what is the maximum number of greenlits that we want is roughly that ratio of cpu worked and network work um and to show an example of that if we are to plot the amount of the response time uh versus the number of green threads we see things are roughly steady until we get to 11 uh green threads you know maybe even before then but then after we go past that the response times just steadily increase uh and so the main takeaway here is that yes g event is great there are limitations to how much parallelism you want to throw at a process though uh and on the blog post you can find a lot more details of how to better optimize uh your your setup here and with that i'll switch over to shiv to talk about how we use this to solve real business problems awesome thanks rory perfect are uh folks able to see my besides yes speaker don't stop okay you're speaking out drop oh gotcha let me let me just uh modify this one sec are my speaker not still up here okay awesome so in 2020 lyft had launched a feature allowing users to compare viking scooter modes against rideshare and transit modes the goal of this product was to provide users with all the information they need to plan a bike or scooter journey including eta routing and pricing after releasing this feature to our users we noticed a glaring problem in the user experience bike and scooter modes would flicker in and out before we go further there's a bit of terminology that i want to introduce we're going to use the term offer to represent a lyft mode along with all of its metadata such as eta and price when you enter an origin and destination to your lyft app your mobile device pulls the list back in to request a list of offers like bikes scooters lift and lift excel back to the problem we were encountering according to our success rates about 0.5 percent of requests were failing two and a half nines of reliability isn't great for one of the highest value user experiences in the left app the obvious suspect for these request failures was timeouts and so we dug into our p95 latency graphs we dug we found these metrics to be consistently below 450 milliseconds in all of our markets which wasn't so bad because it was below the configured timeout threshold of 500 milliseconds however our p99s in some markets were as bad as one second which was twice our p95 latency and our p triple nines were an astonishing one and a half seconds which was 3x or p95 latency so to understand what was going behind the scene what was going on behind the scenes let's dig into the architecture powering bike and scooter modes in the lift app mobile devices pull a go service called offer service every five seconds with the user's origin and destination to request a list of offers offer service constructs each mode in parallel and to construct a bike and scooter mode offers service calls in a pricing service which is a python service we impose a 500 millisecond constraint on the generation of each mode if any mode doesn't return within 500 milliseconds it's going to be dropped from the list of offers so at this point you may have guessed what was causing the flickering of bike and scooter offers on the first bowl lbs modes were biking scooter modes were returned but on the second pole they weren't alternatively the first ball returns no results but the second pole does for the first case you may be wondering why don't you cache these modes unfortunately this feature wasn't available in our platform at the time for various reasons so it it wasn't something we could do so we decided to dig in a little bit into our metrics and we found that the bottleneck for this high request latency was the call to pricing service pricing service is a fairly large python microservice that houses hundreds of different endpoints with which are a mix of cpu and io heavy we had now finally narrowed down the offending call and so this was a pretty useful exercise now this endpoint in the pricing service was calling into roughly six different upstream microservices in order to do things like computing the route the eta and the price and so we decided to instrument each of these rpcs with the timer decorator in python and monitored our charts however this didn't prove to be so useful so the next thing we try to do is employ distributed tracing at lyft we use distributed tracing to track the lifecycle of a request across various microservices this provides you with pretty detailed insights on where each request spends its time and we noticed that requests to upstream services were being queued up and were sitting idle for a long time just waiting to be scheduled this was a really useful exercise for us because it allowed us to confirm that requests were being kept waiting for a long time which pointed to a web server scheduling problem and and so we decided to dig deeper we compared our application and networking latencies for all upstream service calls inside pricing service and we noticed a pretty major oddity while doing this our our service networking mesh that orchestrates calls between microservices was reporting that calls from pricing service to upstream services were taking around 15 milliseconds however pricing services application measure latency was on the order of 50 milliseconds which is a 3x difference this was a pretty useful exercise and it had pointed us once again in the direction of inefficiencies in our request scheduling and pricing service now why was all this happening it all goes back to what roy called out in our performance section because our service pricing service had a mix of cpu and io workloads our cpu and tensor requests or in effectively blocking our bike scooter pricing requests from being scheduled by g events event loop amazing at this point we have discovered a root cause and it was time to fix it and so the the first thing we try to do is move our endpoint to a dedicated set of pricing service hosts that would exclusively run office service pricing requests by moving this endpoint to these dedicated hosts we were going to get dedicated unicorn processes that didn't need to contend anymore with cpu heavy requests from from other rpcs and it didn't have to contend to be scheduled anymore the results were exactly what we were hoping for our timeout stopped immediately our flickering stopped immediately and our success rate went up from two and a half nines to four and a half nines and this was exactly the silver bullet that we were hoping for all along so our takeaways for you from the stock are that we really recommend understanding how gun works under the hood it's helped us a ton and we know it'll help you as well understand the correctness pitfalls that you might encounter like the ones roy called out with the safe bank example understand the performance pitfalls that can occur when combining i o and cpu workloads and also if you're interested in digging into latencies for requests distributed tracing is a really really useful tool and if nothing else works try isolating your workloads awesome that's all we have thank you so much for coming out for our talk you