SBTB FinagleCon 2015: Puneet Khanduri, Automatic Correctness and Performance Regression Analysis
so how many people here remember around November last year Twitter signup was broken for this is awesome usually at this point there is someone in the room who raises their hand and then I have to say that never happened and this is part of the reason why it didn't happen yeah so glad to be standing amongst you guys today yeah I did this at LinkedIn and then a bunch of other places and everybody like everybody every time there's one person who raises their hands like Falls for it anyway so at Twitter at the very least I mean I imagine this being the case across the industry you make a change to your service and then you're wondering whether you've broken something inadvertently and all your unit tests are passing but you can still like some more confidence and you know what what else can you do if you're an SRE in DevOps and you have an even bigger problem then you know you're thinking this is somebody else's code they may change this to it I don't even understand what the changes are and I have to make the decision as to whether or not this code is okay to be deployed or not and maybe I'll canary something and you know see how it behaves compared to the rest of the stuff and and you know but but basically pushing changes out to production is a scary thing and anything we can do as developers to minimize that risk we should do our first line of defense naturally comes out to be unit tests tightly scoped tests let's say a single method or a class in isolation and for the purpose of this conversation let's say we focus on path coverage so let's say you you're you have a method that you're trying to test that has five possible code pads then writing one test gets you 20% coverage and you know you write five tests and you're done but oftentimes it can take you more time to write good tests then actually you know writing the code that is being tested so I mean it's a little bit expensive but it's still doable then you get more aggressive and more ambitious with the you know the level of certainties you want about the behavior of your service and you're saying that you know when a request hits my service it's probably going through you know in this example six different methods and each method has let's say five different independent code paths so the combined you know number of possible code paths or the cyclomatic complexity if you will for for something like this would be off the fifteen thousand possible pads and no one is probably motivated enough to write 15,000 tests for or anything so so basically right like writing a test by hand at this point is you know becoming you know less valuable you only get 0.03 percent coverage and if you get to the point of integration test where you have let's say four services in your stack and now you want to look at the composed behavior of all the services then you know it's so infinitely large a number that you know if I take the example of one of our largest running services in in production you know it would take us about a thousand years processing requests at the rate or running tests at the same rate as the services you know processing requests to get full coverage that's assuming that every request exercises a different code path so the point being that you know it's extremely hard to do testing as the scale increases and it becomes more and more expensive to get good coverage as the size or complexity of the system increases so we need more advanced automated techniques that do not rely on developers to write tests none of this is to say that you should not write tests you should write as many tests as you can but you should become you should be cognizant of the fact that you know the impact that you're making at every level is becoming increasingly small and you know tools like if we can help you when you get to those scales and those levels of complexity so the approach we decided to take with Diffie was that you know if we look at the anatomy of a test you have you know an input that triggers a certain behavior and then you make some assertions on that behavior right so the the behavior that we care about the most is one that is most likely to be exercised in production which is why you know we at Twitter prefer using sampled production traffic to drive that behavior for untested code and then the question becomes okay now I'm driving the behavior in my intested code but how do I make the assertions that the behavior is right and the answer is well you compare it to the old version of the code assuming that you know the old version of the code that's currently in production did not have something bad in it and as long as you know the behavior is similar enough to the old code then you haven't had any regressions well sites are the problem of like setting up environments and and and and those things for the purposes of this conversation but if you're interested in that stuff come talk to me afterwards so this approach however inherently has a big problem of being extremely noisy so when you're let's say you send the same request to your old code and your new code right your old version of the service and the new version of the service and in the response the service returns several generated time stamp right those two time stamps will never be the same and so you think that hey these responses are different there must be something wrong with my service well that's not that's not the case right there can be other sources of of noise if your code has random random generators in it or if you know your instances that you're testing are backed by live production data that's moving then you could have you know right in between read contentions and those race conditions can themselves be a source of noise in the responses that you come that you get back so without having a good story for noise cancellation and this sort of an approach the work on the developer side amounts to looking for a needle in a haystack and noise cancellation is paramount when when you're trying to do something like this so how do we do that well we say that you know take a bunch of traffic and multicast it to three instances of your service so the candidate here is running your new code the primary and secondary are both running the same version of the old code right so there's nothing different about these two instances and the point is that you know when you do the comparison between the responses that come back from your candidate in primary you see a bunch of raw differences but when you compare the old code to itself by comparing primary and secondary you will see differences here and those differences will highlight which parts of the response are inherently noisy and should be subtracted out from the raw differences so that the developer can focus on the filter differences that cannot be explained by noise so let me do a quick demo here this is what what if he looks like for one of our services this is an example service that's running and you can see here that like all the end points are listed for this example service and you're seeing you know the entire response object tree and you're seeing aggregations of of all the different kinds of differences that that this service has seen so if we look at this one for example you know it's a very small small difference if you want to deep dive and see what exactly triggered the behavior to be different you have the full-blown request the full-blown responses for for debugging purposes and the exclusion logic is what allows us to say that okay you know this whole stuff is okay to ignore because primary and secondary are disagreeing with each other for this particular type of error almost as often as primary and candidate are disagreeing with each other right so by doing that comparison of aggregations of statistics across different types of errors were able to do you know a good job of of noise cancellation and so you know coming back to to the developer you know the the real engineering organization side of the world the the natural thing that that came to us from our customers was that you know why don't you automate this I mean I love being able to test my branch against master but I also want to see what's going on with master compared to what's currently in production so I want to come into work every day and receive an email every morning that tells me that hey last night I did this comparison for you and you know here's all the changes all the differences in behavior that I've observed and if any of these differences are unintended then you have the opportunity to take pre-emptive action against these things before they get deployed to production so we did that and now we're able to apply this whole noise cancellation thing and send this reach these rich emails out to our customers and you know you can click on one of these and it takes you straight to what the what the different what the error is and you can deep dive and debug as I just showed you guys so moving on the next thing that comes naturally as you're setting up as your multi casting traffic to you know all the new versions of the code as you're also replaying the exact same load in lockstep against your new code and your previous code so can you then look at the metrics coming out of those boxes the performance metrics coming out of those boxes and do some kind of analysis to figure out if there's been a performance regression right and you know if you're on the final stack you will see that you export a lot of metrics and it becomes humanly impossible for someone to eyeball these individual metrics and in fact humans are terrible at eyeballing metrics so you want a machine to do this job and go through all the thousands of metrics that your service is exporting and highlight the ones to you that actually have significant deviations from the old code right but in this world we start thinking about the comparison in slightly different ways because we're now talking about performance and there's a lot more susceptibility to noise so our candidate now becomes an entire test cluster so you have like you know n number of machines in the test cluster and then your primary and secondary become your reference cluster so you have you know n number of machines in your reference cluster so for the same time series type you get you know a cluster of time series from the test cluster a cluster of time series from the reference cluster and you build this classifier that is able to analyze all of these clusters of time series and make the decision as to whether or not you know there's been a significant deviation in in you know the metric the time series data and it categorizes the time series as past ignored if there is insufficient data or failed if there's definitely something wrong with with something like this right and so as we were doing this whole classifier analysis we you know built a whole bunch of different classifiers to see which one worked and turns out each one of these has its own set of blind spots it works for some things but then doesn't work for the other kinds of things and really what we ended up doing was coming up with an ensemble approach where composing these metrics in in a meaningful way gives us the ability to reduce the false positive rates significantly and give meaningful data back to the user so in this case what's happening is that I'm saying you know there needs to be at least 50 samples in order for the classifier to do anything with this then there is this relative threshold classifier which just which is just taking at you know the relative deviation between or sorry the relative difference between the variations within each of the time series clusters the absolute threshold classifiers taking into account the distance between the medians of the individual clusters and finally the MADD classifier as the name suggests is looking at the median distance of each of the samples from the median of the cluster and that probably does not make sense right off the bat so I encourage you to look at a fun Wikipedia I'm not lying this is a real thing so what does this look like to one of our customers so along the wain of automation when we send you the email about all the correctness analysis stuff we're also able to send you you know an email that lists all the different metrics that we flagged so you know in this example service we saw that you know there's off the order of six thousand plus metrics and of which we were able to filter the the set down to 32 so if we click on you know let's pick one randomly and and click on it okay time today right so you can see here that the primary and secondary are pretty much in lockstep with each other but the candidate is significantly right so this is the classified doing its job and saying that hey you know I had a bunch of test time series a bunch of reference time series and I know which ones the candidate in which one's the the primary secondary so I'm going to use that context to give you feedback on you know this particular metric whatever this is hopefully this is something that makes sense to the application developer but you know from diffie's point of view all we're doing is analyzing raw time series data and highlighting these sorts of metrics back to the back to the user so we have some more work in progress along along these lines one question that might come to mind is you know what do you do if you have write pads right what if how do you replay write traffic to a service what do you do with the environment when the service has downstream dependencies which inevitably it will and so you know Senegal has this really cool feature where you can you know add tracers to your service and use those tracers to not only intercept the request but also all the downstream interactions triggered by that request right so in essence when we do our recording in production for the traffic we're also recording all the behavior or all the interactions that happen with the environment right and so as a result when we come back to staging to do this sort of regression analysis we're able to replay the environment exactly as it was in production so we have all the our pcs downstream our pcs recorded from production we also have the timing data recorded from production so we're able to you know simulate the same delays the performance characteristics as if it was production and then you know we're we're able to even process right paths using you doing that sort of an approach but again like this is work in progress this is not something that's that's in Diffie right now questions and sure not yet we're working on it it we we are very close to it we are very close to open sourcing it so in terms of seee there's been a few efforts within the company where you know they've wanted to automatically promote artifacts saying that if he didn't find any regressions this is clean and along those lines we don't have any tooling that comes with the box out of the box but we have exposed api's so you can get sort of the aggregate results from defeat with an overview report and then based on the report the external tool can make the decision as to what to do with it CI systems tend to be diverse so rather than go vertically integrated in that direction we decided to expose you know API N and leave it up to the developers to to figure out how to consume it yep any other questions all right I'll just do it in order of proximity right good question so does the act of measuring the system affect the performance so we're only when you say measuring I will assume it's like sampling traffic right so we only sample traffic from production but all the experiment that we're doing the comparison that we're doing is completely isolated in staging right and there's no sampling going on in staging whatsoever and you know there is a sort of you know the concern about production performance being degraded as a result of this recording happening but typically what we see is that you know there's a very small fraction of the traffic that needs to be sampled anyway because you've got you know thousands of machines in a cluster and if you sample you know 1% you have 10 instances worth of load and that's more than enough to do whatever kind of comparison and barely shows up anywhere in the performance of the production systems when we're replaying and measuring the responses we're doing all of that at the networking layer as an external client right so there is no impact on on the performance of the thing that is being tested itself someone at the back had the questions I'm just gonna go up to them real quick changes Oh change a piece of software I do it because I want to change some behavior excellent question so we don't have a story unfortunately the story is that you know deaf ears is just trying to highlight any changes in behavior right and if it doesn't find any differences then the artifact may be for example automatically promoted but if it finds any differences but if those differences might be very well intended by the developer and so human has to override and sign off on those changes and say that yes I know what I'm doing this change actually needs to go out in production but yeah I mean unfortunately there is I mean no easy way for Daffy to figure out that you know a difference that it's seeing in behavior is intended or not oh yeah so that's again a good question so we currently do not distinguish between positive regressions and negative regressions so if for example you know you work really hard and you know your system is now processing at half the latency and the p50 is down by 50% we'll flag that as a regression because we when we when we deal with time series data we don't take any domain knowledge from the developers we're just like you know taking everything and we could you know potentially do some some smarts around okay if the name has the word latency in it then you know flag things as green versus red regressions but unfortunately right now we don't I mean it's something that we could look into though yeah any other questions