SBTB 2019: Briac Marcatté & Shajan Dasan, Reliable, High Scale Tensorflow Inference....

SBTB 2019: Briac Marcatté & Shajan Dasan, Reliable, High Scale Tensorflow Inference....

Recording: SBTB 2019: Briac Marcatté & Shajan Dasan, Reliable, High Scale Tensorflow Inference....

hi everyone so today we're going to be talking to you about reliable eye skeletons of flow inference pipelines at-taurah so my name is Brock Makati work in the machine learning occur environment team at aura and I've been at Twitter for more than five years hello my name is Shallon I work and the cortex team which is the central machine learning team at Twitter I work on the inference side inferences once you have a model trained inferences when you serve the model and I've been working on distributed systems for over 15 years last six at Twitter react I'll do the first part of the presentation that I will come back alright so the way this presentation is going to be structured is first I'm going to introduce machine learning at Twitter so how you stanza flow like use cases where how when and talk a little bit about context and then children will go and discuss specific challenges that we faced using tensorflow in our inference pipeline cetera so specifically two parts the scale aspect as well as a reliability aspect so before so in order to dive in into machine learning at Twitter let me first introduce the machine learning lifecycle to make sure we have the same vocabulary and we both understand what we mean in the follow-up in the present of the presentation so as you know machine learning is a vast field so here I'm just going to restrict the conversation to supervised learning to simplify things so we're and how we're using machine learning here is we're essentially trying to find a function f for particular use case so for instance we want to show face relevant content to users through a commercial systems we want to improve our understanding of the content that is shared on Twitter and here the function f is finding the relevant content for a particular user and the output of the that function would be the content itself and so finding that function f here is called training the machine learning model and so the way we do this at Twitter is in Python through batch processing and so once you have this model train this model goes and can be served online or offline as well but essentially the model will not change and the function f is fixed and this is what we call infants here and so typically at Twitter this is latency-sensitive and we will rely on Java and you will understand a little bit better why so the training part we typically use tensorflow and the python api that are provided by the tensorflow team with a few tweaks and then Java we have a little bit more tweaks and this is what the talk is going to be focused on so ml at Twitter so I started mention earlier Sharon and I are part of a central team so typically we are not the people training machine learning models but provide the platform for other teams to be able to train those models and serve them in production and our customers are all dete with engineering teams as well as applied research teams so to give you some examples in terms of customers here I've we've played five five examples bacillus is a subset so for instance the boasted two first example a timeline ranking team as well as as ads team is going to be focused on providing relevant content to 2001 the first one is focused on over eleven tweets so when you open the Twitter timelines essentially give you relevant wit from people you follow and the ads team is going to be focused on giving you ever event ad made the media understanding team on their side is focused on understanding better what's the media content on Twitter and adding additional metadata that can be used by other teams at Twitter and the health team is focused on making conversation healthier on Twitter using machine learning so this these machine learning models require machine learning techniques so we are not going to go into these particular use cases but more around how we make the inference pipeline reliable and scale at a high volume so first to understand a little bit better how and where we use tensorflow we have to backtrack a little bit and understand better the twitter ecosystem so a twitter with if we simplify things we have mostly two use cases which are online critical services so these typically run on mezzos which is our custom address that provide efficient resource isolation and the way these services connect to each other is through thrift RPC calls via a library that is called finagle that is open sourced and developed at aura and these are typically JVM resurfaces on the other side we also have offline processing where that is mostly done on Hadoop through libraries called coding it is also open sourced and was developed at Twitter okay so one one key aspect here is within the two tycho system when you deploy your service or where you deploy your Hadoop jobs you get a lot of nice functionalities because 2008 ecosystem and that gives to the developers a lot of functionalities for free essentially so one key change that we faced when using tons of flow is that the set of abstraction that I use are not the same that the one that we have at Twitter so typically fentanyl session format and the RPC format as mentioned Twitter rely on shift versus tensorflow will rely more on the technologies that Google uses support protocol buffer for tensorflow and G RPC for the photo station format protocol buffer and G RPC for the RPC calls and on the language side as well Twitter as mention is much a VM centric versus tensorflow is more focused on Python C C++ with a little bit of Swift so that was an issue for us because it meant we couldn't use things out of the box so what we did is instead of relying on what the tensorflow team provides for serving is we deployed we created our own essence essentially tensorflow what we call tensorflow prediction services and that means packaging tensorflow behind a JVM finagle Streif service and so tensorflow is a native application developed mostly in c and c++ with a little bit of Python and so the way we communicate we stand so far on the JVM is through the Java native interface and typically what we provide to our customers is the ability for them to deploy tensorflow as a prediction strip service and so we provide a binary package and configuration tools that they can use and deploy on Mizusawa we also provide that as a library so that they can also include that in their added jobs as well as embed and so flow in their existing production isn't there existing services so let's have a quick look at an example of how one of our customer can use the our production services here for their use cases so here what you're looking at is the lifecycle of NAIP request that that is done when when you open the mobile phone and the Twitter app so basically when you open the Twitter app you're gonna get a set of tweets that are based on people you follow and that request API request is going to bounce through a list of services and ultimately reach the tweet prediction services which is a swift service deployed with that particular transfer model that lets say the time line team worked on so in terms of a set up as mentioned previously we don't own these strip services we provide the ability for people to deploy that so we see there's pros and cons tell which is the pros the customer teams are closer to their business logic needs so in terms of required SLS they can trick the operational setup to meet their use case so maybe they require more CPU more RAM or the they can tweak their time at setup and this is gonna scale better as more Twitter teams move to tensorflow obviously the one of the cons it's harder for us to control the version of the software that is deployed and it also puts higher operational cost on on our customer so an alternative model would be for us to kind of maintain a central prediction server that our customer could just basically point to and deploy and just tell us you know which model they actually want to be available ok so now I think this was kind of a presentation on the email at Twitter I'm gonna move forward with run on the challenges of inference pipeline at aura Thank You Bree ak I'm going to talk about some of the challenges we had specifically scale and reliability and talk about a few things we did that work for us scale so there's there are a number of different dimensions of scale that we have one of them is the number of just the number of machine learning models we have a Twitter including experimental ones they are in the hundreds and these models are not all the same or similar some of them are in the other if you just look at the size of the model it could be all the way from a few gigabytes to a few megabytes or a few hundred megabytes and they have different model architectures so you have one kind of models for doing things that are kind of text-based and you have another ones for recommendation systems you have things for media processing media understanding they're all like very different and these models are by the time they are trained and they get to inference day they're just a huge graph of computation and you know they it's and and the way these graphs are built are very very different based on the kind of model architectures and this brings us a challenge of you know diversity of different use cases we have to support that inference side with the same code and these some of these services or some of these use cases are very latency sensitive for instance they might be user facing so when you are refreshing your tweet it's you know coming back and calling us many many times even to get a single tweet and then if you are slow the user is going to see a slow response and on the other side of things there are like batch jobs like a Hadoop MapReduce job I've used carding but it's our way of doing MapReduce in scala and those things are very are not very latency sensitive but they varied throughput all of them are cost sensitive so you want to do this with the minimum number of resources then we have many many services so it's in the order of hundreds of you know in the order of hundred services and the other side of scale is the classical scale we talk about which is the number of requests we get we get about 40 million requests per second that that's how much we serve a majority of them are latency-sensitive and our customers have strict as a laws that we have to try to achieve we usually constrained by CPU because it's a huge graph computation and memory as well because some of these models are huge they take up a lot of space in memory one other thing to note is that our trading is can be done on GPUs but it's very expensive to do inference in GPUs because you know because of the scale we do inference on CPUs how do we how do we deal with this we we have it all starts with a customer coming up with a service level objective they have you know they want to serve the p99 within a certain amount of latency and they want to say this is how much budget they have they want to do a thousand requests per CPU per second something like that and that that's a starting point we have load testing tools where they can provide synthetic requests and they can give us a model and they can they can size our system to tune the number of CPUs or they can change things including tensorflow threading parameters and how much pata they want to go and finally they come up with a profile we'll say okay any large memory machine versus you need a lot of machines but with fewer number of CPUs things like that and they can come up with an optimal model optimal size that that meets there as a low but it's not that expensive so we also do deep dives while we do this to understand where the time is spent how to make things faster or cheaper we we have a lot of what we call stats I'll talk about stats in a later slide that helps us kind of tune and figure out how to optimize things latency particularly is hard to do at at our scale especially tail latency that is like you know some some of customers since they are user facing they have three nines of four nine latency require and here are the few things we did that actually worked and these are nothing new but you know among the hundreds of things you could do we tried quite a few things and these worked for us the first one is the limiting D the number of outstanding requests so to take a step back our service consists of sometimes thousands of instances and instance here is a piece of code running on a sandbox on a machine in a shared machine and this is our mesas are a compute compute system so within an instance we can limit the number of outstanding requests what that does is things don't get cura if the machine is slow what if there is a huge load spike coming in and it also guarantees that the requests that are sowed ourselves within latency so if there is a load spike the at least sir what we have provisioned for the rest of it gets cancelled the other one is automatic restart when something just one instance is slowing down that's enough to get ourselves out of SLO because of the tightness allows we have so we have ways of automatically shutting down those slow instances I always I'll talk about that more the last slide the next is very specific to tensorflow since these some of these graphs are huge it takes a while to load them up so you so when you when you are publishing a new model you don't want to be serving as soon as the model is available on the inference machines you want to first warm up before you let cadiz before you actually serve user curries so the way we do this is two ways one is our customers can along with the model they can give us a set of fake queries if you may that we will play we play and then once the model is having acceptable a latency will will let the user request get through it that's one way customer supplied warm-up requests the other one is sometimes they cut it like you to just make this easier for customers sometimes we don't even require customers to give us this what we do when there is life serving happening we just remember a few requests on the side and when they load a new model or a new version of the same order we just replay those requests to warm up the model and then we take external requests to that so those those are two of our techniques we use the next one is backup request which is quite it's very effective and this is a client-side thing that's not done on the server so when the client sends a request and this is part of the final client library that's use it to their so you can do you can convey the client so that it it waits for a little while let's say up to its p90 its 90th percentile latency and then it sends if it doesn't get a response back it's going to send another request potentially to a different machine because you know there could be thousands of machines serving this and when a response gets back it just picks the first response and ignores the next one if it ever comes back so that's been hugely effective in reducing the day latencies the next one is very specific to how we build tensorflow but nevertheless it's it's a good optimization this utilization optimization I'll explain it this way just to recap as Biak mentioned we use drift as our civilzation format and within our service usually when you get a request in your service it gets utilized a stripped and UD civilize it within your service in your Scala or your Java code and then all of tensorflow is native it's C++ code so we need to get all that information down to the in C++ layer we use gni and one thing you don't want to do with J&I is used a very very verbose structure and it has to marshal all of those little structures and all the sub structures and maps and arrays and all those things in the j'ni into the native code so to avoid that what we do is we don't de lies the request at our Java layer or Scala layer instead like we push a byte array all the way down to the C++ code and VD code we do the trivialization within the C++ code within within native code that that keeps the Java layer deal with byte arrays as opposed to having to marshal a lot of structures that saves CPU and improves latency as well the next topic is reliability we have quite a few reliability challenges standard ones memory leak both in the Java code as well as or Java or Scala code as well as within the C++ code both ours as well as the open source sometimes in the math kernal library sometimes in the tensorflow cola it's it's all over the place the exponent is performance degradation things break or performance gets affected either by check-ins that we do in Twitter or sometimes sometimes when we pick up new bells of tensorflow again turns flow upgrades and some of these degradation and leaks doesn't happen immediately like it like once your service is running for two weeks maybe three weeks that's when you sometimes start seeing these things just just a background on liability just to see the problem just to introduce the problem so in our case we have hundreds of services even in let's say we designed everything to be reliable for let's say one for a one percent failure probability of failure let's say within a month you can run with one person failure probability but given that we have about hundred services that becomes about 63% if you if you compute the actual probability for a month and so this is just to show you the scale of scale of the problem as far as solutions is concerned one very specific lovely problem for us this performance and so we have functional testing we have unit testing all those but in addition we also have what's called performance guardrail which is a set of tests anytime we check in code into our code base this gets automatically run as part of our CI and what it does is it has a golden set of models that represents most of the T to use cases that it launches these models on largest services with these models and it has a set of requests that it it sends to these services and then record the latency RAM CPU usage things like that and then it compares with a baseline and these bugs are or these performance issues are caught before it becomes a problem and we also like uses any time we pick up a new version of tensorflow it's been quite useful in finding issues during upgrade so the problem is if you don't catch it at our our layer it gets to our customers and it's quite it's quite expensive to roll it back because they're trained with these new versions and then they have to deploy the model and the training is like very expensive thing to do and it's also delays the whole upgrade to a new new feature or it delays a lot of things if it gets out of us once you know there is a memory leak we can we can know there is a memory leak using stats and I'll talk about stats and the last light then the problem becomes how do you fix them so to fix them we need to find we need to know where the leak is coming from so it could be from JVM code in that case we use J hat which is which comes with the JVM so J hat what it does is it you can ask D I had to give you the snapshot memory usage at any point in time it gives you a histogram of objects like object of type ABC and the count number of instance of that object that's life after a garbage collection and we can configure our service to take JVM histograms snapshot over time you can say take a snapshot every hour or every five minutes things like that and then we have a custom script that goes through those snapshots and then figures out which object is increasing in count and that that's been helpful in pinpointing where the leak is coming from if it's native leak in the C++ code then we used the email log which is a allocator in and you use that in it's quite easy to use in Linux all you need to do is set an environment variable and then launch your process it overrides the c++ linked allocator it uses the email app and then you can instrument the malloc to give you leak reports that's been working very very well for us the next one is not a memory leak but it's it's a memory fragmentation which shows up as a leak because your memory keeps growing the native C alligator is pretty good at allocating without taking a lot of logs and what that ends up doing is it's going to fragment a lot of memory and and you're under service for weeks at a time and you'll start noticing it both J malloc and TC malloc work with reducing fragmentation with a small cost and performance the other one is like simplify you automate your deployment process of civil T I talked about stats before anything we write a service at Twitter we can export stats which are like just number like how much memory are we using monotonically increasing numbers like you know how many requests do you get and these and and we can also have histograms that measure performance latency things like that these tasks gets pushed to a time series database and there are quarries you can write against the time series data base to create dashboards that helps us see how things are going and these quarries can be hooked up to alerts where where things go beyond SLA or is at a risk of going beyond something some threshold then a person can be paged and even better we also have automated reports where some when something like this happens we can configure to have ways of repairing usually it's just restarting an instance I got to be very careful doing that because it has to be rate control otherwise you may have bad things happen things you know all deciding to repair at the same time so we have ways to rate control it it also helps it let's say you have a memory that happens in a month it'll start repairing maybe a day in advance and it'll slowly get to the whole cluster and they don't all leak at the same time if you stagger it that way so we discussed in conclusion we discussed some of the challenges some of the uniqueness of the tooter ecosystem we talked about scale reliability and some of the things that worked for us thanks and as always we are we are having ml engineers already and we have time for probably about one question if anybody has a question [Music] hello I noticed you separated the candidate said generation from the inference that specification of that communication get formalized by your teams as well so that there's less room for miscommunication between those two components okay so the difference between the features that's used at inference time versus for training is that where you're getting at just before the inference they're having to be not just the features but a candidates that need to be scored okay so this gets to the discus to the to the application that's using machine learning so for instance some of our models are such that they have multi levels in it so just to reduce you have lighter models that are quite fast that can deal with a lot of candidates it brings it down to the candidate then you do a heavy model so that there are systems that are built where you have these multi-phase one light model for for narrowing things down and then one larger model that goes that's a much more complicated model that actually there's a lot more company but on a pure pure set of candidates yes we have multiple services that work that way already and I believe that is all the time we have thank you for coming thank you [Applause] [Music]