Multi Runtime Serving Pipelines

scale.bythebay.io: Stepan Pushkarev, Multi Runtime Serving Pipelines

Recording: scale.bythebay.io: Stepan Pushkarev, Multi Runtime Serving Pipelines

you yeah hey everybody thanks for coming let's get started there are two ml talks in parallel so we're lucky to to talk here so today's topic is multi runtime serving pipelines and city of hydrosphere i/o and this talk is be mostly based on our open source experience and the which is an active development right now and yeah as a company we provide solution and open source solutions for deployment serving and monitoring machine learning models so that's that's alignment of that topic and/or expertise and so on so I believe it's pretty obvious for everybody that the machine learning project has two major phases the first one is offline which happens offline in development training and the second one is when you actually push your model into production and it's somebody console deployment serving or scoring on in inferencing so these are basically two fundamentals like a phases that we distinguish and today we are talking about the second one and when people talk about the model deployment they usually mean the minute deployment of just one model and providing some API to them for scoring like you deployed a model for image recognition and you recognize these images but let's take a look at the real-life use cases when you need more than one model to be deployed in a end-to-end application artificial intelligence application but in this case we have a like call assistant application when we have it listens to the conference conference audio it filters pre-process it recognizes speaker it does some context aware speech to text translation it extracts knowledge from that text and then provide some action for example if it's a sales call it could provide like drive this call and offer a discount so this is I'm not providing you a reference architecture for this type of use case but just to give you an idea what type of meta pipelines I am talking today yeah another another use case is like for example you are up eBay and you have a millions of products and you would need to match this product for different purposes for the application for pricing optimization or any other any other things so you would probably have an image model text model title model and maybe other models that will work in and sample to provide the the final result the final inside to D to the business users so this is again you will have multiple models working in in the same context and in the same pipeline so that was like my high level high level context let's take a look at the model deployment from the operational perspective as operations as software engineers we see the model artifact that has the following the following components first of all it has the actual model data the weights and kayfetz ends of that it has some runtime where the model runs which might be c++ java java whatever there is bunch of dependencies from machine learning frameworks like tensorflow spark ml and others and there is a model that is responsible for api and the infrastructure so it's and i would like to highlight a little bit this the important of a pil on logistics you see that a limited set of requirements for this type of for this component that is extremely important and it is required when you push a model to production so it's a it has like HTTP RPC support it should provide an API to cough cough link or any other streaming streaming engines to consume consume messages from them and score it in real time it should be discovered in that in your micro services ecosystem as a as servus yeah and there are some bunch of other stuff which is which is I believe you are all aware about and the one thing I wonder I wanted to highlight as well so for when you we talk about machine learning deployment the importance and the the requirements for the the level of requirements for this type of API and logistics is much higher than for ordinary micro-services the reason why so for example when you have an order in Java micro service you might have a variety of unit tests you might test it in offline in test environment and staging and you might be sure you could be sure in 99% that it will work well in production with machine learning model this is a usual case when the model works differently in offline training in offline training set in in production so the precise monitoring monitoring not only of a latency but the quality of the predictions is extreme extremely important so we you could not physically push the model to production without all the without covering all these requirements and this is like the example housing how the monitoring could be implemented so we continuously continuously listen to the inputs and outputs of the model and we do some probably clustering or anomaly detection there to identify different different mode of degradation or data drifts in production so for example like the the same model could be could work well for males and could not work well for females and this could be discovered right online and in production by this by their having this like smart monitoring in place we are basically shifting experimentation phase from offline research to real-life production and if we will multiply these requirements for the the API and logistics to the number of runtimes we might have four different machine learning frameworks will have like infinite set of implementing different implementations in flask spring both spring boot and other and other frameworks that maybe it was some with something at some ad hoc metrics metric systems attached to that and have you guys attended the is to a workshop yesterday does anybody okay I guess for those who attended UC the really interesting alignment of the problem I'm sure I'm just just I'm describing and the story that and voy and Easter is are telling so having multiple runtimes with different dependencies different languages they require a unified way to provide an API and logistic and routing introduction so that's why we took the antboy and we attach that as a sidecar to our main main application a main application represents the actual model deployment and delegated all the API when logistic step stuff to the and boy so our our main main model docker container with main model is really tiny and clean and just provides a business a business logic and actual serving serving and or inference and influencing logic and it doesn't know about entire world it knows just about envoi which is sitting you know which is sitting right right behind you so it's a high-level architecture I will skip it really quickly so we like web strongly believe in in function as a service paradigm so we basically treat a machine learning model as a function as similar like AWS function that could be registered in a cloud and it could be triggered from anywhere of your application so we have a functions registry we have an actual a mesh of serving runtimes powered by Android and the integration with infrastructure with CPU GPU and spot instances so and what are the actual pipelines so the models we deployed the functions we deployed to the cloud they are actually actually a building blocks for the for the applications yeah and applications are like virtual public endpoints that for the models or compositions of the models in that case in that case it's just for a user experience let's take a look what we have here but for demo for those who of you who familiar with with tensorflow and the image recognition we trained SSDs single shot detector model so it's it's just the state of the art image recognition model and architecture so we basically just describe recovered that from checkpoint and saved to the to the model repository to the s3 here and this s3 is s3 bucket it's mounted as a repository for her ml lambda the the open source product that we are which is in active development right now so we have a like we could discover a couple of models that are also we sit there in a in that repository we could just browse their versions and this is the actual SSD single yes see what what's happening on the screen okay sure it should be better so for this case we have we have and you could you can test deployed the models when you deploy it you just provide you you select the the actual environment you want to deploy it and on GPU on CPU or and runtime that I've talked I've taught talked about and this particular model as a single detector here there is one version which is deployed on GPU so we can send a pixel there and and get result results basically a bounding boxes of the object objects detect detected on that image so this test doesn't mean doesn't make sense because just one empty pixel and in other model is deployed on CPU on simple CPU and now we want to build a pipeline the pipeline has a the actual model scoring and post-processing stage so the pipeline description looks like the following it's pretty simple you have a SSD pipeline you describe the first stage that you have you wanna you wanna route the 50% of your traffic traffic to the model one and another 50% of the traffic to the model 2 and these are actually GPU and CPU versions of the of the same model but they have just physically deployed and in a different runtimes here and you have a stage to the second stage which is like we call it in this case an SSD labeling it's just post-processing and it's doing something with a model output here so and one would deploy this kind of pipeline we we see where it's it's being displayed here and now we can send the traffic to that pipeline so it's to this application it has a public endpoint and also it could be we could send the traffic to the Kafka it's it's also will be like consuming that traffic from Kafka's scoring that yes executing executing a pipeline and sending the prediction results back to the other Kafka topic so in that notebook we are I'm going to so this is by the way the the model output looks like this it's a pretty canonical missing detector model here so here we have just a while loop and we are sending sending like random random images to our application and it started to to get scored yep so you see that new images and you scores that appear here let's take a look at what happens with the metrics so as I mentioned the metrics are really important and the our pipeline should emit as much as as many metrics as possible to for monitoring for tracing and in this case for trace tracing we have a Zipkin that receives that these metrics and we can observe some latency wait wait and see metrics here so and we can trace every request and just browse out how long it leaves in a in what stage so here we have it's six seconds it's obviously a CPU CPU instance which is slow it's the same model on a CPU instance and we have we can filter by shortest and this is well like it's basically scores a GPU instance it's just below 300 milliseconds of course it's not the state of the art it's not a performance benchmark because the model is not highly optimized but it's just to show the mechanics how can you deploy the models in different runtimes and build a pipelines from the Builder pipelines from these models so the couple of follow up questions I could cover yeah so why not just one big neural network you probably everybody is excited about the Google experiments and Google Translation so with an Nvidia experiments with self-driving cars when they just trying to Train one big neural network so it's it's a good research it's a good good good cases but it's not always possible and it's yeah always it's require often it requires a lot of black art and drought there might be and people usually start with a rule-based models then move to the machine learning models then move to then probably merge some stages into entrant and to end to end a neural network and basically every stage of more of the meta pipeline i i've just i just demonstrated to you might be in fact the entrant neural network but the the stages could be yeah there might be other stages that are new entrant network in their own so why not just one Python script because usually when like data scientists they do a training pipeline they just write a Python script it's just a broad array of functions and they just execute it in a chain and yeah of course we need a modularity and stages could be the stages of these pipelines could be developed by different teams and you can even order particular voice speech-to-text model from a third party provider that is that has a like in each expertise in that particular case so and you anyway you need to assemble you have to you have to do a little bit plumbing on sticking all these models together so we need more better versioning we need to version every stage of our pipeline and to track down the changes that are happening in an inner life so and of course some we want to like take a lot of some cool stuff from microservices world like request shadow shadow in cannery testing and other stuff so the rube you've probably heard there is like tensorflow serving and so why could yeah yeah no when why we're not using tons just at under foster so the first of all there are other machine learning frameworks rather than tender fostering you might use deep learning for J you might use psychic learn and actually the data scientists they tend to experiment with a new framework every day and you could not just limit them to use one JVM based or tensorflow based models so in order to provide a freedom we it it would be great to support as many frameworks as possible and we need better versioning if we dig deeper into the tender fostering they kind of reinvent versioning we wanna we like like have a darker version and it provides more of a natural wave to version your models and it's it provides better immutability of your models in terms of flow serving whether you have to it basically it acts as a load balancer it reinvents load balancing and reinvents versioning so well and this is already exists in a been up in other world so we want to utilize the best practices from the from the rest of the world rather than like following the tensor flow serving para paradigm so and we are we don't want to deal with state for example when in intensive fold serving when you have a model version one the first version of your model and you want to load the second version of your model you have that intermediates they eat that okay the model is not being low that yeah that they it should know it's not ready to receive to receive a request from the from the outside of the ball you have some like logs and and mutexes there so we don't need to do deal with that if we have a dr. pepper in a perversion model version and we just scale it horizontally with all the cool frameworks we have we have in place yeah actually that's it I'm done have just one minute I updated 30 seconds for questions I apologize probably after the previous that introduced you to deep learning it could might be a little bit advanced that after introduction we we jumped right into production so please yeah thank you guys [Applause]