Model Monitoring in Production

SF Scala: Stepan Pushkarev, Model Monitoring in Production

Recording: SF Scala: Stepan Pushkarev, Model Monitoring in Production

[Music] yes so welcome everybody my name is Stefan good craft today I'm going to talk about model monitoring and production just a quick intro about myself why actually talking about this topic that's a 2016 I posted that blog post and Katie nuggets just was trying to socialize evangelize the the fact that we basically as the community we need better better tooling more tooling for in this case it's just big data but actually what I meant is machine machine learning and data science what what would code right now and the few takeaways were from that blog post was we need tools evolution better tooling as I mentioned education and cross skill so machine learning engineers will learn darker will learn AWS cloud etc and improving the process so like DevOps like at that time I was calling it DevOps like experience for machine learning so we needed that in to streamline the processes to describe our workflow machine learning workflow so it was it was very high-level blog post and ideas but then materialized into a few open source projects that we started and we got some traction from the that early days and then community and actually spun of the the company that called hydrosphere which is the messaging was also very what kind of broad of pay huge do something better something good in machine learning space for like better tooling and other things that I mentioned then we were categorized by different analysts and like Gartner like an analyst as a model management company it's good to be categorized somehow so here's the you can introduce yourself and say I'm like I'm running a model management company yeah it's served by hot place still pretty wild you see many startups many tools open source cloud cloud tools and this talk is basically based on our journey that we went through from the simple tooling to doc your eyes the models provide a nice nice interface for machine learning engineers and data scientists to more sophisticated things like okay now you're in production what's next and the first thing that we we face in production is actually the monitoring whatever with the deploy at the end of the day it's everything is software so to be monitored to be alerted to be debugged and observed in production so I would use this sub driving car analogy just for the sake of like simplicity here you remember these things from way more the in back 2010 they reported the kind of the they reported that ween they made those loops in Bay Area in sub driving node and at that time it was considered as a real win so we were expecting to have our self-driving cars in our row on our road pretty soon and after that they kind of they drove that the screenshot was taken about a year ago I think right now it's like 10 million miles for further testing and in self-driving the self-driving mode here so and the question is where is the last mile so we drive with dry of weeks floor we explore that inference space we train our models similar simulate the traffic in a kind of innocence with we test our models in a simulation environment and real-world environment and it kind of never ends and now it's 2019 and we have this launched on a very small quite like over over fitted area maybe I would say like that great the greater Phoenix area they basically over over feed the models to this specific area so just they learned every kind of feet of that that space and the that's and this is this is like way more it's a Google it's like the Google's of the world they still solving this these problems of exploring and uncertain uncertain inference space here for smaller smaller companies like any enterprises this is this is still a huge issue of course and just for the let's have some fun just to understand like visualize how big our inference space this is Vietnam and by like yeah if you can imagine the self-driving car how much how many years do we need okay that's like yeah let's stop it so you know where I'm going - yep so that's kind of yeah you can imagine how many years you need to basically to learn this type of like culture - this type of environment for to be to deploy a reliable sub driving car there so of course a bit I'm kind of I'm using this it's super like edge case but this is just to illustrate you like the that journey from the Bay Area loop to exploring different or urban areas or over in different states and different cities in different countries and different cultures and we did some also simplistic tests we took a Berkeley data set for self-driving cars the data said that that his has been taken mean girl and Berkeley trained the image recognition model one of the resonate model as I remember or mobile net here on that data set and then deployed this model in in a different urban area it looks for our for us human beings it looks pretty much the same but we just we were like researching how the model will behave there and if it's possible at all to identify that drift between Berkeley area and an another area that that may look there is the same so we what kind of we use that so the and we happen to have just a slightly different traffic lights there they have this like green error placed in in a different in a different different Square and all of the sudden our like models that we with there are like canonical architectures like resonate and others that you know they're not reliable enough to and that models and the datasets that we that we train our models with they're not reliable enough to detect that like very slight concept drift between differ to different urban areas and we were we were experimenting with different models like monitoring models that actually identify this constant drift between Berkeley dataset and other different data set so that that was that experiment led us into more general approach general idea how can we basically build a system that that will help machine learning engineers or machine learning operator operating team to identify the those edge cases as we as we kind of go live in production and feel that those edge cases back into the training process so we can close that loop between the first model the first model in production and the next model in production so very kind of toy example here as well just just to kind of illustrate that concept drift things so there is a you can imagine there's a classifier that can classify between the orange circles and green square green squares so everything else that is beyond this data that model has learned so far is basically insurgency we we don't know what's there the there might be some gradient edge case in in between those those clusters of the of the data we've seen so forth then might be a new concept that we also know seen there just a new traffic light here there might be a concept drifts of the world the world is being changed over time so we also should be able to detect us now the title of this flight is it's not correct but anyway so just to formally formulate the kind of machine learning problem here yeah if we talk about like image data we kind of prefer we research that problem in general but the image that is just easier to illustrate here so and the image image data is the most complicated from that from from all the different use cases where you can imagine here so so yeah in produc for like we used for instance like the monkeys but the black faces for training and then all of this all of the sudden we have these like monkeys put a red faces in in production and can we can we basically just tell data scientists tell machine learning engineer hey there is a new concept here so a bit theory so what what should we monitor in production for the machine learning models of course system metrics latency throughput it's a it's a software we should we should monitor be the kind of that boring metrics that I am calling boring matrix model metrics was there some like first-class citizens are like quality metrics like accuracy the confusion matrix etcetera if you have ground truth labels available so we can like continuously recalculate this the same the same metric that you used you used in in training process and just plot it over time so that accuracy is not just one number that you have accuracy for training you have accuracy support for testing you have accuracy for different like maybe like evaluation data sets and you have accuracy over time and production that's easier but not not always kind of possible then what else you can do you can monitor your inference output and do a profiling for instance if your model outputs the classes we can learn that distribution over time and say hey like in training in my validation data set where the we have I have like particular distribution of my classes and in production it is different is it an issue maybe yes maybe not you just you just probably need some like insight hey there is something different and as a data scientist you can you can take a look and make it and make a decision out of it some other data metrics that you can monitor you can work with so poor request metrics so each individual request could be anomalous for instance or it could be out of the certain range there are batch metrics that could be done on a kind of sliding window so for instance like he s test so we you can apply KS test for Wichita feature or all of the features and basically compare it with the training distribution so KS test compare comparison the two samples are from the same distribution tamo Golf Smirnoff test statistical test it compares save two samples of data are from the same distribution no no wait not a/b test just it's a statistical test you can Google KS test so the like coke coke Calma Gaurav Smirnoff just so the there is a and we call it overall metrics so you can like look at your training data as a whole and at your production data as a whole and say hey this in training data I have a particular distribution of my males and females and in production I have a different distribution it might be good it might be bad but you should like this that's that the more static picture usually talks a lot and provide good insights what's going on in production in general why is that okay so what methods we can use to to track this on super per request metrics ideally it's gonna be like unsupervised so we don't because the you can imagine that for instance you have your model have like a hundred features as an input and for every feature we have to like monitor that teacher and you have every feature would generate like ten different metrics or twenty different metrics you it's super time-consuming to specify some ranges for each feature and some expected values for that feature so the better ways to do in unsupervised way so just to learn these distributions and then either distributions or other like snapshots of overall training data set and then use it in production in an unsupervised way so some cluster II we can use clustering we can use tree methods like isolation forests some PDF PDF estimators we can use like self-organized maps for for this actually yeah there is there's also like image and text specific math methods basically the idea is is to extract features and apply tabular methods to that those features afterwards so and and there is also like ganz I have some slides about Ganz in general actually there are many many methods you can use to kind of to monitor your your your data in for the introduction I'll show you a mind map we have so with like about hundred different different methods there and yeah and auto-encoders I'll talk briefly about Odin colors and games now why is that for fancy animation so unsupervised batch metrics it's for for instance for like just tabular data we can you can calculate missing values mean max histograms quantiles all that all the statistics you can gather you basically track it it's super it's super cheap and yeah the good thing you can also it could be calculate and get weighed in in real time in production so the for image the profiling image profiling profiling it's a bit different but also pretty cheap and kind of achievable thing is like you can calculate your brightness colors Brisky and and other like just just a high level high level characteristics of your images so it can give you some insight for instance if you're like if your camera is yeah like blurred somehow and it may affect your the quality of your predictions and you just don't know about that or the the like the the hardware thing that started to produce just different different just different even images in different resolutions sometimes it's the resolution actually increases and it it affects the quality of the predictions in them in a bad way the for text also like text specific profiling just monitor sentiment monitor unknown words one a group part of speech distribution also very general picture but I said as a machine learning engineering don't wash your model every day in production and if you can if you just look at this once a month you can just briefly briefly the like decide if it's good or bad like it should it should should it be like taken care of or not also variational auto-encoders and Kia's tests on unwrought features so it's also like batch matrix I mentioned it before there is open topic topic for multivariate if comic or comic or of smirnoff test but it's partially kind of covered by autumn colors because auto-encoders if you like do if you encode your your data it's basically you can grab you can think about this as a kind of statistical snapshot of your data and of course all those like metrics and profiling metrics you could you can track the changes of like mean of covariance and other like a change of missing values of friends like yesterday I had like 100 missing values do they have like thousand missing values it's something that total work look at so each match each statistical metric that we calculate here in a profiling stage is actually a metric to track over time so what I would I mean like the the that and at the end of the day the single machine learning model it actually generate like hundreds and hundreds metrics to track in general yeah so feature by feature the multi variant KS test is is super computational heavy it's kind of there is there is no there's no currently a solution to how to how to like do that at scale so for I mentioned it as a like possible solution but it's not a we've we've never used that in production so as I mentioned like many methods i and the challenges there are generic methods and there are very use case specific metrics like model specific metrics here a few few words about against and the concept drip detection in for concept detection basically the idea is to train your discriminant discriminator to learn that blank blank space and be able to be able to classify not fake between not fake verses not fake but be able to classify like drift versus good data good input so the high-level idea is there by I'll not go into details it's just a separate one-hour presentation and you have anything for detecting that somebody's are we growing your model for yeah yeah it's it's the same idea yes yes yes yes if you if you hit that that's that space that we trained our discriminator for yes we can detect us and alert and make an alert Oh somebody is sending I'll give you a demo so it's you can say this yeah a similar similar thing but a bit more interesting variational out in coders own I think yes I guess you guys all know how old encoders work so there's I can coil network there is intermediate representation and there is a decoder network and variational auto-encoders they're just basically apply a slight change on that they apply some constraint to that latent vector here so we can learn learn basically a distribution of your distribution of your probability distribution of your latent space here that will help us basically to build at the end of the day I'll skip this just just for the sake of time saving at the end of the day what we want to do we just want to build this compare probability distributions of two data data sets whatever method we will use like variational autoencoders statistical methods others we we want to get this this type of very simplistic explaining it in very simplistic way but different methods different like research papers all of them don't work don't try to reproduce them but actually the promises just just to have this two histograms plotted in your in on your dashboard and say and like notifying you hey take take a look and take a look at this like long tail and see what was there lets me let me just switch to some demo I'm not gonna like this is an open source stuff except some monitoring features I'm not going to demo this like they do a product demo we also have the deployment port and and versioning part not so sophisticated as mentioned in a previous talk but what I'm gonna do in this in this demo now so I'm training the I'm training the minis classifier I'm trained NIST classifier on a particular data set you see it like some metadata here just the economical model it is being deployed right now as micro service and I'm Sur and I would like to send kind of simulator production traffic basically sailing through G RPC images minis the nice data set let's look at the profile and monitoring so it started counting with me just yeah so this is a pure request metric so it's totally it's counting as it goes those metrics are more like batch metric so they they wait for some data to get aggregated these are you can see you can see basic with the debug what what's under the hood so those are requests that are we receiving right now and some metadata metadata that we capture along the way I would like to stop the stop this kind of good data generator and would like to send letters instead of digits to that model so just to like simulate that concept drift so we've seen digit so far and now we will send letters so so this concept drifting it started like it became became red actually I'm just saying in sending a mix of mix of both let me just filter a filter just failed requests you see it's just letters instead of digits it's a toy example but just to give you an idea it was what may happen in production so we can imagine everybody has its own use case it might be just a configuration issue upstream issue just an organizational issue you deployed you hand it off the model to sound somebody and he started using this model in a in a wrong way for instance were poor more like sell drive car like you scaled use cases have yeah it's actually 0.8 it should be it's nearly it's a in this case it's a reconstruction error where for this under the hood there is a variational autumn color in this particular example so it's kind of metric specific what whatever metric you use this metric can output like any number and you just set up a trash hold there and that - line now - line is being learned from the training of course so the but you can set it manually if you want the like the the output from autoencoders is reconstruction error in the in this case reconstruction error is how well houghton color can reconstruct the original image so we just and basically the original image reconstructed image and the kind of the error between that so the that's in this case that's the level of drift Ness I would say yeah there's this stuff with it's not a SAS product it's usually deployed on your V PC so we don't oh yeah you mean that way oh yeah of course so the if you if you don't need if you don't need your machine learning engineers to look at the D production data you shouldn't do that some of the data is we just do not provide this like the back the back capabilities it's just a higher level metrics however for for machine learning engineers you you you have to see that data it either unknown anonymized were in a row format so it's there is no other way to provide an observability for machine learning engineer rather than showing a real data to him so and this is Bob this is usually for security first first customers who we just provide this higher level net without the details have my model comments they don't know where they get a different answer and outside the company like one of them to really mess up and so connect a the work phone that is so different than this this is kind of an analysis of a model in isolation that's a comparing multiple models the outputs saying do they agree within their comments so that a structure that you support present because it looks like this is all we do support with so called like manual metrics basically you just take you you take your confidence and you set up a trash hold yeah you can yeah gotcha that's the that's so you were heavily on each chart here we have a compare with so you can you can run two models on different models in parallel and plot those like metric metrics sorry and models and models here so the that is the same you can you compare and actually it's a good it's a good topic just because it's it kind of it opens a different conversation what is an a/b testing for machine learning like canonical a be testing where you split the traffic between version a and version B in machine learning you don't split the traffic you send the same requests to version a and version B and then compare an output and filter out so we call like response based a B a B testing so you yeah yeah yeah so we do not have that decision-making like controller what what requests to to return to the end-user however we have the capability just to compare them and make make some sort of analysis on that yeah yes exactly with me when mirror we mirror the traffic and it's just a separate separate instance we do not we do not like interject the actual production traffic you know just few other examples like what's the text use case for instance I'm not gonna run this simulation just just to make sure quickly show you this one is just a cluster based monitoring for like for this text input the use case Amazon reviews we train the model the model to to get again I think the sentiment from the Amazon reviews and then like as an experiment saying what we're sending so slightly different messages from different websites also reviews but but the website also was sending like Wikipedia like data Wikipedia like abstracts they're just to fool the model and see how the system is how the system is like toll and reliable for this type of input and like a different set of metrics for tokenizer at I think we have here is it's unknown just unknown words like how many words in you're talking in you how many unknown tokens you have in your vector here simple check no rocket science here no autoencoders but what we found really useful for text you see text use cases just to identify new new words or new new tokens in your data set also like once you do the here you can different applications of them the actually the machine learning application or AI application may have more than one model you know this there is time with pre-processing model like the main scoring model post-processing model those just for in google's of the world yes they recommend to do end-to-end neural networks however in a real world we still have a lot of like just module a separate models chain all together into like into a pipeline such a real-time pipeline for for this purpose we kind of we interject the request ID just like as a best practice from all the micro services world and also we kind of snapshot all the data all the requests and responses within reached with our metadata into as tree into parquet so to have it available for kind of post-production analytics so we can build like dashboards whatever you have analyze models one of the things such including latency the big problems of sites in the Rambler educated vacuum sizing down the partners every dissipating said what's the biggest thing I can afford about that into these separate monitoring system [Music] it's not to be outside technically I mean you're doing about accuracy you're a bottle of the solution for that is it yeah that's we we have to solve we have to solve it for ourselves of course we're also on kubernetes we have to like we have to auto scale down scale and all that we have is the metrics that that is with with we don't we don't sell it as a kind of feature but it's just I think the natural thing that every like deployment system on top of kubernetes should handle just Auto scale auto scaling and yeah those when you deploy the models you do not specify the hardware parameters we auto scale it that's good insight we don't do that right now but that's a good insight we can we can chat about that for since we look at the particular model of a particular version and the kind of that micro services in instance basically it's just the latency between the input and output of this model this this particular one is on on just a gateway level so just round-trip latency for the thing I may wanna mention here so of course we're not graph are not like software and there might be super complicated and now analysis you would like to do with the with your model after after in production and what we do we basically provide a database for you like s3 based data Lake weeks that we continuously but basically in real time we snapshot all the requests into s3 into parky files enriched with the with our metadata and so we can do interesting things so it looks like a particular model here it's a it was glue catalog so there the trap like columns that are red that represent the actual model itself like the label the confidence the input target organized input and all sort of all sorts of the metadata that we generate along the way we also expose it to the to the s3 so we can you can build a queries like hey give me give me a particular model joined with another model the upstream model and fewer with the car where the confidence is by greater than a particular value and where the a particular check has failed and in this case I am just querying all the animals all the Amazon reviews requests that have more than particularly trash hold some so they have a lot of missing words in this case so the the missing words mean it means that model have never seen those though those words before so we can basically mine and explore that day that data to make the next iteration in your model the development here so the kind of one of the one of the feature I like is kind of openness we do not lock that data we exposed everything that we that we have into s3 in a very kind of optimized and readable format so we can hook any like bi tool here we can hook just to like Jupiter of Base Jupiter based like data exploration here so and in in some of for some of our customers it it becomes your like ground truth data like at the end of the day just because it's fresh out of production and this is there is no any like intermediate intermediate state behave in in between the the production and the model training here the basically that's it I'm open for questions you can be a profession and training Nevada new waters [Music] then you also only did it without you genetics Lucy it is a woman say aye even without result leaving my result data you were cheating production we did it has changed change prediction both of course so in if you have that ground truth data and you can calculate your accuracy fine of course you should like you should calculate it continuously and plot it upon on your on a dashboard if you there but most of the cases that I've seen you don't have the ground truth data available right away it's either some like very expensive labeling process or it's the data for fraud detection for instance that fraud will when you discover it it's it's too late you know if you discover it at the end of the day by your accounting team it's already too late you sure you should have some early signs of this patterns were just when you model what when when you're when you are able to calculate your accuracy and your confusion matrix that means that kind of your model already failed you know your model already failed and okay no that's that's fine you know we can the actual labels okay let's wrap it up thanks for the questions happy to chat afterwards [Applause] [Music]