SF SCALA: Model Versioning: WHY, WHEN, and HOW, Manasi Vartak
Recording: SF SCALA: Model Versioning: WHY, WHEN, and HOW, Manasi Vartak
[Music] thank you all right let's see if I can navigated that all right cool hi everyone my name is Mona C I am the founder and CEO of Verdi III i am also joined by our CTO Corrado who is hanging out there so hard questions some of them I might send them in widths away but we'll see how that goes so super excited to be here and speaking about model versioning this is a topic that I worked on for my PhD and have been building systems in this area for about five years so would like to share how our understanding of model versioning has evolved and share some thoughts around why you should be doing model versioning I heard some conversations before that well why do you care about it having more than two versions of a model and hopefully this will illustrate why that might be a good idea and we'll keep this pretty interactive it's a small group so feel free to ask questions as we go along cool a bit about us we are a small company based in Palo Alto whoever came up from South Bay like us kudos to us all for surviving that pretty good commute so we're based in Palo Alto like I said and my background is that I got my PhD from the c-shell group I used to work in databases it's a large scale data systems is what I've worked on through our grad school created this open-source system called model dB it's a database for models and that was really motivated by the kinds of workloads we were seeing out there which is people are building a lot of models they were not really keeping track of them and these models were being used to make pretty key decisions across the enterprise so that was a motivation for model DB we got a bunch of open source adoption it also ended up inspiring other open source and proprietary tools in the space it was very cool to watch and I was privileged to also do applied ml at a bunch of the large companies in particular I worked on the feed ranking for Twitter so full your news feed is a little bit better because of the work that my team did so as I mentioned Conrado is the CTO at Verona he and I worked together at Twitter he was the tech lead on the Twitter ML platform called cortex they're a bunch of really cool blog posts out there if you're interested and most recently on the nvidia self-driving car platform so we bring kind of an interesting mix of cutting-edge research along with production chops and what we do is we have built a platform building off of model DB to do versioning deployment debugging which i think is really cool and important for ml models and then monitoring today I'll just be talking about versioning because in our view that is where it all starts and hopefully I'll convince you guys studium on that to do versioning if you don't already cool so why must our version models well machine learning and practice is messy how many people here build ml models okay wonderful this should look very familiar until their model is accepted or you're tired and just want to go home you're gonna do this you're gonna specify model train evaluate debug and refine and my code always starts off looking really pristine you know I'm just reading a spark frame and then you know applying something to it accuracy not so great keep going at a UDF change the kind of classifier that I'm building do that some more and now I have a hyper parameter search by the time out of 50 whoo I have no idea how I got there and the analogy to kind of software engineering is that in software you know what the requirements are you know what the tests are and then you can keep trying until you get there in ml however the requirements are fuzzy we don't really know how to test our models quite yet and it's very empirical which means it's trial and error you try a lot of variations and those variations actually matter because they inform future exploration that's how Bayesian optimization for a hyper parameter searching and all of that works so what we did atom idea was that we looked at cattle and we looked at the competitions on kaggle the top 10 teams that can every competition and we looked at the number of submissions here so as you can tell a lot of those numbers are about 100 so people are building way over a hundred models before they can identify one that they want to put into production and this is only real submissions hyper parameter searches are usually not on there it's like multiply that by ten and that's usually the total number that someone builds so while we're building all of these models we are not keeping track of them very well so the state of the art looks as follows anyone does that my final final my best submit it in Europe's all of that fun stuff or this one is from a actual cattle colonel and someone is actually tracking each version of their models by adding a little comment now if you're a software engineer and you do that you will either quit the job or you shall be asked to quit the job but for a data scientist we do that all the time another thing that happens is it's hard to share these models you might imagine this is some slack messengerís a friend shared with me and it says does anyone know is so-and-so ever updated her notebook to fix the issue with the dependencies something like that apart from that we're close to deploying right guys this will these will power our self-driving cars and we need to hold ourselves we need to do much better here so we're optimistic that we can build a robust system to do model versioning and I'll talk about how we did that at bottl DB and now how we're doing that at verdun okay so before I move on just to summarize if someone asks you why should you care up model versioning there four key things one is safety you want to know what version of your model is deployed if a model is breaking you need to roll back you want to have a high speed of delivery so if you are on git then you can use the ICD systems like Jenkins in order to build and deploy your models you want insurance so if you're a team lead and if someone leaves your team how do you preserve that knowledge and we have had to do this multiple times at a large social media company we spent about two months recreating model just because the person who created them had left the company and there was no trace of what they did there and the last one is productivity if I'm a data scientist I really want to know what I did before so that I can determine where to go next so a bit about model DB it's still an open source project burden now sort of maintains it and it became fairly successful at particularly for a research project was adopted at some of at those companies along with research groups and startups and we're really kind of carrying the torch forward from there all right so that's why let's go on to very briefly when must you version models and we'll make this interactive one must you version models like at what time yeah yes okay yes thank you and the lifecycle of a model when would you want to version it maybe that's when you're training so you need to do it yesterday absolutely so I would argue that you need to version them when you're building them just as you version code when you're developing a piece of software if you're only versioning the artifacts or the pickle files or the checkpoints from your tensor field model it's too late because you're not able to go back and see where the model came from and as we all know here development is just one part of the whole model lifecycle we also do deployment and we want to know what version of the model is deployed something's breaking when you're monitoring you want to know if there's drift and the key point here being if you version models in the first place you can actually read the benefits across the entire lifecycle alright so let's get to the meat of the talk how do I version my models and this has to do with the fact that models are a little bit different from just usual code and it's as follows so let's maybe begin here the key reasons why we're doing this is we want to know the state of deployed models at all times second we want to be able to reproduce a model if you guys work in a regulated industry like AI for insurance or FinTech then being able to reproduce a model for a regulatory body becomes a pretty key use case for you and then preserving work in analysis so let's think about the ingredients of a model right we have the code here's an experiment that I ran if I want to recreate this particular model then I need the code the second one if anyone wants to yell it out we think that there are four key pieces to a model version so I want to yell out others and we'll just go through this fairly quickly thank you okay perfect did not plan but comfy so you can have the same code my code is identical what I'm going through is I'm cycling through all the hyper parameters and then building different models so the same code is producing different models and therefore I also need to associate the hyper parameter values that were used in every execution all right so we want to hazard a guess for a number three thank you perfect I'll come back to that one data your if your data is different your model is different and right now we don't do a very good job of tracking where our data comes from so in this case I'd stored a particular file to disk and it turned out that it was a post process file in which I dropped a column already so next time when I run it it told me hey you don't have a column that you were expecting to have so small things like that end up affecting in the model significantly and the final one that I'll mention here that is super key for production environments is what are the dependencies for the model all of us I think work in Python Python to you Python three Python 3.6 3.5 scikit-learn tensorflow all of that needs to be captured in the way where your training environment actually turns out to be the same as your production environment and the reason why we think that we want to separate out the components of a model version than two for parts as opposed to hey let's just build a docker container and then we'll be done with it absolutely you can build a docker container but hopefully you're gonna reuse this data across multiple models or you want to share the features in that case taking a more modular approach to model versioning is a more effective effective way to go so there are two so we covered code data config and environment there are two models specific wrinkles that come up one is what we call model artifacts these are the pickle files the check points and so on the fact is model training can take a long time particularly for deep learning models and so unlike a Jenkins build which will you know take 10 minutes or so your models are going to take a long time to build and so you might want to store intermediate states of your model as you go along and the sort of poor man's version of model versioning is hey at least version your model artifacts because then you can roll back more easily however if you want the whole sort of robustness associated with model versioning you want to do the entire thing and the last mo specific wrinkle is versioning is fine except if you think about git git has no concept of what is a hyper parameter and what is just another variable in your code however as a data scientist you care a ton about what your hyper parameters are you're gonna want to analyze them afterwards graph them search them and so on and so as we're thinking about model versioning we also need to think about the metadata we want to associate with it so that we can reuse it later for queries visualization and so on make sense so far awesome ok so sum total is when you're thinking about model versioning and you want to implement it in your own company you're thinking about the model the code config data and environment so those are things that you want to capture and there are different ways to capture it and I'll talk about how we particularly do it in our system let me briefly touch upon what this means for the ML workflow because we work in Jupiter notebooks or Python scripts whatever is appropriate and in that case we start with some notebook we're reading from either a file on disk my sequel s3 maybe something more complicated Kafka stream and what we think is the right way to architect the system is to have a model repository of model versioning system that is a combination of a get server that is going to track all of your code also your environment if it's being stored in a requirements or an environment yamo file you need a database so let me start from the numbering actually so as you start running your jupiter notebook you want to snapshot the code because you want to be able to go back to that next you're probably going to read the data in from a database and that's the time when you want a version the data because that's where you know what the state the database was then next you gonna train the model and as you're training the model that's a time when you can start shoving metadata about your model into some sort of database this can be your hyper params but more often it turns out to be tags like hey this particular model is a logistic regression perhaps or you have metrics that are created during training time finally when you have the stored when you have the Train model you want to store it into some sort of artifact or a blob store and you could technically store it in to get accept these files tend to be very large and that's where a system like s3 artifactory whatever is appropriate just in NFS sort of file system that thing also suffices so when we're actually talking about a model versioning system we're talking about three very separate components and if you end up removing or ignoring any of these then you get kind of an inadequate experience for the data scientists alright so how do we implement it in model dB let's call it v1 because v2 is coming next year in model DB v1 at time kubernetes was not really a thing so we were just based on api's you know here's my hyper parameter I'm gonna go store it here's my metric I'm gonna go store it so in the new version we decided to approach this taking inspiration from how kubernetes is architected so we have resources so to speak we have code we have config we have data and environment the same things as before and what we ended up doing was providing users of the system a way to either supply these as llamó files so you can have your data ya mole that's gonna specify here's a query that I'm running here is a time stamp that's going a bit more into data versioning that I'll touch upon briefly or you can have a llamo file that specifies your config so that's one way to do it the other way that I think data scientists find much more appealing is an API based method and I'll show you very briefly what that looks like within model DB that's an API based method that lets you log code that lets you log your datasets your environment and so on so that's how we think about a model version so effectively our model version sha so to speak is a combination of these four shots that have been gotten by these disparate systems okay questions so far before I go to how do i branch and merge which is yeah yeah so we define that there needs to be a Python version if you're in Python if you're an R something else and we need dependencies we also need what hardware you're running on because if you're on the TPU then it's gonna be very different from C CPU 100% that was something that even with floating points and things like that it's yep um so all of these are yellow specifications where our minimum bar is we need a know Python or your dependencies and also what hardware like at the level of CPU TP or GPU other than that we're not as picky but it might make sense to add more annotations there represent and I think that's that's a result the stochasticity of how models are built it's not a deterministic system so we're gonna how to deal with that absolutely yeah and perhaps then adding the precision there too might be a good way to go for those settings so here it's a ya know file our abstraction that the backend is kind of a key value if you want to put in more key values if you're familiar with annotations on kubernetes you can add in hey precision matters to me but yeah so far we don't work with a lot of folks who are using TP use to be honest and so we haven't run into that as much [Music] we don't string so that if you want it or no where is he from you don't care about our privacy we need less information and we don't want to force everybody to have to be on a second permit so by providing some baseline and you can extend as much as you want stretch every piece of information we can have as much knowledge in future yeah just about everybody knows you don't tell them who's in there that's a fair point and maybe like imposing um sort of more a larger set of requirements might be a way to fix that in the future yeah yeah that's where awesome thank you okay to kind of summarize that if you in your environment make sure that you're tracking what hardware you ran on what was the precision things like that and know that you're gonna have little changes in your weights and things that are gonna come up because of set issues awesome okay so the natural question is hey you're talking about versioning how do i branch and merge now if you take the approach that what we're thinking about are really yamo files for all of these resources then this is not terribly different from making PRS against a kubernetes or palm chart where you're saying my conflict changed in this particular fashion or my my sort of config yeah no change or my environment yeah no change and so making PRS actually it's pretty much the same people always wonder hey how do you actually make PRS against four models what does it mean to you fork a model and the key idea that we found here is models are a result of these four components so figure out which one of those components is changing and then think about forking off that piece first and then merging it back in a really big hairy question here is also data versioning because data all lives in very disparate systems it might be a file system a blob store a database and versioning ends up looking very different for each one of these systems what we have chosen to do is we have written connectors for different kinds of data stores and we are only exposing a uniform versioning schema depending on the kind of system you're living in so what do I mean by that if you're working in an are DBMS then we assume two things we assume that your records have time stamps and it's an append-only it's very similar to how Vertica popular columnstore does this so if we have the time stamp then we're always able to recreate the data that existed when you ran that particular query in that case our schema for the our DBMS it's just what is a query you ran when did you run it and then we're counting on a connector to Vertica or another column store to be able to fetch that data again there are if you talk about blob stores s3 has a way to version each of your objects on s3 and so where appropriate we found that falling back to the existing system support it's a good way to go but this is an active area of work for us for a lot of people here as well because you can just go around storing copies of your data and if you end up having a workload that is a lot of updates as opposed to just appends then you can quickly run into scalability issues but happy to talk more offline on this we are actively figuring out what is the best way to version data in the variety of systems that we work with right now awesome so I will do a brief demo to kind of show you what model tbv 2 looks like within our system and show you highlight what the data versioning gets you and things like that and after that I will open for questions well the stuff is open source so feel free to check it out after and our requirement is that you're on kubernetes and we have Helms that you can apply yes you can absolutely so things that are different here are more are more along the deployment and monitoring size so if you just wanted to do model versioning use model TP you're gonna be great yes we prefer them to be separate just so that you can it's more manageable because some of these become very very long and very involved quickly particularly if you're thinking about I have a large neural network architecture then perhaps you want to reuse that file in another piece of code where you're reading it in and then you're creating a graph from it something like that so we would recommend keep it separate but you can experiment what sticking it all together exactly yep just like modularity is what we're going for yes it's a new transfer learning records in another bottle that would be a model this is a lottery for our picture today is everything the one China or point your external kid right we've seen people do it both ways and we try not to be that opinionated about it because ensemble's is something that lots of people do out there so what they'll do is they'll have a different repo per let's say like constituent model and then they'll have another repo that brings in those models and then stitches it into a larger one and that lets them iterate independently might not work for all cases though or visa versa we came from the external reference so you have a reference to the version of every model that you're stitching together so suppose your ensemble is I'm adding up four things and I have what version each one of those four things were perhaps I'm missing something yes so for every model you're actually picking a particular sha that you're saying hey I'm gonna include this in my model it's just another model record it's just a model record for us does that because it the one of our users right now builds they're doing kind of what Google does except on a smaller scale much smaller scale they're scraping webpages they're building 40-yard classifiers and then they're combining them to give them a top level classifier and what they do in that case is they have separate projects for every one of those 40 classifiers they are picking a particular model version for every project and then using it in a top-level project maintenance party grabbing this big exchange moment right right subtitle control and I would say like this is not very different from having dependency pins for your Python libraries so trying to impose that for models do it's amenity of educators for that we first made its agony Canaan fire this is bad way I see new that have a condition [Music] yep okay maybe I'll jump here and that might answer some of your questions all right the mental model is off github essentially where you have a bunch of these repos which for us are projects within the particular repo you have experiments which are similar to branches an experiment run is similar to a commit so to speak and each one of these commits has a shot associated with it so that's a unique identifier for this particular model record and that in turn comes from a unique sha for click the wrong thing that comes from a unique sha off your data it also has a unique sha for your code version which I thought I dare you okay unique shaft for your code version also whatever hyper parameters that you used as well as the requirements files so essentially the sha is a unique identifier for your model awesome so that's our setting this is just what we had with model D beam and we found that just having a central place where you can see all of your work so far it's pretty powerful unlike your git commits you don't usually go and analyze your git commits to see kind of where you messed up I guess however for models you end up doing that a lot so think about what it means to review a particular model in order to put it into production if I'm managing a bunch of people then I want to know hey so like you tried a bunch of things what is the space that you explored one thing that my pet peeve is I want error bars on things because I don't know kind of are you cherry-picking a particular value things like that also looking at whether you have explored the hyper parameter space adequately and you can build your custom charts and things like that but the key point here being for commits you don't care as much about the metadata associated with it whereas for our models you care about the metadata immensely and so it's a first-class citizen in our system now since this is a versioning system you can compare models in this case I'm comparing these two models they have different IDs which are the showers associated with them and you can see what changed in the inputs to the model so hyper params metrics of course if you're these are just attributes or metadata so if your features ended up changing in this case my data set actually was different and so it's telling me hey your ID seemed to be different you might want to go and see what's wrong with your data set you also can see that the get code is different and these as they're linked to the github you can actually compare them and we're just linking to github there's nothing kind of rocket science II going on there now I'll touch upon data versioning so in the previous one we saw that the data sets were different and very reasonably you might want to know what changed so this is a simple case where I was just reading a data from s3 and it turned out that I had gone off and I had added a new file to our demo directory in this case it didn't have a direct impact on my model because this is a different data set so this is the kind of interesting thing that goes on just judging from the fact that the data sets were different you couldn't have made out whether it would have impacted your model so there are still some humans or in the loop or a human intervention required while making these sorts of calls so the final thing here I want to mention before handing it off to questions on the next speaker is once you version models as you're building them you actually have all the ingredients you need in order to deploy it so when you're deploying a model most people today are doctor izing they're models which we think is the right way to go unless you also have a lambda setting and so on a separate conversation if you already are storing your environment which is your requirements text if you're storing the data came from the artifacts then deploying becomes just a one-click I'm not gonna demo it here have your new it offline because that's not the topic here and you can also do interesting things like defining a particular model API so if I build a model someone else is going to use it I hope so I want to tell them hey I expect to get this set off I need to get 40 inputs I want to produce one output you can also start associating distributions with it so that you can do interesting things with it like monitoring downstream but you get all of that the key point is from doing versioning and so my key takeaway for everyone in the room is as you're building your models think about how you can very easily track all of this data and I will quickly show you what it looks like with a model TB client so for the model DB client it's based on api's unless you're using the context that I mentioned before you log hyper parameters you go off and you log attributes you can log metrics you can log your requirements you can log your data set and you can log your code so instead of doing a lot of data scientists don't want to think about llamo or work in a different fashion in that case we found that this abstraction turns out to be pretty helpful and easy to use so this is the for this level of extra effort you end up getting a lot in return and so as you guys are thinking about how to implement this think about how to minimize effort for your data science team as they are building models any questions okay awesome so that was a quick demo and to briefly summarize version your models it will pay off dividends many-fold and Conrado and I would be happy to take questions yeah so the versioning is the same it doesn't do deployment and monitoring also debugging which gets more into explain ability so to speak so the operationalization is on the further platform but the versioning is in model DB we think yeah exactly so we use the same basic abstractions and we pick them that way yeah yes an early version is already on the Verdi github if you want to give the early version a try but q1 is when we're gonna do a more public release of it and if you send me an email I will be sure to ping you when it's live yep so if you look at v1 it was it had it was a research artefact that's not getting to all the issues that were going on with v1 but we did get a lot of great people using it we have rewritten it from the ground up and so our back-end systems are way more robust it used to be MongoDB it's now a relational database which can be plugged in and out we transition yes we also transition from thrift to G RPC because it was easier we have introduced data versioning model dbv one did not have data versioning and all of this works on kubernetes now that's where we run our systems and so way more robust and we've scaled tested it to millions of models like we can stand behind the fact that it's gonna work at scale awesome thanks very much [Music]