SBTB 2023: Emmanuel Turlay, How to make ML teams 80% more productive.

Recording: SBTB 2023: Emmanuel Turlay, How to make ML teams 80% more productive.

so this talk is basically recounting my experience working at Cruz as an ml infrastructure engineer uh and all the learnings that I got from there so uh I'm not at Cruz anymore uh I left about a year and a half ago now I'm founder uh and CEO of a company called catic that essentially buil open source mobs tools so just a quick background about myself uh I'm originally from France I started my career in a different industry not even industry in Academia actually uh doing partical physics research uh then I moved to the US worked at instacart for four years then Cru for four years and then I started my own company uh a couple years ago um so that that's the idea so when I joined Cruz um after instacart I was pretty excited to join a a robo taxi company I thought I would enter sort of the greatest the best and greatest infrastructure on Earth around machine learning turns out it wasn't really the case and the reason for that is that uh they did obviously have some models running on the car for object detection tracking and and so on but many of the behaviors uh of the car on the road were still dictated by algorithms and euristic uh as opposed to machine leing models so the hand full of models that were on the car uh were trained on local machines or local GPU machines lying around the office uh data sets were stored on hard drives uh that you know you didn't even know where they were um and models were trained once uh 6 months ago and then uh we don't really know how to retrain it so you know pretty messy setup um and so that's when uh we saw an opportunity to start a machine learning infrastructure team um it was called machine learning platform at the time time and the idea was that we wanted to make all of this more repeatable more traceable uh more professional essentially uh and so we started working on a number of different things the the first thing that we tried to address was access to data as I mentioned uh data sets were kind of spread around some on hard drives some in S3 buckets uh some in databases and it was very hard for ML teams to access those data sets and so they would spend most of their time foraging for data and making sure that they had the right data set updating the data set with the latest label data and all of this was very manual and uh was kind of a waste of their time so uh in 2018 uh we introduced a new um data processing framework internally called Terra it was basically a wrapper around Apache beam so I don't know if any of you has heard of Apache beam but it's essentially an SDK to uh describe data pipelines so similar to spark if you will but a lot more usable uh and it basically is the front end or the the code the SDK for the uh Google data flow product uh so Google data flow is a hosted data processing pipeline uh similar to like hosted spark if you will but it's much more easy to use uh and it's it scales really well so we were able to scale it to like thousands of workers to process you know pedabytes of data within a few minutes uh so that was very practical and so we built uh this library to ctera to make sure that everybody was using the same component so if somebody had to do a particular operation for example uh uh rotating images or cropping images or or uh any sort of like group by certain Dimensions like segment time or anything like that or vin numbers then everybody was using the same logic uh so this was a a success internally suddenly teams were uh having a much easier time accessing data and they spend a lot less time generating data sets and so they could focus more on um on iterating on models so this was the first sort of win that we had as a as an infrastructure team and that was crucial to establish our credibility within the company because M engineers and data scientists typically when you go to them to offer them new tools that like I don't care about your tool I don't want to adopt something new I just want my model to get better uh so if you if you tell them about like traceability scalability and so on They Don't Really Care as much so it was important to establish our credibility within the org and then the next uh challenge we tried to tackle is to enable automation so Cruz had this pretty ambitious Vision at the time uh that they would you know collect all this data on the road all the sensors collecting data all the time and then when the cars come back to the warehouse uh the the hard drives get offloaded into an ingestion Bay and then uh you do some data mining to figure out what are the sequences whether the the timestamps the scenes that were problematic uh so so that you can label them and then add them to your training data set and then the model can you know train in uh more sort of um tail end of the distributions and so they wanted to do this very automatically so that you know every two three weeks you would have a new model uh coming out and being shipped to the cars uh and obviously this requires a lot of automation because this entire process is very very manual and so uh we had to figure out how to enable this and again in a way that is sufficiently easy for a Workforce of like hundreds of Engineers to use by themselves without having to offload uh projects to infrastructure teams so this is essentially what I was talking about this continuous learning machine the idea is that you start with this canonical data set uh you train a model you deploy it uh to to a car for example or you know this works for any other systems not necessarily just a car and it generates inferences most of your inferences hopefully are correct some of them may be incorrect and so the tricky part is to identify those incorrect inferences that's air mining once you have those incorrect inferences so uh basically it's a model failures uh you you know get them labeled uh by humans and then you add them to your chronical data set and you keep doing this Loop over and over again every few weeks uh so that the model trains more and more in the corners of the parameter space so we had to build infrastructure that would uh make this as automated as possible you still obviously need some humans in in the loop there uh you know for to screen models before they they come out uh like review metrics you'll also need labelers but uh everything else should be ideally automated um so this is a quote by kvot the founder of cruise uh when they started launching outside of ss so they launched in uh I think in Phoenix uh Austin and some other cities uh and essentially because all of those tools all of those processes were automated so data ingestion air mining training and so on they were able to train models for those other cities uh very easily they did not have to restart from scratch to basically reuse stim pipelines and pointed to new data sets and we're able to get the cars to drive well on those uh those streets uh because you know different states different cities have different traffic rules have different weather different visibility and so on uh so they were able to launch uh so every new city that they have to launch in is not as hard as launching in the first CI as a point um so why do you need to build such kind of a continuous system uh so first you obviously need to track your inferences you need to persist every time you make a prediction whether you have an online system that does recommendation for example or any type of machine Learning System you need to persist those inferences alongside some metadata so you know whatever your users are doing at the time where you are what are the circumstances of the inference what are the features obviously of the inference and so on uh now once you have that so you have a set of features and inferences you need to find a way to identify uh inferences that are problematic obviously you don't have access to ground truth because uh this is you know real production machine learning so you need to find other ways to identif ify um failur so it could be user behavior for example if you're doing a a recommendation system or search uh um system uh if your user don't click on any of your recommendations obviously something's wrong uh or if your search results are not being clicked on same thing in the case of Cruz it was for example uh if a a test driver uh would like grab the wheel that's a a major signal that something's off not necessarily because we trained those uh those drivers to be very conservative even if there was no you know uh we don't want them to to think basically so that's just one indicator another one would be distance to the CL closest object on the road so for example if you are too close to a uh a skateboarder or another car uh so you can analyze the distributions of those distances and see well this is the Threshold at which we're comfortable if it goes below you can say okay well maybe look let's look at those at those scenes so you need to find those like nonm ways to to identify those uh those failures uh and then you can use some techniques to un label those things so you can have like simple models uh that just do one thing is to try to figure out a rough labeling for for for this scene and this rough labeling will help a human labeling service so for example if you ask uh a human Workforce to label objects they may take I don't know maybe a minute per uh per image to to draw bounding boxes if the bounding boxes are already drawn by a rough model and they only have to adjust them or validate them it's much faster and much cheaper so uh you you you need to to set that up uh and then once the data has been labeled you need to have an automated ingestion pipeline for this label data and pass it to your uh data processing pipeline to convert the data into a uh a feature data set so that you can start your training pipelines as well so all of this needs to be automated obviously um and then once the model has been trained evaluation simulation and so on it's also to be automated so that the uh next thing that humans need to do is to look at metrics and results and decide whether or not a model is is fit for for deployment uh so essentially what I just describ is end to-end automation so from the moment the data leaves the car to the moment it's being sent to the human labeling Workforce to the moment comes back and then the model is is trained and shipped all of this needs to be automated without any human intervention so that takes obviously a lot of infrastructure to to do this uh so I'm going to go into detail here as to like what kind jobs specifically are um are necessary to do this so you start with know data processing so you have some raw data sitting in the data warehous Maybe you know uh images uh light R Point Cloud sound Clips tabular data you have metadata about the different drives that the cars have been done and so on um so data processing is your first step uh then obviously training evaluation using pytorch typically um so this is pretty straightforward um I mean not the jobs but this the the sequence of steps steps is pretty straightforward after that what's really important is regression test and simulation so regression test essentially is every time something happens on the road uh very much like test driven development you know every time you have a bug you add the bug to your test Suite then you write a bug fix make sure the test pass and then you keep this test in your test Suite forever to make sure that every time your ship changes there's no regression in your code base uh so it's very similar here every time there's a scene happening on the road that uh was suboptimal uh you add it to your test Suite then you iterate on the model to make sure that uh uh in the simulation in a simulated environment this um sort of uncomfortable scene is being resolved so for example there was too close of proximity with another um sort of uh object on the road then you make sure in a simulated environment that this doesn't does does not happen again and so you keep all those segments of um of of driving in your test Suite that grows over time thousands and thousands of scenes that you have to re run every time uh your your model goes through through this process uh and simulation as well as I mentioned uh we would have to simulate uh the outcome you know based on a real scene what is the outcome with the updated model uh and this is done in a simulation framework that essentially uses you know game engines to simulate you know uh real scenes of of San Francisco with different lightings different uh time of day different places in the city and and so on uh after that you have a number of steps to optimize your model so that it can run on the on the car as you can imagine the hardware that's available on the car is not as powerful as the ones that you can have in the in the clouds for example it has constrained uh latency has to be extremely fast uh because of the uh response times that need to to happen in the car um and uh also the power consumption uh is is limited so uh there's a number of techniques to you know prune your models to make it lighter to make or quantize it for example uh to make it um to make it work and also you have to make sure that it works with a particular GPU that's there on the car so yeah those steps also have to be uh automated um and then metrix review that's typically the point where a human is going to look at a dashboard and look at all those test results and see whether or not the model uh first does not regress on any of the prior scenes but also improves the metrics that you're uh that you're looking at uh and eventually you want to merge your model into your uh into your code base and then deploy it to the car uh so once that's done uh you you know the car drives around and then comes back uh at the end of the day and that's when you ingest the the sensor data and you do aor mining uh then you do do auto labeling then you send it to labeling Workforce uh the labeling Workforce is going to label data obviously and then return it to you and then this you close the loop so this requires you know many different systems and backends you know a data processing cluster is not the same as a training cluster uh the simulation framework runs on a different different type of um of backend as well so all those things need to be orchestrated um so uh all those things here can be automated you can have some handoffs between those different steps you can standardize the way that data is being passed that uh metadata is being tracked between all those jobs uh those other aspects though obviously require human intervention so you always need to have a scientist look at the the metrics or some human look at the metrics before you decide uh to ship it out and same thing on the other side you also need to have humans labeling your um your content um but everything else can be uh as automated as possible so things that you need to automate this uh one thing that we we built in is something we call lineage tracking um the idea is that it's compete complete knowledge graph of all the assets that went into your your your pipeline so this is an example uh in Orange here you can see different jobs so data processing model training and so on you need to track everything that goes with it so the configuration of your training job the raw data that was used so you know to be able to exactly reproduce the same results so uh exactly the uh the the rows in your tables that are being used and so on um who is the owner of the code that is running the the code that is running itself the logs of this of this job all of this needs to be persisted and linked back to the the final outcome so the final outcome in this case would be a trained model that is ready to go out to the car it's important that you have a database somewhere where this model idea ID is is linked to all those other assets so that you can easily debug all the way back up to the the raw data uh if something happens on the road you should be able to go back in your D in your database and be like okay this is the job that trained the model this is the input data set this is the configuration we used these are the resource the cloud resources we use like what type of gpus and so on so this is important for a number of reasons uh traceability obviously you know you want to know what you did so that you can uh reproduce it and and debug it uh but an even more important aspect is compliance because especially in a a safety critical uh product like like Cruz and you know it was in the news recently uh as as you may have heard uh law enforcement is entitled to ask do you have full custody of your assets can you reproduce this results can you prove uh that there was no nefarious influence on the model and so if you don't have a database that links back your final product or your your model and your infan is all the way to all the individual pieces of data configuration code that were used to produce this outcome you may be in trouble so uh it may be less important if if your product is not safety critical but in general I think it's a it's good practice to have this exhaustive lineage tracking um so another aspect that's pretty important is full tolerance as you can imagine those long automated pipelines they can run for multiple days uh sometimes even just the training part can last seven days uh we've had jobs um uh running that long and so it's pretty common to have failures in Cloud environments uh you know it could be uh database failures could be timeouts could be you know external services are down uh any or your cuberes nodes are being preempted for whatever reason and so if you don't have a way to recover automatically from those failures then most of your jobs are going to fail you're going to waste a lot of compute power you're going to waste a lot of human power and so it's important to be able to recover from uh from from those things so like you know timeouts every time you query uh an API or or a file from like GCS S3 whatever you should obviously have to be able to recover from that network errors uh node preemptions and know third party services are down and so on so the way to mitigate those things is first of all well retry everything you know that's pretty the first thing you want to do every time you have a failure uh you try to you know you wait a few seconds and retry so like automated retries everywhere uh anytime you have to fetch something remotely you have to have some some retry mechanism uh then you need to persist uh checkpoints as as frequently as possible whether it's data processing checkpoints when you have a long data processing job or when you have a Long training job uh you know as often as possible you persist a checkpoint so that you can Implement warm restarts so the idea of a warm restart is that maybe your job has been training for like a couple hours uh and so you've been persisting uh checkpoints in cloud storage obviously not on your node in case the node goes away uh and so your job can be restarted and instead of tra of training from scratch again it starts training from the last checkpoint so you can automate this so that the end users the email Engineers don't have to implement those by themselves is if the framework itself takes care of that it means that um uh job failures are kind of even fairly transparent to users they don't even know that they're happening and the jobs will just uh proceed from the last checkpoint so that's that's pretty important uh caching is also very important because uh let's say you're iterating on your um training evaluation Pipeline and so you have a training job and an evaluation job and you're Ting on on the evaluation part you obviously don't want to have to re to repeat the training job obviously you could you know extract the evaluation job and just run it by himself or by itself on the um on the final model checkpoint but if your entire pipeline has the right caching implemented you can blindly rerun the pipeline and the steps that haven't changed the same input same code should basically not rerun they should just use a cached uh output and so can that can dramatically accelerate the the rate of experimentation uh within your your ml teams if you have this those types of of systems um and then finally once all the failures that were not be able to be automatically caught or retried or are self-healed uh you need to obviously log them track them what is the the root cause of the of the failure and then sort them by the most impactful the most expensive or whatever metric it is that you were try to minimize and then address them one by one obviously and so build um fault tolerance around those particular um issues um so the next thing that is important for those automation to work is observability um so if you don't have logs of your jobs or if your logs are not being persisted then you don't know what the failures are uh so persisting logs uh in the way that can be you know browsed through easily by your ml teams is important failures exceptions for example python exceptions if it happens at the you know 200 Pages log uh is very hard for somebody to debug that so if you can automatically trct your python exception store them in the DB or somewhere else and then show them in in the UI uh then it means that you're the ml Engineers can quickly see what the failure is and fix it as opposed to spending half a day uh scrolling through logs um infrastructure failures obviously you also need to have uh visibility into that uh so that means having some observability at the infrastructure level uh you know resource usage for example like GPU utilization is important because gpus cost a lot of money and so you and on also jobs are slower if you don't utilize the the entire GPU so uh you want to make sure you track those things so that you can um investigate the the cause of the of low utilization and and optimize that so all those things have to be tracked and and available for for review then another aspect that we try to optimize is for H de development uh so the idea of of uh iterative development is that somebody should be able to make changes on their local machine so whether they develop on the laptop or on a Dev box so make changes to logic training codes hyper parameters input data whatever it is that they are trying to iterate on uh run it on their local machine on a small amount of data to validate that uh the code is correct that there's no major failures and then very easily scale this job out to a cloud cluster without having to go through a lengthy deployment process of like you know U getting a PR reviewed merging the pr getting through a cicd pipeline to produce uh you know um container assets and then deploying it to a cluster all of this takes way too much time ideally the user makes a code change on a laptop then they Co they call a submission command line and that packages their local environment so their their local code their local dependencies their libraries and ships it to the cluster right away without having to go through uh of course for for production pipelines you need to go through you know code review and PR and so on but as you're iterating you need to be able to quickly Just Launch a job from your lap top um so that's uh actually not trivial because you know packaging local code can take a while and uh there's many different aspects that need to be taken into account especially at Cruise uh the the code that runs on the car is C++ and sometimes we would uh use pbind to expose C++ code to python pipelines which means that if you make a change on the C++ side at at runtime when the user submits the the job you need to compile and build the C++ Library link them and then expose them to pbind for the python code so all of this needs to happen at run time whenever the users the user submits uh their pipeline so back at Cru we used to use basil maybe some of you have heard of Basil it's a build system coming out of of Google I think it was called Blaze when it was at Google and the idea is that every Library every piece of code is represented by a build Target and so you can use basil to automate this entire process that automatically detects detects the changes that you have on your local machine will rebuild and recompile those things and build the corresponding Docker image and then push it to remote registry and start the job um so a lot of different pieces of of Technology goes into this but from a use from a user's perspective they essentially call a command line to submit a job and depending on you know the magnitude of the change within a few minutes the the image is built shipped and the job starts on the remote cluster and then the other side of uh attri development is that uh when the the job is completed the results should be readily viewable in the I so uh we don't want users to have to store their own plots on S3 for example or have to download any assets into a local notebook they should be able to go in the UI click on the job view the the plots the metrics the visualizations that they're looking at um the that they produced during their job without having to do any any additional work um so this was pretty important uh and it worked pretty well um over there so um I was I hope I was able to convince you that this kind of continues ml um processes are important to keep models up to dat and fresh and you know pick up on new trends uh this was for example particularly important uh D during covid when Co started many models became irrelevant because they didn't pick up on the latest trends you know people were seeing at home ordering more online and I'm here I'm talking about ml in general not just not just Cruise uh so basically Whenever there are changes in the underlying data the model needs to be updated and so if you don't have a very streamlined process to just almost click a button and get your model refreshed and and and have metrics then you're going to spend weeks you know catching up to Trends and it's really not scalable and so the way to do this is to have this endtoend automation built into your your mlops platform and that requires all those um those uh kind of aspects that I described in in this talk um so that's kind of what we built at Cru and really enabled uh to move much faster um and so that's uh the obligatory self-promotion slide at the end of The Talk uh where the idea for catic which is the company that we started after Cru um was that we would build this into an open source product that we could um uh sell to or or let others use essentially uh so we basically took all those learnings that we got from all this time at Cruz and we tried to build it into this open source product uh that is available out there free to use works very well on your local machine don't even have to deploy anything um and uh yeah that that is the idea and that's it for me thank [Applause] you if there's any questions happy to answer them otherwise happy hour I think is soon we can also talk there yes trt conversion um so um I think there was um a Nvidia uh library to compress models and uh to make models smaller and hold uh in memory uh better I think that they actually just released a version for LM I think it's called trt LM or LM trt I think it's a combination of pruning quantization different optimization techniques that are wrapped into a library so that uh models can run on smaller devices especially important for LMS because they're so huge um that um yeah