Scale By The Bay 2021 : Alon Gubkin, Building an ML Platform from Scratch

Scale By The Bay 2021 : Alon Gubkin, Building an ML Platform from Scratch

Recording: Scale By The Bay 2021 : Alon Gubkin, Building an ML Platform from Scratch

thank you thank you so hi everyone my name is alon i'm from tel aviv in israel um and ataporia we build a product for data scientists and ml engineers who want to build their own customizable animal monitoring solution i've been involved with machine learning and devops for many years and i saw a lot of people struggling with setting up their ml infrastructure and that's how i started this workshop so okay so since i only have 30 minutes today i'm not going to do the full live coding session but rather a quick overview of what we built so the full live coding session is available on youtube it's around two hours if you prefer reading instead of watching there's also the tutorial in our blog so all the links are right here in this clinic and later on send them in the disco channel as well and if you just want to dive deep into the code everything is available on github and i'm going to send the links to all of those in this code okay so so here's the story uh so when you have one or two data scientists in your organization and they start working on your projects you're probably focused less on your ml infrastructure right but when you have more data scientists or when you start deploying models to production then the need for proper ml infrastructures become become becomes crucial and since every every organization is unique there are different needs for insta infrastructure across demo pipeline uh and here you can see some common one you have data collection and versioning right you have this maybe you need a feature style you probably need a way to manage your experiments and how to package and deploy models how to then you know push them to production and serve them and how to monitor them so maybe some of these problems are relevant to you maybe some of these problems are completely irrelevant to you and maybe you have other problems that are relevant to you but are not written here so i believe that there isn't a single way to build your animal infrastructure and before even starting you need to deeply think and understand your needs your problems prioritize and find the best solution for each problem okay so what what's uh what in the workshop so we focused on the following problems the ones mark the stream uh so we use dvc for data versioning yeah this is an open source tool for data versioning ml flow for experiment experiment management and packaging also ml flow basically tracks all of your experiments all of your machine learning experiments and send them and basically stores them in a centralized place then we build faster api based model servers and show how to monitor them for concept rift and performance regulation using emporium so all of that was on top of aws but in theory you could also use azure google cloud or any other cloud provider actually since starting this workshop i've actually seen people who do this on-premise which is pretty cool uh it's important to know that uh when building your your own ammo platform you shouldn't take any of those tools for granted even a pr but you should evaluate the alternatives some of them are written here because maybe they are more appropriate for your use case okay so before jumping into the architecture and code let me make sure we are relying on the basic technologies that we are going to use so let's talk about kubernetes and i like to explain kubernetes in the following way let's say that you have a traditional operating system like linux windows mac os android ios usually the traditional os is installed in a single machine right a single computer and it allows you to run multiple applications on top of it so in a simplified way kubernetes is similar to an operating system but it can run on multiple machines not just on one machine so a traditional operating system can run on a single machine and let you run multiple applications on top of it and kubernetes runs on multiple machines and let you run applications on top of all of them so for example if one of this application is a model server a model that sells predictions and suddenly you get a lot of predictions then if it was on you know a traditional operating system it might crash because it doesn't have enough memory but with kubernetes you can just keep adding more machines therefore you know more resources and make replicas of the application make it run multiple times and this way you can very easily scale and make your system much more reliable when you install an application on ubuntu for example you run apt-get install i'm sure all of you know this when you install application on an ad read phone you can go to the play store and kubernetes has multiple methods to install applications but i think the most popular one is helm which i used extensively in the workshop so for example you can do helm install ammo flow to install ammo flow just as a reminder lava flow is the experiments management tool open source tool that i talked about before so it's important to note that kubernetes is not really an alternative to a traditional operating system the the cougar in this agent actually runs on top of linux usually but at least to me it helps understanding it yeah and most of it is based on docker containers so we're so in the workshop i use the following aws services again this is just a basic alignment so we're all on the same page so we have s3 s3 is basically a file storage service and each s3 instance is called a bucket so in a bucket you can have files and folders just like a disk on on a normal you know on a normal machine so in the workshop we have an s3 bucket for the models and s3 buckets for the data etc ecr is a docker is docker container registry so it allows it allows you to push and pull docker images from it so in our case we are going to have a docker image for each model and push it to ecl eks is managed kubernetes so basically aws take care of a lot of details behind the scenes and just make it pretty easy to to use kubernetes so that's eks and finally we have rds so lds is a managed database and specifically we're going to use postgres postgresql it's an extremely popular sql database okay so let's talk about the architecture so we are starting from here from the uh this point and let's say that we have a data scientist who wants to start a new project a new model we are going to build a template for him so he can easily clone the template and start working on the model from there so this template will include a very simple model and that model will be the starting point for the data scientist so we're going to use a tool an open source tool called cookie cutter for templating and uh in the workshop i actually show how it works so the data scientist can then start training and experimenting and each experiment is going to be sent to mlflow so training and expert experimentation and then they each experiment goes to ml flow okay so everflow is an experiment management system it's going to run on top of kubernetes and that means of course that will need to spin up these kubernetes as well and we're going to use eks for that okay eks it's the managed kubernetes service in aws terms can be a bit confusing but uh it it makes sense in the end um so yeah so amphlo is going to save the model file themselves okay they usually pickle files like for examples to light gpmls but it doesn't matter it can support any type of model so we send models to ml flow and then those models are going to be stored in this artifact storage okay so this artifact storage is an s3 bucket basically like a hard disk in the cloud and all all the pickle files throughout all of the history will be saved here so this bucket will contain the model pickle and mlflow is also needs to use the um this model method metadata database so this is a postgres database on top of rds rds again it's the managed database service in aws so we have this model metadata data database and every time we send an experiment every time we train the model and send an experiment to emerald flow then the pickle file is saved in this s3 bucket and you know metadata about this experiment is saved in this database so what do i mean by metadata it could be for example the date it could be some metrics it could be any of those stuff okay so that's our flow and additionally so this is how we we're going to track our experiments and additionally we're going to use dvc so dvc is a shorthand for data version control it's also an open source tool and each version of our training data set is going to be stored in dvc it's also an s3 bucket and hard disk in the cloud so yeah so why you might ask yourself why is it so important to store all the history of your model and data and i think there are many reasons for that but one important reason is when you have an issue in production you want to be able to to see exactly what model and and what data set that model was trained on so i think that's a real best practice when you are designing your ml infrastructure you want to have full you know full history of your data by the way this is a little more a little bit more advanced but something that's also important is data lineage so i can talk about it later on in the qa if you'd like so this is the dev part okay this is the model development part now let's look at the production path so we are going to run our model servers those are mobile servers on top of kubernetes so in this example we have framework servers and these servers are going to load the model that they sell from mlflow okay so this model server talks to ammo flow and it loads the pico file from the s3 bucket where the model is stored okay so on the kubernetes we're going to have ml flow and the model servers but one problem oh i guess it's not really a problem but one thing that uh you need to take care of when you install an application on kubernetes is that this application isn't exposed to the internet automatically okay most applications are internal and you cannot access them from from outside so to since we would like to you know to call those model servers externally we need to expose them and the way we're going to do that is using terrific so prefix is an api gateway and it's going to help us expose those applications to the internet or you know not necessarily the internet it could be it could be an external saddle something like that so let's say we have a web application that wants to make a prediction okay so uh this web application will need to to create an http request it's going to specifically in the workshop it's going to be an http post request okay to slash my model the model identifier slash predict and in the request body it's going to have the parameters the features the features of the model before pre-processing this is going to traffic terrific knows that since the address start the url starts with slash my model then the correct model server in this case is for example model 2 and then model 2 will will make the prediction and return return the response the prediction to the web application okay so that's that's the general architecture of of the um the tiny ml platform that we are building okay so let's talk a little bit about infrastructure so usually when you set up infrastructure in the cloud in aws it looks like this it's really complicated you have a lot of parameters and there's a lot of room to make mistakes um so just one funny uh one funny thing that one funny story that i can tell you here is that just when we got started uh i made a mistake when i i you know i i think i created a database uh and i think in the provisioned iops i think i put like something like free uh like another extra zero here and then the cost was like 100 more or something that that's crazy i just simple mistakes can cost a lot of money so just this is one thing to know so instead of doing this instead of defining our infrastructure in the ui we are going to use a concept called infrastructure as code so specifically we are going to use pulumi now there is an alternative tool for infrastructure has code which is a little bit more popular it's called terraform um but essentially they are the same thing so what what's infrastructure as code so instead of defining our infrastructure in you know the ui and the complicated ui that i showed before we are going to define the infrastructure in code okay so in this case this is typescript code it's a type javascript so in this case we in one line we basically define a kubernetes cluster on top of eks eks is the managed kubernetes in aws and we define an s3 bucket okay so in the code we define resources now it's not actually going you know running this code so so okay so i can run this code using polumia so i i have this file and i then run follow me out this makes sure to spin up those resources in the cloud okay so if i say if i write here that i want any case cluster then polomi makes sure to create this eks cluster in aws okay i hope this makes sense and one thing to note here is that if for example i comment this out for example let's say that i only already run this and then i you know i already run this code and the resources were curated and then i comment this as per bucket because because i decided that i don't want it anymore then polomi will know to make the diff like it knows what's currently in the cloud it know it knows what i want to have in the code and then let's make sure to to remove that s3 bucket but not touch this eks cluster okay so it makes a diff like get div just not get it makes a diff between my code my desired infrastructure configuration to what i actually have in the cloud so what's okay so we talked about the concept of infrastructure as code but what the what are the advantages of that so first of all each change to the infrastructure can be code reviewed okay so you are much less prone to mistakes so each change in the infrastructure is not you know going to the ui and changing something it's more like making a pull requesting github okay so that's much safer way to do this a second advantage is that you can easily share infrastructure components you know if i build some cool component using eks and database and stuff like that i can just send my friend this source the source code for this component and then he immediately have that so i believe that this is very strong specifically for the envelopes fold because you just have so many tools but this technique allows you to quickly iterate and and play with different tools so i think those are like the main concepts uh the main concepts of the workshop of how basically the architecture and and and how we build the workshop so now i'm going to show a little bit uh of the code and feel free to ask which questions on the way and just as a reminder let me go back to the slide here so if you want to to follow the full live coding session it's around two hours and it's available on youtube so this is like a full co live coding sessions from scratch uh the github repo is available here and if you if you prefer reading if you prefer reading a tutorial then i'll also publish this as a blog post uh in this your url okay so let me show you a little bit about the code so um okay so the code is divided into two projects the first project is called infra and the second project is called model template so just just can you please verify that you can see the github please uh yes we can see the gift up okay thank you i just wasn't sure okay so uh yeah so the basically the code is split into two projects infra is the shared infrastructure for the ml platform okay so uh you know the if i'm going back to the architecture so most of the things here are common you know to all of the models so we have the kubernetes and we have those artifact storage right it's it's not as specific to to a model and while template is the cookie cutter template that we talked before that data scientists can clone and and start with to build their models okay so the infra contains eks it contains the terrific it contains the artifact storage that the model made a little database etc and the model template is them is the template that you know data scientists can clone so let me show you how it looks like i'm going to go to infra and i'm going to go to index dot f trip and the first thing here is to create a kubernetes cluster okay so one thing that's very repeated in the workshop is is okay how did i know how to write this code right and with plum it's really simple you can just you can just do something like polomi kubernetes eks okay and then you can just pretty much click on the first link and and then you just copy paste that yeah it's it's it's really that simple and that's what's cool about it so uh when i wanted to inst for example to create this postgres database this is the database that's used for ml flow the model metadata database this one so basically in the workshop i just google like polumi lds postpress and then i just copy paste the code and just you know make sure that it's what i need so that's pretty cool so we create a kubernetes cluster we install terrific traffic is the api gateway it's a reminder it's this part okay so it knows which your url i am requesting and then it knows to redirect to redirect to the correct model server so installing traffic using helm by the way um is the package manager for kubernetes uh here we are basically creating the model metadata database okay so this is the m4 database and this is the s3 bucket for mlflow it's this one okay this is where we store all of the model pickles themselves and here we install animal flow again we do this with helm chart and yeah basically that's that's the that's the concept it's it's it's um you know just taking each part in in this infrastructure right here and then implementing it using uh using polami now one thing that's interesting is that well we talked if i'm getting going back to to this uh to this slide right here um so i talked about using polumi for you know cloud resources like s-free buckets kubernetes databases stuff like that but in the workshop and i think this is a pretty good good practice i also use infrastructure as code to define uh what runs in kubernetes okay so instead of running so if you're familiar with helm uh then you know then you would install traffic by running on your command line helm install terrific like an active command that you need to do but instead of that i think it's it's a really good practice to to define those resources in in your code okay so this way what you run on top of kubernetes and all can also you know code reviewed and shared with colleagues so that's pretty cool and okay so that's that's basically the um the infrastructure part this common infrastructure and i wanted to show you also so this is this directory and i wanted to show you the model template itself as well so this is the template that data scientists can clone and it's pretty simple to clone you just you can just like cookie cutter and the url of this directory so with cookie cutter cookie cutter is a way to to uh to create code templates it's pretty convenient so you know i have some defaults here so so the data scientist can just you know enter the name of his model of or her model like they can write you know fraud detection and then you have you're go they're going to have like a fraud detection directory uh with everything they need to get started so one pretty cool tool that we used in the workshop is called point three so point three oh so every time i fall into this uh mistake it's further python it's basically a really convenient way to manage packages and dependencies in python so instead of you know having different python versions and requirements.txt and stuff like that what will makes it really easy so you can see all of the dependencies of the model right here so this is also a pretty cool tool that i use and the code is splitted into solving and training this is actually pretty simple pretty straightforward serving looks like this it's basically a fast api server that can you know call model.predict and return the results pretty simple and the training code is is well it's a live gpm model but you can have any model that you want here the only special thing about it is that it uses ammo flow okay so it uses xamarin flow to track everything to to make sure that you have all of your history awesome um so another thing that we have here is a github action that basically push pushes the model to kubernetes every time that you uh commit to master so the usual workflows okay make your dev in a separate branch in a pull request when you're done make a pull request you know merge that or squash that to master and then this is going to build the docker image and push that to kubernetes so everything is done through a make file so you can also run everything locally so install dependencies train the model pull the data from dvc and deploy so polomi is actually i forgot to mention that but polomi is used both in the model template for the model specific infrastructure and the general infrastructure awesome so i think i have one minute left so thank you very much again the um all of the links let me show that all of the um the live coding session youtube video is available here the code is in github and we also have the tutorial right here thank you very much you