Scale By The Bay 2021 : Joshua Arvin, Lat Designing and Building Complex Machine Learning....
all right so let's start so this is going to be a quick talk however we're going to dig deep on the best practices and the veteran moves when dealing with complex machine learning engineering projects and workflows so of course if you were to let's say learn engineering techniques using let's say serverless technologies and serverless concepts and also use containers to to solve your requirements creating a simple application if you think about it is kind of straightforward and simple but once you have to deal with real requirements that's where things get a bit trickier so we'll talk about those uh details in this talk and we'll share we'll share also how do you deal with machine learning engineering requirements along the way so to start i'll introduce myself i am joshua arvindla i am the chief technology officer of new works interactive labs i'm also an aws machine learning hero so currently there are i think 32 machine learning heroes around the world and then i'm also the author of the book machine learning with amazon siege maker cookbook so the funny thing is that today is actually the the the launch date of this book so it took me about six months to nine months writing this book last year during the pandemic and i made use of the extra time i had since i was able to save about three hours of commute every day so it took me about six months to write almost 900 pages and then the final version has 762 pages so feel free to read this especially if you want to be an intermediate or advanced user of of doing machine learning stuff in the cloud all right so to start off with uh no machine learning talk is complete without defining what machine learning is so machine learning um so simplify the simplified way to describe it is if you're trying to do something and it's not straightforward and you cannot hardcode it using your normal programming logic code and you're using let's say data and generalizing a model on top of let's say training data and trying to let's say use that model to check if this is a cat or not a cat especially for new images then that's one way to describe machine learning of course there are different types let's say supervised unsupervised and so on but for now the goal here is to deep dive on how to do machine learning engineering so basically machine learning and engineering combined in order to do something like this so if you have other requirements that involves machine learning then having that engineering skills and platform and everything that helps automate things that's going to be our topic for today also um here's the simplified process so if you have been doing machine learning for quite some time you're pretty aware that this is not linear you're you should be aware that this involves let's say um something like loops where after you perform a certain step you're going to go back and maybe retrain the model at some point because of course the models in production do degrade after some time but first let's uh to make things simplified first the machine learning process involves data collection you will need the data in most cases and then you prepare the data and clean it and then you do data visualization and analysis to understand how your data looks like you perform the feature engineering task and request a task and steps and then you perform model training and hyper parameter tuning so when you have um the hyper parameter tuning step that's where you have the opportunity to produce multiple models and then choose the best one because what hyper parameters are is think of them as configuration parameters before your training jobs start yeah so we'll talk about that later also and how we can use the cloud and serverless and containers to to solve that specific item next would be model evaluation the goal of the machine learning process is not just to create a model the goal is to create a model that solves this a specific problem best meaning there's some sort of metric or way to measure it and there if there's less error then that's probably the model that we're going to choose finally we deployed the model of course after you deploy the model a consumer or some other system would be using that model through some sort of api so if your if your model gets deployed here then if you're and you have a mobile application your mobile application needs to be able to talk to that api so if you look at this simple process you'll start to realize it will involve both machine learning and software engineering and yeah devops capabilities to get this done for one thing if you have deployed a model it would be weird to assume that if you have a new model how do you find a way to replace an existing model already live in production how do you know which model is better do you perform a b testing and so on so the last part model deployment is actually one of the three care parts that we have to deal with because in addition to dealing with deployments you also have to monitor your models there because there's a concept of drift and there are different things you need to to measure and detect well a model is live in production so before we start talking about all of the different tools and concepts and techniques out there let's look at the picture of a hammer because if all you have is a hammer everything becomes a nail and if you only try to use one tool and you try to use a single tool for all jobs then you'll start to realize that maybe 75 of the the jobs that you have there um you'll start to realize that your current tool may not be the best tool for all those 75 percent of the jobs so the best move here is to be aware of all the possible options out there because the best tool for the job depends on the actual context on what you're trying to solve maybe it's going to be the team that's going to work on the project maybe there's already an existing tool being used for the job and if there are certain machine learning frameworks that's already already being used why replace them if it's the best tool for that specific context so there's going to be a lot of argument between oh this tool is better than the other tool what's important here are the concepts because once we understand the concepts most of these tools more or less are working on top of the same set of concepts so now let's talk about the first important concept serverless so what do we mean by serverless so there are different ways to define this um one way to think about it so one one sure thing that you you are probably aware when talking about serverless is there are no servers to manage of course there are servers behind the scenes but if you are the developer or the engineer as much as possible you do not have to worry about the nitty-gritty details of taking care of the server let's say patching system administration work and so on so your question would be is there still going to be some management work when doing serverless stuff of course but it's going to be a bit different and a bit more straightforward and you want to focus your time dealing with the business requirements because when you have a requirement you have a project it's not about using the tools to build something it's about solving a specific problem so if you have more time now given that you will not you will no longer need to manage the servers it means you have time to work on the custom stuff and when you're doing serverless other things that you'll need to think about would be cost so for cost the the good thing here is that if you have a staging and production environment and they're they're using the same configuration most of the serverless solutions out there scale based on the usage so if there's only one user trying to use your application then you're going to pay for the usage of that one user which is sometimes free sometimes negligible and then when your same application almost configured the same way in your production application um if let's say one million users use that the the cost will will properly scale as well so that's one of the advantages of using serverless and there's a lot more let's say event-driven architectures being able to [Music] do this do that but those are some of the advantages of serverless so the the definition of servers did change um depending on who you talk to but that's one way to define it the next one would be containers when you have let's say something on the left side which is a refrigerator and you have let's say fruits there or vegetables if you are if you are trying to manage things at scale and you want to let's say have a micro service architecture then uh and and you want to utilize some of the advantages of container technologies that say self-healing being able to restart the container being able to port easily what's in your local machine to production being able to do some other automation things then containers will definitely help you do that so in our example examples later we will combine both serverless and container concepts and techniques to solve our specific requirements because a lot of people think that serverless and containers are competing concepts however they're completely two different concepts and we should just be aware of that so one tip tip i would share with you this is a very important one is to choose whether we build everything from scratch or to use an existing framework when dealing with complex machine learning projects and workflows as much as possible we should be avoiding building everything from scratch unless it's absolutely necessary or it makes sense in that case however in most cases using a specific framework um and platform would do the trick so here's one of the scenarios that i would like to share let's say that you have a team of data engineers and machine learning engineers and data scientists working on a project let's say we have four or five members there and then one of the members of the team before the project started decided to build everything from scratch meaning that the entire project was built everything from was was built using something that's custom and only that developer knows the ins and outs of that platform or tool and then what uh what happened there is that person resigned so the the child one of the challenges that we have to deal with is that how do we make sure that the knowledge is transferred properly to the other members of a team especially if that person is leaving already and joining a new opportunity so if you were to use something that's uh available out there let's say this open source tools like tensorflow mxnet by torch and yeah and then for platforms that say sagemaker and other alternatives in other cloud providers which are properly documented and heavily documented it's also easier to find candidates outside your company and have them join the company because they might have probably used the same set of tools in their own company and in terms of also trying to predict the other requirements that you have in the future if there's a new feature let's say model monitoring that thing that you built something from scratch uh may not necessarily have that feature yet so it might take you an additional one week to two weeks to build it while this platforms or frameworks might even be offering it for free which is already part of the batteries included in that framework so that's something you need to think about but most of the time try to prefer something that's not built from scratch meaning use an existing library or framework especially when you're dealing with more people in your team the next one is making sure that the tool and framework or platform that you're using is flexible enough to handle your needs of course when getting things started you'll be surprised wow this new tool is able to solve my machine learning project and requirement in just four lines five lines of code or something like that maybe five steps but once things become more complex it's it becomes more um critical for your platform to be able to support almost anything what do i mean by that let's say that you want to support this 2021 or 2022 new model which is available out there if you're not able to use let's say containers to easily port your your new model to your platform then you're going to have problems because you're not you're not going to be able to use that model in production and generally when there's a new architecture or model out there there are cases where the performance or the the results are significantly better than the previous uh model families out there and you will be tempted to try this out and compare with what you have but if you are unable to test out different models in your platform then that's going to be a limitation or blogger so now when we include container container technologies and techniques in our platform and in our pipeline you'll realize that actually things are much easier now because if there are different members of your team and they prefer different tools we can actually combine them and even even use different languages so for one thing um let's say data preparation and cleaning we have a specific container there which is primarily focused on data preparation and cleaning so we start first with a small data set and then we try this library maybe in r or something and then that r library or our package would be cleaning the data using custom scripts inside the container and then when you run it you'll be able to deal with let's say larger data sets when you are now going to deal with production level requirements the next one would be model training and hyper parameter tuning so if you have a specific container there let's say you have a custom code in python and you're using this new model or algorithm then feel free to use that and your platform should be able to deal with all of these different requirements uh with each container just trying to communicate with the other modules or components of the the workflow or system and then when you deploy it the goal here is to find a way to deploy it really fast when you're doing proof of concept work or when you need to use a specific set of tools let's say you want to build your own flask api so this is one example of the custom script and we're just using a very we're just using a dummy model our dummy architecture here so what we have here is we load the data we prepare the model and what's not shown here would be the main function which initiates the training job training task and then here when we want to create our own custom api then we just follow what the platform is providing and the good thing here is that uh sagemaker and other alternatives out there may already be supporting this because in most cases people will just follow the hello world tutorials and not really dig deep into what's really possible out there especially when you have the capability to customize things further so you will need of course the coding skills the devops skills and things to uh and other coding lines of code that would allow the the api to be as flexible as possible using let's say additional configuration parameters let's now talk about hyperparameter tuning or automated hyper parameter tuning when you're doing machine learning and machine learning engineering one of the things that machine learning engineers and practitioners and data scientists are doing is that they run training jobs and they check the the quality of the model using evaluation metrics so there are multiple ways to do it if you were to do it manually and it takes let's say two hours to complete the training job then if you need to run five uh training tasks training jobs it might take you two plus two plus until ten right so two times five especially if done sequentially especially if done manually but with the usage of the cloud uh let's say aws gcp azure you'll be able to do this in parallel or concurrently depending on how you want it maybe you want the first four jobs to run first and then the next four which will help save you time because in addition to worrying about the infrastructure cost you have to worry about the cost of the personnel also so there the good thing there is that the automated hyper parameter tuning jobs would automatically configure and tweak the hyper parameters and then you're going to have um your babe you'll be able to produce multiple models and then what you'll do what it will automatically do for you is it will choose the best model out of the different models produced by that automated hyper parameter tuning job in sage speaker one of the things which is kind of hard to find but it's really there is performing hyper parameter tuning jobs using different model families so if let's say that you have a three three three three type model here and then here you have linear regression or something like that instead of trying to run multiple automated hyper parameter tuning jobs you can technically just run one and configure it properly to support multiple families and then just use a single objective metric to be able to help you choose the right model yeah so this is an example also of some code which is also provided in in the platforms so for example in sagemaker there's something called siege debugger not a lot of people think that this is powerful but this is super powerful because it allows us to debug the the running training jobs so for example you have let's see a neural network there custom neural network using let's say tensorflow and keras instead of us trying to wait for something to go wrong what we can do here is we can specify a set of rules that say lost not decreasing rule along along with other rules and then when that rule is violated you can now use the different serverless services of aws let's say uh sns or something like that and then and then you have other services of aws that say cloudwatch cloudwatch logs or something like that being able to connect those building blocks so that you will be able to work on other things and then when something fails in your machine learning experiment a notification will be sent let's say to your email address so this kinds of things which are usually available for free that's that's that's one of the things that you need to also research because in some cases people are not aware of the free uh features out there which you can utilize to speed up the workflow and speed up your process internally this is a very important topic ml explainability when you have a model even if you have a really good model there are cases where you're not able to deploy this into production because for one thing you're not able to explain it and then you're not able to explain it to the stakeholders sometimes they would prefer a less powerful model but something that's easier to explain because of course if you have to deal with regulations and other things and your stakeholders really want to understand what's happening inside a model you want to avoid that black box treatment as much as possible so by adding a couple of lines of code you'll be able to use let's say the shaft values and the same infrastructure um the same server as infrastructure of aws to help you manage these things because the good thing here is that you do not have to to worry about the actual servers and you just need to worry about the code and the servers would automatically spin up and you would be able to run that ml explainability job and get the results inside s3 so this one uh is one example of what you'll see so here you'll see that um you have four features and then two of the features are not really contributing to the final output so here you'll see that um feature one and feature zero are the ones that's contributing to the final output and you'll be able to know that okay if i tweak these numbers a bit the final value would most likely change as well there are different ways to deploy a machine learning model so before we go straight to the workflows it's important for us to know that there are different ways to do the same thing and there's no one right answer for everything meaning that you have to choose what's best for the job so for one thing you can deploy a model inside the lambda function so if you're trying to significantly reduce the cost and try to make it free then maybe this is possible especially if you're just going to trigger your model um twice a day right if you're going to use the model to in and invoke it once once or twice a day then maybe you can deploy it inside a lambda function it's also possible to set up an api gateway and a lambda function and trigger the sagemaker endpoint that way you'll make you'll be able to make things a bit more custom in some cases especially if you need to process the data a bit before invoking the sagemaker endpoint it's also possible to use api gateway mapping templates with six maker and we can also deploy the model inside target with additional lines of code so once things become a bit more complex the platforms that we have should be able to support us so for example if we need to have multiple models inside a single endpoint if let's say we want to use different types of containers inside that endpoint then we would be able to save one cost also because instead of us having multiple endpoints for each of the models it's better to have a single endpoint for multiple models especially if you want to optimize on cost also and if we want to compare their performance also and group them together similarly we can use um let's say have something like an a b testing setup using production variants where if we do not want to completely replace a model in production we can just do a b testing there and as much as possible we want to do is do it the serverless way of things where we do not have to worry how it's done inside we just configure things a bit config level and then the actual platform would do this for us and finally we can also deploy a model inside a lambda function that supports container images so in aws this is kind of new i think this was last year or last last year where instead of us having to worry about lambda layers or trying to fit it inside that small small container we can actually use docker container images and bind it to lambda functions so that we'll be able to deploy models there so before we go straight to the workflows we have to understand that the more experiments we work with the more focus we need to give to cleaning things up and making things a bit more organized so the last thing we want to do is to be sad to be unhappy with the work that we're doing and in order to be more happy with what we're doing and to be more fulfilled what's important here is that we should be able to clean things up a bit make things more organized and automate things as much as we can so that if we need to perform the same experiment the second time around or perform let's say model retraining we should be able to do that without having to repeat all the steps and trying to look for the code maybe an existing workflow is just there on standby and then if you need to trigger it because we have new data then maybe that's one way to do it so the question there is that is it possible and the answer to that is yes and there are different options out there uh no need to use the specific tools that i'm using here but feel free to research on your own what are the different machine learning workflow tools available and i'm just going to share i think two two different options here which are easily portable and easily uh integratable with what we have earlier and here we have something called the data science sdk which helps combine and integrate sagemaker and step functions because for one thing step functions is a different service which helps us manage workflows using different uh different services of aws so with the data science sdk by just adding a few lines of code in addition to our original siege maker sdk code we'll be able to produce something like this and the advantage of using workflow tools is that even if you are let's say kind the hands off in the project let's say you're a manager you do not want to dig deep into the code and find ways to make your setup more observable meaning if that there's an error let's say in the model step or training step you do not want to go dig deep to dive deep into the code and to the logs and just look at the image is there a red here is there a green here is this process completed how do we check the other executions so that at least would help optimize things so that people can monitor this separately also with the data science sdk in addition to what i shared earlier we'll be able to combine it with other options and make things a bit more complex so for one thing if what you saw in the previous slide it's just a straight line what it can possibly do is make things more complex and there's an if else condition and so on and the if-else condition is going to be super useful especially if let's say you want the quality of the model to be above a certain score so if the score is let's say more than 90 or 95 or maybe even higher than the previous model then that's the time you redeploy it there's also another option called siege maker pipelines where this is fully integrated with sagemaker studio and sagemaker itself so that um instead of us um using uh something that's connected to other services something that has full integration with stage maker studio is also an option so for one thing the lines of code are different but in a say in a good thing here is that this has been fully customized and this has evolved after the siege maker sdk has been prepared meaning that let's say you have sagemaker processing you have a very dedicated um sklearn processor from the sagemaker sdk as well so you don't have to install something different and at the same time um for every block of code that you have in your sagemaker python sdk usage you'll probably need maybe one line or two for each one in order to connecting connect things and similar to the data science sdk you'll be able to run this accordingly and the good thing here is that with the say with the tools that i'm sharing you'll be able to build more complex projects and pipelines and workflows because you'll be able to build something like this and my advice when doing something like this is to be aware on how long it takes to test it before it is used to in production so what do i mean by that if you're going to use real resources online you have to be aware that some of the processes here the oblongs here might take three to five minutes to complete so it might seem just three to five minutes right but when you're iterating and trying to get things working the first time around you will probably need 30 to 50 attempts to get something to work especially if you're pretty new to the tools that you're using so my recommendation here is to replace a certain oblong with let's say something that may represent the oblong temporarily but something that completes within one second so for example in this case instead of using let's say sagemaker processing and waiting for it to complete three to five minutes while you're testing things it's better to replace it first with let's say a lambda function which would complete in one to two seconds so that at least when you're testing things out you'll be able to do those 30 attempts and 30 iterations within just a few minutes instead of waiting for hours just to test things because it's kind of hard to test things when you're using real resources and now that you have that the next thing you have to think about is how do i make sure that i do not over engineer the setup for one thing if let's say that your system is supposed to create new workflows by just a few clicks of a button you have to be aware that you do not need to over engineer let's say the front end parts so if your your application may not need redux or other state management libraries because your application just needs an api then just focus on the api and focus on the documentation so that a lot of other users should be able to use it depending on the end users and also it's important for us to be aware that capability and experience does matter when dealing with this tools and platforms because when you're using these tools we cannot afford to experiment live in live projects the goal here is for us to get something done in the shortest time possible so if people are not experienced enough to use these tools my recommendation here is to have a training program first and small experiments first before trying it in real data sets because real data sets may have gigabytes or even terabytes of data and the larger your data sets the larger the resources that you will use so when you're not yet familiar with these tools you may end up spending a lot of money that you may not really need to to spend so that's pretty much it um it's good it's a very um complete yet straightforward talk and hope you learned something from from what i shared earlier so thank you again and feel free to ask questions in the other channel so again i'm joshua arvindlad and my topic for today is designing and building complex machine learning engineering engineering projects and workflows and how we were able to utilize servers and container concepts to help us solve the business and the project requirements so thank you again and have a great day ahead you