scale.bythebay.io: Asif Khan, Deep Dive: Continuous Delivery for AI Applications with ECS
Recording: scale.bythebay.io: Asif Khan, Deep Dive: Continuous Delivery for AI Applications with ECS
good afternoon on a Saturday afternoon so lovely to see you guys my name is asif I work at Amazon Web Services as a cloud architect and I happen to build the first AI application to be put on containers earlier this year on on the ECS platform so that's why I get to talk about it in this presentation for the next 30 minutes I'll show you how to write any application take any i/o application which is built and then go through the deployment pipeline and quickly ship it and what are those challenges are we need to work around and I'll show you a demo also so when we talk about machine learning it's not a new term AI has been there what has happened in the last 10 years is because of the availability of computer resources and cloud resources it becomes super easy to experiment to create those models train those models and and deep learning has really picked up therefore four broad categories of deep learning that people are focused on today are one is image image understanding if you take me a picture like when you go into check-in for an airport for your flight for JetBlue for example you will you can now stand in front of that kiosk and little fish recognize your face using facial recognition and which is using deep learning at the back end and it will show you that hey this is awesome check mmm cool and so it's already here there are other parts of deep learning which for example if you use alexa or any of those home assistants it uses natural language processing speech to text text to speech and and so on so forth and everybody's talked about autonomous cars and and some of that is already there where is where is deep learning today deep learning today is in our lives before we realize it so there are applications where for example deep learning is being used to help us make choices give us recommendations and based on patterns and behaviors so when you look at Alexa one of the examples I really like to talk about sorry this Amazon go Amazon go is a cashier less or cashless kind of a grocery store experiment it's in Seattle only I think but it's a it's a great experience you go in you pick up your your your grocery and you move on and you walk out without lines without like taking out quality of wallet and and your bill comes back on your Amazon account and it's fairly correct and you can dispute it not disputed and a lot of that is powered by deep learning so that being said we'll talk about whoo yeah we're all deep learning is as we move on but let's talk about how do you build these applications building algorithms is great we are we will have more and more orgasms coming out but to make it really tangible and usable I need to build application and be able to deploy them super fast so I'm going to talk about this lifecycle or the circle of ml ml life machine learning life so I've the bottom half of the screen I've got my machine learning or data scientist folks who are crunching garage amounts of data and they're coming up with algorithms they are coming up with models and they built the best model it's consumable they've published it now that's just the data engineering part of life what about the application engineers and it's almost Greek and French conversation right now and that's where we are and we are improving as we go so there is a DevOps team whose job is to take this code from zero to hundred and make it repeatable and make it deliver code really fast the the front end or the application engineers focus is the customer experience the business logic for example when you look at home estimates from Zillow or at film we don't inherently think as consumers what is the algorithm running what is the accuracy of the algorithm how did it get there we are looking for the experience the customer experience hey I want the home estimate and it should be accurate and I should just quickly be able to get it there's a whole bunch of ingenuous focused on that and there were ups kind of price to bridge this gap but today there's a there's a divide which exists on one side there are great ml models on the other side the question being asked is how do I really use this how do i scale this if I have to deploy it to 1 million consumers and call these api's how do I really make this happen this shouldn't be new for anybody but DevOps practices have existed for for quite a few years but it's really about continuously building code like when it came out in 2008 with small commits frequent commits make sure your code works and with when docker came out in 2013 they made this very repeatable packaging was super simple now you can take the same imagine deploy on your laptop to your staging to your dev and enter your production and and that's the the DevOps focus that hey we want to get this going get this fast get this efficient so many benefits of DevOps but this this slide is very abstract because automation is the key of DevOps like the key key soul of DevOps I could write my script and try to run it but it's not as efficient if I can build a build pipeline or a deaf workflow pipeline so when when the cloud public cloud came out in a table where our AWS came out with 2006 with ec2 before that the world was I remember the world where you used to write scripts each of us were writing scripts and making them work and at the lowest level of Linux but with infrastructure I scored all that became easy I could define it in a simple yeah Mille or a JSON file and push it and my resources are ready I don't have to wait for somebody to give me a box or things like that now things have further further further moved on where for example with Dockers Advent and how many of us have heard of container management platforms like meso scope binaries GCS go everybody Cole I'll have this count so that's a next level of efficiency we are gaining as we go on and I'll talk a little bit about communities ECS and other things so let's put this together what are the ingredients I need to build an AI application and go from there so at the deepest level you'll have to make a choice or your data scientists will make a choice of saying which is the engine I want to use do I want to use tensorflow ready to Apache IMAX net what is that that deep learning framework I want to use to build my models and that's a whole set different set of people but there are other ways of doing deep learning also for example they're AI platforms that exist today spark spark ml is a great example that you can build your models very quickly and the highest level of abstraction you have also these managed services that consumers are coming up with for example in Amazon there's Amazon Lex and poly and and recognition with recognition you can call an endpoint you don't need to have the data they are engineering in today's data science expertise in-house you can call an endpoint drop a photo and get back the result what is this it gives you eight different facets of of emotions and things like that so the world has progressed very fast and these these things are becoming available for developers to use I'm gonna I'm gonna don't go by this picture but I'm going to focus on deploying this on containers using containers as a as a way to get this agility going deploying it very fast so going back into maven packaging how many FF is used maven or Java developers okay some of you so you know maven so maven was great when it launched the pom file we could define the dependencies and but very quickly it went sideways where I remember we had a private member represent REE and for a library I had a major version but not the minor version so I had to go through that pain of going to my my ops guy and saying can you please give me that can you please give me access and and and maven kind of it did certain things well but the packaging was not solved like what it tried to solve docker came along and did a phenomenal job of helping us package solutions in a repeatable way the bits were all together the application the OS runtime the dependencies are all packaged together so I don't have to worry about what's on my laptop what's on the machine there and go on as docker came along running docker made running running containers super simple on my laptop where I could just say docker run and and works and docker relies on C groups and they use the namespaces under the hood with Linux kernels for example but running it one once is easy but imagine running 10,000 containers so you've got n containers on M nodes so you got an O of N squared problem very quickly and that's hard hard cluster management problem that's a cloud scale problem and that's where kubernetes came out last year Amazon released ECS in 2015 I believe way before kubernetes existed there was there were other flavors of that at the time only mesas existed but at the core of it is you need a cluster man a management cluster management service so what it means is if I want to be able to reliably deploy my application on containers n number of times and get that same SLA back that it doesn't fail if it fails it comes back up and all the benefits that the containers provide me so at the core of this solution is the managed service which is basically the brains of it which change the cluster management and it does the scheduler and all that you are deploying your containers on a VM like on a Linux OS and that has a bunch of containers deployed together in a in a construct called a task in communities world it's called pod so think of it as interchangeable and these tasks then you can schedule and on each of these nodes there's an agent which is a thin wrapper on the dock or agent which helps us talk back to the to the managed service or the cluster management thing in the front of this they are load balancers and you can absorb that traffic the load balancer is a shock absorber so we have an apologies because of the tax spoken but we heard we have looked at all the different pieces and we know how to scale the solution and how to make this well but it's still a broken world how does the data to a scientist talk to to a apps engineer to a DevOps engine and how do I make this repeatable and super fast so when you look at this picture a data scientist lifts in this world he has his jupiter notebook and there are a Mis that you can launch and the jupiter notebook is there and you can play with it we believe and earlier this year we published this blog is we can use an object store as a communication model what that means is I build my model and I store it on an object store like s3 or something else where I can store at with high durability and as I'm doing my data crunching and building new models I'm uploading them to the object store to be available to everybody else while that is happening at the same time my app engineer at the top puts code commits code and that navara for me as an app engineer I have been an app developer all my life is I push code and things just happen like this whole thing should be transparent to you to me but that's a lot of work that ops need to put put in place so you push your code into a code repository like github a Jenkins pipeline triggers that it triggers a Jenkins pipeline the pipeline picks up that code builds it onam on a Jenkins build server or a code build server and then you can push it to an image repository in this case it's Amazon ECR which is a private image repository like docker hub which is public you have other repositories like way also that you should look at once my image is available here using cloud formation and I'll show you that you can provision a whole cluster of container based cluster like VP sees your networking stack your task definition your service just like that using infrastructure as code and if you put this in auto scaling which means I tell it run 10 copies of it that's all I have to do at the task definition level it will run 10 copies of it given the constraints I've given it and talk some about some of them the cluster manager will pick up the image and start deploying it to those many copies as long as the image is available so this this is a lot of back-end that was just set up in no time now the mobile developer I talked about Zillow as an app of Redfin as an app that's the focus there and the guy is saying this is an image of the Golden Gate Bridge tell me what it is and deep-learning think of it as a image classification as children looking at things if you go to a child and ask show him a new picture and ask him what it is the chances are he's gonna relate back to what he knew or similar things Heath you knew and he's gonna do that probability and come back it looks like this you know what is this it looks like a television that's basically deep learning how like we are just doing it on computer except in our brains now so and then the the picture goes in to a DNS endpoint because you need to be able to from a mobile app you need a friendly name to go and hit you go to my product calm the DNS redirects you are to a load balancer as a shock absorber and as your traffic scales and it goes to containers with dynamic port mapping and your container selves your traffic or runs your machine learning algorithm and then you can have metrics pushed out to a monitoring tool like data log or cloud watch and things like that but when you look at this complete picture what have this we just done we have built a communication mechanism for different parts of my engineering world so I have data scientists who can upload the model I have back end engineers who actually deployed or DevOps engineer who build the pipeline my back-end engineer wrote the code and in the code he would pick it up from this bucket in Amazon s3 that pick up the model every time so every time a new container runs a new model is getting pulled and you're getting the latest deep learning model behind it so with that I want to jump on to a demo I want to show you this working and I don't um I think I'm doing fine on time so this is an open source library we open source like a project earlier this year and everything I've talked to you about is basically this architecture you can clone it you can run it and things like that so I'm gonna show you how to run this I have I'll cheat a little bit on this on this demo because otherwise this takes a while so I already have a stack up in a US East one which is basically I want to serve my East Coast traffic from that region on the cloud so I would say okay let's do this but let me launch a new stack while we are here and I'll walk you through the different pieces so what am I giving the system I'm putting it a script I think of it as a cloud formation JSON files are that a template that a table understands there could be a templates for other cloud providers but this is what a table understands and this template as I go through and I I can say this is called MX net scale and I'm going to take the defaults and give it permissions to create stuff and so on and so forth so what you see here is really and when it really kicks in you will see multiple stacks getting created and I'm not sure if you can read it at the back so let me go go to zoom so what am i creating I'm creating my networking stack and all the networking parameters all of my cloud infrastructure at the click of a button as long as I've coded this Val I'm creating my build pipeline through this and I'm gonna also build this while that is happening let's switch over to something which is pre baked so I create my VP sees my networking stack I've created my build server built build pipeline I've created my ECS or container orchestration mechanism the nodes the the the task definition the service definition the load balancer all of that I'm creating so let's dive deep and understand what what is really happening so let's look at the code as a starting point so I have my code which is called image class classification predict and when this comes up so there are two files in this one is a build spec camel and a docker file so let us look at the lock docker file first so what am i doing at the docker file I'm basically installing an MX net library in the container because I need that framework to be able to do inference so this is really using the inference I'm not doing training I'm assuming the training is available and I'm basically saying launch my Python app which is my application I also have a build spec yeah Mel the builds McDonald tells the build server what are those steps you need to execute so it basically builds a docker image it gets the login to the repository and then pushes and tags the image going back into what the app is doing so let's go back and look at our Python code so the app is basically saying when I hit this URL at slash image go and get the params and then predict what it is and the predict basically works by saying it's a it's an array a numpy array in MX net and I'm basically asking it hey what is it that you're doing give me the result and then I'm kind of outputting that the probability is this so that I can read from it that's all the code I wrote that's literally machine learning code I wrote like using MX net as a framework but I need the framework to be able to run this in containers n number of times so I've written my code I've tested it I believe it works now let's good to go look at the dev ops code part of the pipeline that got set up so this is a very simplistic pipeline you guys have done Jenkins this should look very similar this is a code pipeline but it pulls code from code commit that I showed you and then it basically builds our code on a build server the manage build server and it basically provisions the environment you can pick the environment and and things like that and then it basically runs all those docker commands and runs and builds that builds the docker image like it like all of this and finally it pushes it to the image repository that I talked about or showed you let's look at what got pushed to the image repository so when we go to the image repository I have a image app okay I lost the screen okay okay so this is like a private image repository like like your own docker hub living in your security constructs of your own organization and what I've pushed is the bits and I've called it latest I can also assign it permissions who can push who cannot push and things like that so if I say Asif I can say Asif is a principal and he can only push pull then he can do only these activities on the image repository so this is super important when you're designing systems to be product production ready ready so that's that's great let's look at the cluster that we just created so in this case the cluster is called MX net ECS that random string and okay this is sensitive I have in the cluster I have one container node because I'm using like I'm not running a massive cluster and I've picked up okay I need to do something about this and I'm using the dock of docker as a runtime engine and things just out of the box I have a service and I said as I as I explained to your service is basically a way to multiplex down to the containers and then I have a target group so when docker gives you different bridge modes networking modes host bridge and things like that the physical port if you go in the host mode you can use only once 8080 and you're done so there is a concept of dynamic port mapping that you can use to multiplex an ad on the load balancer to some random port on the on the physical on the ec2 instance okay let me try this yeah it's just a loose end that's it yeah so behind the service lives a task think of tasks as a parts pack I mean in communities world for example so I do the task I'm saying what are my host port my host port is the random port that I talked about 32 769 my container container port is 5000 but my load balance has 80 so I can hit this API at 80 and then I can I also have integrations to my monitoring so I can go back and show you all the logs that this application or the containers are generating will go by it's a it's mostly 200 so it's not causing any errors right now which is great and then I can I can configure all this and the image that I have as you can see I'm pulling from my ml test repo I've given it four CPUs this is super important to know so with MX net one of the things you can do is you can train your model in GPU but run your prediction in CPU so that's why you you see me using a CPU machine so think of that when you are running your deep learning applications so if you have a deep your training is running on GPU you need those cores but you can always run your inference on on CPUs so I've given it four CPUs to run my each container gets four CPUs at them at the max and I would say also put a hard limit on memory of saying you should not use more than three gigs of memory and and otherwise you are just killing the system and you're not being a good citizen on the on the node and it's essential if this doesn't come come up it fails so with all of that and we will see how far we went here and something failed there because and you can see that very quickly that's one of the buckets was not created or was there and it kind of threw up on me but that's okay so if I go back now and I look at my master stack and look at the output I get an API a URL a very friendly URL that I can use with my mobile application my front-end applications very quickly I'm going to show you that with postman what this when it comes back comes back says it's a suspension bridge with probability of 62% which is pretty cool imagine building a deep learning application in 15 minutes or 20 minutes and you have an endpoint the agility it gives you as application developers a front and the back I'm an engineer and the communication it enables we can go down into the bucket and see the the model if that's of interest but going back to the the architecture or the system architecture that existed and yeah it comes up so what we have just seen in the last 10 minutes of off of demo is basically as a data scientist as an application engineer as a mobile engineer as a DevOps engineer all of this can work in harmony and has to work in harmony to deliver really consumable and and really nice customer experiences like when we look at the Zillow app and say the estimate is it's 800 K there's a lot of engineering and and the pipeline that went through let's say during that process one of the data scientists came up with a new model the change is very simple it's an upload nothing else changes is just I train my system on my cluster and I uploaded the image similarly if I changed my application front-end that mobile client basically updates goes to the same pipeline and deploys without impacting anybody else and that's what containers kind of give you that that that demarcation and agility for so that's all I had to present I wanted to leave some time for Q&A at the end so I hope you you drive something out of this I how many of you knew this before from before I started ok you knew this nice so you're probably building any AI platform already so you said you knew this from before they yeah yeah exactly so a lot of AI platforms are getting built just as an annotation and startups large companies we publish this blog and maybe I should also give you the link and I'll this will be available this ah it's a 404 good let me copy this it was the wrong click so and there's a space in between that's what is the problem so this this is basically the whole talk that I gave you is based we built the project we open-source it we wrote a blog to make it repeatable and all of this is there for you to try like you can go through the code what we did and this is how a model looks like under the under the hood basically you're defining nodes and you're saying hey it's a convolutional network CNN and it what are the different parameters and it's basically the the math that it understands and the back end of this the whole predict part of it is already already open sourced so hopefully you can go back and take a look at this give us comments maybe do a pull request on the github repository and let us know how other thing is if things don't work it raise an issue we'll we'll take a look at it and fix it but a lot of the conversation around AI platform people who are trying trying to build bi platforms are building it kind of in this same same genre not exactly doing the exact same steps but different people are different doing different steps but this is what the problem needs to be to get solved right so with that I'll say I'll take a pause and see if there are any questions that I can answer the question about [Music] so I'm not sure if you've got the the thing there was Volta and Vidya Volta that was announced like the support for that was announced so I would say take a look at that but as compute gets faster so the new classes of instances coming up all the time from Amazon and similarly other people are trying to bring up other instance types but I think when you're building platforms GPUs are GPUs are super bored and when the training part of it but when you're running it on on the on the inference side of it you need to take a step back and say that you want to run it on what is your use case you are trying to solve Wow we always start it try to think that we start from the customer experience and say if I can do that on CPUs and get a performance that I want why should I spend more so optimize your architecture for cost security operational efficiency and then things like that that we normally called well architected does that answer a question okay thank you any other question how much was this value so I'll tell you the cost let me go back to the picture so you don't pay anything for ECS ECS is free okay you don't be anything from Java of course you pay for storage and the cloud you pay for two things storage and compute right so all the nodes that are using they'll be belt by built by the by the second so I'm running a pretty large p2p to kind of a GPU cluster so it's gonna cost me a little bit but you have the choice of taking the the instance type that you need and there's a free tier that you could use and and it builds you by by the sense or things like that but the the curve and I showed you that in the picture was compute is getting cheaper we have cut prices like above 55 like the last count was 56 a year ago I think we are very higher than that the cost is going it's basically the Moore's Law right and deep learning is able to use that your your your cost of running deep learning algorithms or training is become so so cheap and so affordable inexpensive that you could actually run a whole model training and pay few dollars to like a few hundred dollars and get away and and then a lot of this that you saw is free like you pick for example if you look at code build one of the biggest sacrileges I've seen out there is Jenkins running on a node and running build for two hours a day but you are paying for the whole day a managed environment you pay by the minute you pay only for the time the suck the compute cycles you use so when you look at these architectures and see the efficiencies it gives you from cost from speed and all of that that gives you an impact s/3 s/3 there are different storages and s3 that you could use so when you put all of that together from the platform perspective and the other platforms that you could use not just Amazon I think the cloud in general has become has made things affordable inexpensive and you can do these experiments very fast and build you know we we tend to say now that builders should build and they're as not be an impedance right so think of it like that if you want a a real number there is a simple calculator on Amazon so you can put in those instance types and storage and you will get a number out I haven't measured the cost honestly but I have not paid I don't tons of things but I'm I'm no I don't see a billion on the hundreds so and I Amazon pays for it so I don't who else any questions cool ah they go yeah so the github project is definitely the blog is a starting point the the next thing I would do is take a look at the github project basically if I can go to the github project it should yeah this is a github project I would start here and have I was pleasantly surprised when customers started using it and they would ask me it's working or it's not working and they would add to it or remove from it and they would fork it and do their own things this is a good start you can also go to the AI site on our side deep learning we have a whole micro site called deep sigh deep learning and can learn this similarly on on the container side if you go and say ECS resources you will find a lot of documentation white papers what other people are doing with containers and so on and so forth and you will also see the blogs and the reference architectures that we talked about here how do you build pipelines and things like that and similarly if you like it's as simple as searching for deep learning and it gives you the deep learning ami so if you want to do the training on Jupiter you could basically launch the deep learning ami the Amazon machine learning click and it launches your environment and you have a ready to go jupiter instance like literally you can start using with tensorflow with MX net whatever you want to use and so on and so forth so I would say this these are the starting points you should go back to and take a look but start from the blog go to the github repo then start digging deeper and if you have if you issue a pull request I'll be the happiest guy in the room okay cool thank you thank you for listening to me thanks for your time I hope people valuable