scale.bythebay.io: Rustem Feyzkhanov, Serverless Tensorflow 1.0.0 on AWS. Wait what?
Recording: scale.bythebay.io: Rustem Feyzkhanov, Serverless Tensorflow 1.0.0 on AWS. Wait what?
hello everyone today I want to tell you how to port your deploring models on a double slammed and make them serverless so basically let's start from like introduction in case somebody don't know about what basically that was Liam the concept is about so basically we have different types of like as a service types like on premise where your hardware like is owned by you in your company and server certain software is a service where you don't own anything and function service is somewhere between where like almost everything is hidden from you the only thing you can basically change is called which is good so basically a SS lambda is Amazon in implementation of functions of service so how does it work basically there is a lambda configuration which is stored which is code libraries and the configuration in terms of memory and maximum time and there is a container pool which to put in simply just easy to instances when trigger starts it's basically can come from API or other ones they produce event during the event AWS uses configuration files and take container from container pool to you to create this warm container which processes they went and then produces the response after that it is kept warm for four to five minutes so in case new event will come through this trigger in these four minutes it will not reconstruct the whole process again it will use the warm container so what can act as a trigger so it can be from then things like dynamic DB or a stream meaning like each new file in a three will get processed or each new database entrance in then idividual will process called watch is more like standard cron kinases which is basically stream of which will be processed like each event will be processed by each lambda impaired gateway it's basic API and legs which is basically a double years chat board platform meaning you can use your AWS lambda as a back-end for a chat bot so there are a lot of prongs basically that it's very easy to deploy you don't have to worry about like any docker or a patch or kubernetes you don't have to know about it you just know to put your code inside it's very easy to connect inside AWS infrastructure it's very scalable so it's it can of thousand lambdas can run simultaneously if the you write to their support they can increase this number to ten thousand so and it's relatively cheap I will show it later to today but it cost like a fraction of the cost for the same job on ec2 instance there are some cones which stop from using AWS lambda for some production tasks so basically logging is not it exists it just not as great is it I would like most people would like it to be it's very hard to debug it locally so it's very hard to have like your local environment which will be worked the same way as your AWS lambda e and the thing which I mentioned before about warm containers works pretty unpredictable so sometimes it can work fast in the fast way sometimes they can work in a slow way and there are certain limitations with remus types of tasks you can work with so it's like no more than 1.5 gigabyte ram not more than 500 megabyte discs and not more than five minutes execution time one additional thing which is not mentioned in direct documentation but only in FF IQ that when you provision memory you also provisioning CPU with lambda so if you want your model to work faster you have to provision as much memory as possible so next about tensorflow so basically no question here like well understand why we would like to port tons of flow models on a double slam des because like it's the most popular framework right now so the good news is that right right now we have tons of 1.4 meaning we have carriers inside sons of model inside core which is best news we have Google version of boosted trees so I didn't see in the benchmarks but most people say it will be as good as XG boost maybe I think better and there are lots of other stuff you should probably see the release because it's a big thing and so like why would we pour tons of flan damn de so basically we because cost is so low we can get something like 20,000 runs just $1 we have like out of concurrent executions and we have playable model meaning like if you're dealing with events where like you don't have like for example hundred tasks per hour but you have like a very peak number of tasks this lambda can actually help you to reduce your costs so what is the current implementation problem with porting tensorflow on lambda that basically the limitation which i mentioned before is 550 megabytes and like the most critical libraries which is tons of client dump i like together exceed this limit and there are like additional dependencies which you have to add so basically it's solvable so in my github project you can find the exact shell script so the only thing you need to do is use docker to have your local Amazon Linux and then run the script on this line X and then in the end you will receive a needed library so because like the only 50 megabyte we are already used 50 megabytes so when you actually import your model you have to use s3 so it's three works pretty fast if your lambda is in the same data center as your s3 so it's so you can basically update your model and you don't have to update your lambda so you can also change model as you want like if you dealing with tons of flow you're actually importing wire PB files of proto butcher and if you're dealing with Kara's you will import h5 file so where to get models the simplest way like if you want to you can port your own model no problem but if you want there like a lot of github projects that there's tons of flow zoo which basically has a lot of pre-trained models which you can just plug and play basically so some projects which were which were made with my implementation of telephone in the most lambda so the one in my github which is basically just image captioning which is porting inception version three so it called this one costs like twenty thousand images for one dollar it just produced like captioning for the image and actually another interesting project which was done using github a doubles lambda is accessible web so it was done by me it was done by a pre-nursery so basically it's about using image to text model so for people with disabilities be able to look at the web the same way we do because like it's a chrome plug-in where you can just click on the image and it will tell you what's happening on image you using deep learning model so in conclusion I want to say that lambda plus dancer for from laugh so it's like I basically think that surrealist answer flow is the future for deploying deployments because it's just so easy to do and implement so since we have time I will jump to code so basically here is my github project so so I put a lot of different libraries so it is all just one of them so basically what you see here like for files a few files one folder and let me start like if a huge trail file it's very simple you just run it so basically just deals in installs right dependencies libraries and does a lot of magic cleaning them basically so you don't have files you don't need and compress so but also you already have prepared pack dot zip it's it has model which I mentioned before English captioning so basically it's contains this source folder which is just libraries plus index dot file where like everything is happening so and if we look into index dot P I so it's just like you just import your libraries as you would in your average project and you just need to done the only thing you need to do is use like don't like we skipping some tons of code so basically in handler what you need you just download like model and some additional files from s3 and then you just like import it into tensor flow in standard way and then the only you just put the image and then it will produce your result which is kind of like predictions so let me switch to lambda u I so this is like standard lambda U I but so where you can just upload it as a zip so this pack I mentioned before and it will work from scale scratch on your lambda because like right now it uses my public s3 bucket so there is no problem with it so you can just basically run test and you can see that it runs in four seconds it's not real time but it's real time each kind of so if for some task it could be usable so as you can see like for pan the picture it produces the results of captioning of um the image so and everything works pretty great okay so presentation this presentation available on this link and the github project is available on this thing so thank you very much for attention I would like to answer your questions [Applause] it's possible so it's like so the guy for example a who did project for like he used image to text model so the only limit on the model is that I limitation which I mentioned before is 500 megabytes so like you will get limitation in terms of RAM so it's hard to exceed like 1.5 gigabytes but 500 megabytes is kind of a limit because some of those are more than that what I mean problem with the main problem is that I think it's initializing the whole thing so this is one of the issues with lambda so it's with one container like if you keep your container warm it will be faster but it but still it will be three seconds just because it's not optimized so my personal perspective is you should have you should optimize your model for deployment so and I don't think inception is like optimized for this you that's the problem you don't have direct access you cannot provision GPU yeah and obviously you lambda is cannot have a GPU by definition its CPU so yeah but you like if you just have if you don't train and you just want to deploy your model I think a CPU is okay [Music] well that's the problem if I like it best-case scenario if we could like put model inside this archive it would work extremely fast but since it always has to download model from s3 and run it so even with work container it runs the whole process again so you cannot have like in like imported model and you cannot have like just model so that's the problem yeah I checked and that's basically when I checked like execution time problems with like on each like line it was like the problem well one of the ways to solve it you can dock your eyes the other thing and you can run actually docker inside and abuse lambda of course it completely like the completely removes like principles of cirrhosis but you can do it if you are [Applause] you