scale.bythebay.io: Sunil Mallya, Building model testing infrastructure and scaling AI predictions
Uh good afternoon everybody and thanks for coming. I know it's Saturday and I'm between lunch. I'll try and make it quick and entertaining. Um so Yeah, we got out of the San Francisco office here. I lead the customer deep learning stuff at the AWS. Um I'm pretty passionate about both deep learning and serverless computing. So, I thought I'd give a talk at the intersection of both these topics. Um my background has been distributed systems and machine learning
Prior to this, I actually co-founded a company in the computer vision neuroscience space. I worked at Zynga, EMC and so pretty diverse background in terms of companies I worked for. Um So, let's let's take a look at um Actually, I'd love to know how many people are here are interested with the serverless part of the talk. It's pretty much all. How about the machine learning deep learning side of things? Oh, okay. That's great. Perfect. Um so quickly, you know, if we think about you know, what's the pipeline that we go through thinking about you know, creating our model and deployment, right? So, we go from data, we process it
The data never the data is never clean, right? So, we need to get it into a format where it is it's actually feedable to the machine learning algorithm or the process. Um so typically, what we do is in case of supervised learning where we have data and label pairs, right? And unsupervised learning is where we just have data and we're trying to find trends in the data. Now, with supervised learning, what we would do is we'll always divide the data into our training and validation and test set. The idea is more often than not, we're trying to create a generalized model. So, we need to know hey, with the data that we have is the model generic enough that it is going to run in the wild. Then we do feature extraction, we do the model generation, we do the loop, we test, we look at what what the metrics are and sometimes the hardest part is actually defining the metrics for your model evaluation. So, we'll start with okay, we'll we'll try and just you know, be serverless all this time and see what we can do with the current tool set that we have. Um we put the data into S3
Now what, right? So, AWS Glue, we can categorize clean ETL the data. Uh we also have Athena where you can run standard SQL queries. But you're kind of assuming already the data is sort of processed or in some loose structural format for you to do that. But the real game is we have dirty data. How do we actually put it into a semi-structure or an actual structure where we can we can process. So, about a year and a half ago, I built this serverless map reduce paradigm that I thought it'd be appropriate given there are a lot of big data lovers here. Um So, the idea is can we just run map reduce, a well-known paradigm, just completely serverless? That is no provisioning of resources other than our an actual server, right? So, just using S3 and Lambda, can we set this up? Now, um I probably should have had the architecture first, but I'm just going to go through quick. Okay
So, um what we have is let's say your data is stored in S3. Now, you can you can actually specify with a driver function. You write the mapper function, you write the reducer function. And the driver basically creates this is a script that lives either it can be a Lambda function or it can live on your computer. This is just initiating the creation and provisioning of the functions. So, you basically map the you can map the files on S3 and your mapper it can provision a bunch of mappers and the count can be proportional to the number of files you want to process. And it runs. And then you have it actually writes it to S3
But now, here's the part, right? Like we need to do the reduce step after we do map. So, in order to make that serverless, we have a coordinator Lambda function that basically gets invoked every time the mapper writes an output. So, that's because Lambda is stateless, we use S3 as a mechanism to maintain state and where we are in the process. So, that does the magic and then recursively reduces based on the outputs that we get. And then finally, we get the output. Um I'll I'll walk through this uh actually. There you go. So, execute the driver
Uh we then enter the map phase where we provision the mappers. They write the map output to S3. The coordinator gets invoked. It checks hey, have all the mappers finished? It goes recursively. And then once the mappers are finished, it does the reduce steps. Now, we don't need to wait for all the mappers to finish, but this is just you know, an implementation where the idea was to show that we can actually do this serverless. And then the reducer goes recursively until a single output file gets created. Uh this project is actually on GitHub and um I will um I'll actually go through a use case
We'll run it we'll run it live. Uh I have a video just in case you know, it doesn't work. So, let's just say the demo gods. Um So, here's here's what happens. So, how many of you are familiar with VPC flow logs or flow logs? Just one. So, anyway, so what happens is um uh a lot of times people provision their own private networks. So, we'll be generic here. And now you want to know what's the traffic between each of the interfaces
And so, the idea is you can log every packet that comes in your network. And then you can look at okay, where what pairs and what's the traffic in and out of every ethernet interface. Now, that's the sort of what VPC flow logs is. And surprisingly, you can create more data than existing data because each packet is logged. So, very quickly, your data set grows at a more than exponential space if even if that's possible. Um So, what we do is we have a bunch of files. So, in this case, I have um I think 4,000 4,200 files. Each file is um each file is essentially a log
Let me show the um thought I had a file here. Um anyways. Um so, as you can see the number of keys, we have about 4,000 421 files. And uh the total file size total data set size is um 4.2 actually 424 gigs. Um so, massive massive data set that we're processing. And what we do is we provision the uh Lambda function three Lambda functions that I called out. And the mapper logic is fairly simple. As you can see, it gets the file it get the data from S3
It does the split each each line is new line. Uh I mean it's split by a new line. Um and you get the interface ID, you get the status, and you get the bytes. And we're just aggregating the bytes. Uh so, if it's in the SQL equivalent of this is basically aggregate according to the ethernet ID. That's that would be the SQL equivalent of what we are running here. And you know, and we have the uh reducer. So, it it basically creates three Lambda functions and I've just run it in pipe request response mode so that I can calculate the stats on my machine
But everything else is happening remote. So, as you can see, we get the mapper outputs. And the second first column is the number of files. Second column is the number of lines. Third column is the time it took for that mapper to basically run. Um And we'll wait. uh I think it's almost done. We can go into S3
Um I don't know. I changed the name actually. I should have changed the name for Yeah, I mean this was the one that I ran earlier just to uh make sure it works, right? So, 7:49 a.m. That's when uh the state was created. 7:51:24. That's 2 minutes and 6 seconds. Uh we actually have had uh we actually have the task uh finished in in that time. So, um And it just collects the stats here
Um as you can see, we processed uh uh three for uh 303. That's 3.54 billion lines. Uh and the cost, more importantly here. So, we ran in 2 minutes, right? 420 gigs of data processed in 2 minutes. And the total cost was 53 cents. Like So, um The for for tasks that are massively parallelizable where the map task is the bottleneck, uh this this paradigm seems to work really well. Um so, this is the summary of uh what we saw. Uh we have three and a 3.5 billion lines of a code
I'm sorry, lines that we processed. Uh 4,200 uh files. Uh and roughly each of them are 100 gigs. I'm sorry, 100 megs uh each file. So, which makes it around 4. uh uh um 4 420 gigs. Now, what's interesting is MapReduce uh is a general is a very generic parallel programming paradigm, right? So, uh even though we've sort of moved to better, you know, paradigms uh so to speak, uh it's still pretty powerful where you can express a lot of algorithms uh in MapReduce. And because it's inherently parallelizable, uh it's pretty cool that we can go build a lot of things
And what can we build? Like pretty a lot of different uh linear algorithms and some non-linear algorithms as well on top of this framework. Uh so, it's an attempt um you know, it's like, "Hey, let's do some um let's actually run some K-means clustering and see if K-means clustering is possible using uh this." Um so, it works pretty well. Um so, the algorithm is pretty straightforward um where we If people don't know what K-means clustering is, we have a bunch of data. We're trying to find, "Okay, if we say five clusters, what what what data points go into which clusters?" So, uh we we initially uh we randomly initialize the centroids of the cluster. And um for example, uh what I ran I'll I'll I'll show you some graphs later is uh running where new stories originate. So, there there's a global um information or news data set which gives you, "Hey, the new story happened in Tokyo. New story came out of San Francisco." And they give you the geo coordinates. Um so, the idea uh the idea here would be, "Hey, can we actually find where the gravity of the news is for a given day?" Um so, let's say we want to identify 10 centers uh in the world
Uh we would initialize it with 10 centroids. And here the centroids would be the coordinate system uh that's the lat long, right? So, we'll pick random lat longs uh which valid random lat longs. Um and then what we do is we take a point, we see how close it is, uh and then uh we update the centroid, and so on. Uh now, you repeat this until your centroids converge. What that means When the centroids stop moving, that means that you've essentially uh converged. Um yeah, there's no further movement. So, one of the visualization here uh is um so, this is just on uh I I didn't have time to run it in the morning today, so this is year-old data, but uh that that's what we see as uh that's the center of uh news stories. So, if I pick five uh stories
So, you can see it sort of moves uh you know, day to day. And uh yeah, I'm pretty sure that we'll have more red dots these days uh in the US. Uh So, how is this useful? Um you know, one of the things you can think of with K-means clustering is um you know, log log clustering. So, what are the Let's say you have a bunch of servers where which servers have the most error rate? Um or if you have Apache or Nginx logs, you can kind of see which uh what's buggy. So, uh all all this is pretty useful where you can just have a entire pipeline which is serverless, right? Like and the cool thing here is unlike your MapReduce clusters where uh you're limited by the number of operators um or rather number of jobs you can run on them. Uh here you're not limited, right? So, as long as uh let's say your Lambda function limit is 10,000, that means you can run 10,000 functions concurrently. So, you don't clash with anybody. You can keep running your um your programs in isolation, and there isn't a overhead in terms of um uh you know, "Hey, I need to go to IT to get my machines provisioned." And all of that
So, uh pretty flexible. Um so, now let's uh I'll I'll sort of move on the uh deep learning side of things. Um so, as we kind of think about three distinct phases, right? So, data processing, training, and inference or prediction. Right? We went to the data processing side of things. Uh we now looked at how can we train things serverless. Um There are actually uh mechanisms where uh you can actually load um and uh train on a Lambda function, but we are currently limited by the whole uh notion of having the time aspect, right? Like currently we have a 5-minute uh uh timeout on a Lambda function. So, you can sort of chain Lambda functions and so on. Uh but uh you know, when you're doing real complicated um modeling, uh it's it's sort of like you you you need the GPUs
You need the horsepower to train on large data sets. So, we're not quite there in terms of training neural networks on Lambda, but certainly doable. Um so, quick poll. What Where do you think we spend the most time in these three phases? Like if I say how many people think that you spend a lot of time on modeling rather than uh data processing? Is this compute time or real time? Real time. Oh, okay. Yeah. Data processing. Uh yeah, I'm glad people agree
Yeah, that's true. We we tend to spend a lot of time um on data processing. The model, you know, kind of soon follows. And on the writing a prediction and inference side is pretty trivial. Um But actually the fun part is uh Like what would you say the typical cost uh you know, spend on inference versus training? Like anybody wants to give a split? How many people think you spend uh 50% of the time on inference and 50% time on training? No? 70% on inference and 30% on uh training? No? Higher or lower? Higher. Higher? Okay. Yeah, so that's true. So, practically what we have seen is uh it's typically upwards of 95% uh of the cost spent on um inference
Um so, this is Again, this is talking to various teams uh outside and within Amazon as well. This is this is what we observe. Uh is a lot of the time is spent on inference. So, in terms of sorry, um lot of co- the cost is uh significantly larger uh on the inference side. Um so, which is why I wanted to spend a lot uh you know, time focusing on inference and uh prediction part of the side. Um What I've done is uh I've actually made open sourced two of these uh repos. Um I just didn't have time to make the third one, and somebody had already done it. So, um so, MXNet uh Apache MXNet uh uh there's a repo where you can um you can have pre-trained models deploy, and rest of the logic is taken care uh by the uh by by the framework
Uh same with Keras. Um So, you can have trained models. Uh, and then, um, even TensorFlow on Lambda is available there. Um, so, I'll walk through quickly, uh, how to how to do this. How many of you here like notebooks? Right? The fair bit, right? Like, so, I think it's like, uh, you know, it's kind of annoying where you have to use too many different tools and move around. So, I'm like, "Hey, I'm just going to write a Lambda function and edit and put it in just in Jupiter, right? Like, I I I just want to be able to modify deploy my Lambda functions right here." So, uh, actually wrote a notebook. Um, uh, where you can see that, uh, it's a little hack to say, "Hey, don't execute because, uh, you're probably not going to link all the libraries that are needed." Um, but the idea is, um, what we, um, Lambda function has a limit of 50 mags. Now, so you, if you have a large enough model and some of these packages are, I mean, I have like 400 kilobytes to spare, uh, with all the zip
So, you can sort of, it's very difficult to add those, uh, models there. So, the idea is, what what can you do? Um, so, best practice would be to store the model in S3, uh, and download the model because we, the bandwidth to S3 is pretty pretty darn good. Uh, and, um, I I think, uh, I forget the numbers now, but I've been able to get 35, sometimes 70 Mbps. Um, and because this is all in the AWS environment, you don't pay for the data transfer. So, works out pretty well and cheap. Um, so, um, I'm just using the HTTP link here just to be generic, but you would want to use the boto client if you're using Python to download. Uh, but the model is downloaded outside of the Lambda context handler. And the idea here is that you're not downloading the model on each request
Right? So, uh, you get to keep the model in memory and execute, subsequent requests don't need to download that. Um, so, this is, uh, a very simple, uh, model here. Um, actually, this one is a geolocation model, which is, essentially, given an image, it tells you where the image was taken. And, uh, we'll demo some of that. Um, so, it packages the code, uploads, uh, creates the creates the Lambda function. Uh, but what I've done is I've integrated with with, um, a SAM, serverless application model, which creates an API gateway endpoint and hooks it to the Lambda function. So, you have a prediction API in no time. And all you need to do is basically plug in your, uh, make sure you have the IAM role correct
And then, plug in your account ID, and plug in your bucket location, and that's it. Rest of the stuff is all automated, uh, and, uh, you can work out. Um, so, we can just This is a fun. So, anybody knows where this is? Antarctica, yeah. It's actually a pretty remote place where they're hardly one of the most remote places, Svalbard, uh, in Norway. And then, actually, the model is pretty cool that it can actually detect where, um, uh, where it is. So, let's actually I'm going to try and see if it can detect where we are. I have no idea
I've not run this. Uh, now, it says it's it's, uh, Michigan. Uh, Illinois. Uh, it's it's sort of hard, right? Like, um, there's no real This could look like any other building, uh, and unless that's actually part of the data set. Uh, but it's actually pretty good at, um, let's say, uh, I think we've done enough of, uh, the bridge. So, let's, uh, let's take a look at quite quite Um, now, I'm trying to get a different, um, So, No, no, it just overrides, right? So, I mean, here's I'll I'll I'll show this example. Uh, anybody know where that is? We'll find out. We'll we'll know
Um, so, that is New York. It does get it right. Um, but let's not fix it on the model here. The idea is, um, you know, how how can we actually run, uh, and what is the what is the, um, how long does it take to actually run these models, right? Like, that's the focus here. So, I I I did some benchmarks, uh, and, um, my screen is frozen. There you go. Um, yeah, so, the model, uh, I deployed a ResNet-18 18, uh, on, um, using the same mechanism. Uh, it's it's a model that, um, it's a simple classification model
Gives you given, um, what's in the scene, right? Like, is that a is that a person in the scene or is that a otter, whatever. Uh, um, this is a classic ImageNet model, which is about 1,000 classes. Uh, what we see is, um, I mean, even at like 75 requests per second, uh, on the left side in San Francisco, like, the average was about 1.2, uh, you know, 2 seconds. Um, which is, uh, which is pretty decent, uh, when you think it has to go download the, uh, model, keep it in cache. Uh, it has to download the image, pass it through, uh, pass it through the model, give the inference, and get back. And on the right is a map of, uh, latency across the world. I just wanted to see how, um, how how that affected. And, uh, not bad, like, the worldwide latency was about 1.5 seconds
Uh, so, not quite real time, but again, we all have different definitions of real time. Uh, and when you take into account the cost aspect, uh, this works out to be a fairly cheap affair to, uh, run your, uh, even your production workloads. Um, and similar, um, so, what I wanted to do was deploy, uh, an object detector model. So, so, let's see, uh, given this dog, we can identify, uh, where, you know, A, if this is a dog, and where the dog is in the image. And the execution is a uh, because I've not executed this, so, Lambda, as you're aware, um, there's a warm-up time, right? Um, so, the model has to be loaded. Um, so, let's see how long, uh, it takes. Uh, probably going to take about 10 seconds, maybe. Um, Yeah, so, the model prediction time itself was about 1.15 seconds
Um, that's faster. So, and yay, uh, we almost got the dog. Uh, it's Yeah, we missed a little bit of the nose. Uh, uh, but, uh, you know, uh, so, it's pretty close. And, uh, similar, uh, what I've done is with This is with Keras. Again, um, similar code where we download the model, uh, keep it outside, uh, the context handler, given some helper functions. Uh, as far as you're concerned, you just need to override the Lambda handler to do whatever the task is and extend that. And the rest of the stuff should just stay the same
Uh, and, um, this is actually running, uh, a SqueezeNet model, which is a much smaller model than a ResNet. And I did benchmarks yesterday, and, um, the round trip from here to Oregon, uh, at about 100 requests per second was, uh, 30, uh, 380 milliseconds. Um, I do have a blog on this as well. If you if, you know, um, platform agno- I mean, we are framework agnostic, you know, we um, so, we're an open platform. I wanted to make sure that we have enough support to run pretty much, uh, any available deep learning framework out there on AWS Lambda. Um, okay. Now comes, um, right. Um, so, things to consider, right? Like, uh, yay, we have deployed our model, um, but what's We're always thinking about as engineers, and somebody who's telling production, is what's a what's the CI/CD pipeline look for this? How do we think about that? Um, A, where do we store the models, right? Um, B, how do we version these models? Like, we need a model management system so to speak just like uh you know, GitHub for models or something like that
Um, so how do we measure the performance and latency uh as we uh kind of like release new models? Um, also how do we even roll out new models? How do we know uh model A is better than model B and so on? Um, so one of the hardest things is what's the ground truth, right? Like, how do we evaluate a model? Uh remember like we kept a test set and a validation set so that we can test on the model. But, hey, that was done on the data that we collected. How do you actually know that this is a viable option and what we see in the wild? And let me tell you, when you deploy the models and things in the wild, totally different. Doesn't matter how much data you collect, it doesn't it doesn't always add up. Uh so it's always good to kind of like keep in check and looking at the ground truth and have a strong uh model evaluation technique. Um, so traditionally, well, we can AB test our models, right? So, AB ABn test uh and essentially um as I said, like you take a lot of time to find what that metric is that defines that model A is better than model B. in the real world. Um, AB testing
Well, we need something more sophisticated like uh so, enter multi-armed bandit. So, uh AB testing could work uh but multi-armed bandit is a more sophisticated uh system which can help us uh evaluate the model. So, essentially uh the idea here is let's say you had multiple arms and the classic example is a slot machine. Let's say you have 10 slot machines which you're trying to understand which lever do you pull to maximize the chances that you're going to win uh um you know, win from the one particular slot machine, right? So, you're maximizing your outcome. Uh there are different strategies. Um, one of the strategy is, you know, hey, let's choose the best model. We will run it for 70% of the time, 30% of the time we'll allocate to other models in the system. Um, you can do probability matching or you can learn the distribution as you get the data
You can learn the distribution of which model works based on the context of let's say a personal user or somebody. Um, uh what what some of the challenges with a multi-armed bandit is like um it it basically you will have a phase where it'll explore and then it'll just switch and tell all the traffic to one model. So, there's no smooth transition and you kind of want to hold back and there are strategies where you can hold back and kind of see if it's still valid what you saw valid yesterday is still valid today and so on. Um, also, you know, you're wasting resources, right? Like, let's say you have 20 models like uh and your your model is the source of how you generate revenue. Now, if you give a bad model, you kind of there's an opportunity cost associated. Uh one of the ways to mitigate that is using a prior. So, essentially waiting uh uh each of the model. Now, how does this look with what we talked about uh you know, predicting these on uh uh our lambda uh talk, right? So, So, here's the multi-armed bandit uh approach
So, um Took some time to draw that. Uh but, the idea is you have uh uh a sort of a router lambda uh where uh you either can store a state like in S3 or in this case like uh percentages of which model uh needs to be um you know, the either the prior or the epsilon distribution uh or each of the probability of the model. All that can be sort of hardcoded here um if you want in the lambda function so that you don't you know, hit the S3 uh uh to fetch that one file. Um, and then uh you want to make sure you're using something to collect um collect the data. So, given we're talking serverless, DynamoDB is a great tool here uh to uh to collect all your metrics on the inference side. Um, and now, on the same lambda function, you can have different versions which are pointing to different S3 models uh or models in S3 and and you can think of this as real time, right? So, uh you can have either using Dynamo Streams or you can have a background lambda function that looks, collects, uh modifies the um the percentages, the serving percentages. Uh so, you have a loop, a system that learns which model is performing better and directs the traffic accordingly. So, you're not static in terms of uh what you saw
So, this is what I highly recommend as a strategy when you deploy new models um and um you know, previously in my company I've used a strategy uh and I know a fair bit of uh more sophisticated um uh pipelines uh do look like this. And this is one implementation that you can leverage for uh serverless uh uh deployment. So, I know I went a little too fast. I wanted to kind of give uh a spectrum of all of this. Um, so I'll pause here and I'll take questions. I'll be out even after the time. More than happy to uh um you know, give you guys insights into uh my work here. Thanks
Thank you. All right, that was a fascinating overview of how to how to do ML on on lambda. Uh questions? Let's check the question box. Maybe we could uh keep the model inside the script like serialize it in a JSON and put it inside the script. Well, that's the problem, right? Is the 50 meg limit on lambda uh which is why we can't uh because the whole package itself is as I said, like has 400 kilobytes to spare uh given all these wonderful shared objects and binaries that we have to put in. Uh it it just takes uh a lot of space. Uh sorry. Is there any plan for Amazon lambda to increase their limits? Uh well, uh if you kind of typically look at um we we we have a continuous process in the fullness of time, yes
Yeah. I mean, we've gone from uh we released it with 1 minute and then we went to 5 minutes uh uh about a couple of years ago or last year, I forget. So, right now one of the limits is that you have to to download models from S3 uh to your like local uh lambda disk. Is there a possibility or maybe you know, like uh to import model directly from S3? Maybe it requires some modification so but to to bypass this 500 gigabytes limit. Um, no, it what it does is it's actually not uh it's actually keeping the model in memory. Uh so, it does a it uses a temporary file mechanism. I mean, you can uh because the lambda uh using the temp space is not I wouldn't recommend because the temp space itself is 500 megs, right? Whereas the memory limit is 1.5 gigs. So, it's better to keep it in memory than write it to disk and read it
Uh because you don't you don't want to pay the IO, right? Uh uh of disk reads. So, it's easier to keep it in memory. Yes, so but there are you you can basically read straight from S3 without using hard hard disk to store model. Yeah, sure. Yeah, that that's that's what I do. Like, if you use the name temporary file, um you can use uh the name temporary file is a temporary file created in memory and you can only when you flush it, it flushes to disk. Okay. Any more questions? No, then we'll let you all go to lunch
Thank you again. All right, thanks, guys.