Scale By The Bay 2021 : Adam Gibson, Deploying & serving optimized ML pipelines using graalvm
okay great uh thanks uh so today i will be talking about deploying optimized deep learning pipeline so when i say that i don't mean models i mean everything etl uh putting on a different hardware even compiling you know and optimizing graphs and then bundling that into a binary using bravia um for those helping for like a growl vm tutorial um move that to the hallway track this will mainly be focused on i think a lot of the prerequisite topics because this is a fairly new thing that people are seeing and the use cases a lot of people aren't exposed to it uh quite as much because they don't they either don't need to because they already have big servers or there's very specific use cases where this stuff is required so just this is a caveat for those attending the talk today um so getting right into it uh so i'm an o'reilly author i've been maintaining the eclipse deep learning for jframeworks since 2014. uh so i've been at this a long time if you want to reach out my handle is the same on twitter github and linkedin uh i'm i'm at skymind along with my colleague eduardo who spoke uh earlier at express ai so if you have any questions also feel free to jump over to the express ai booth for more so things we build that you can download today uh deep learning4j a conduit surveying java cpp um so heavily you know there's a big focus on kind of the jvm stack that's where i've been for a long time now um you know we kind of started as kind of the place to build and deploy ml models that's kind of so what we're doing you know that's what we're doing today but you know we've expanded a bit you know trying to focus heavily on making uh faster applications running from the jvm but otherwise uh you know you can you can use stuff from python um so we're behind quite a few of the you know like java cpp if you're not familiar is actually you know the baseline packaging for a lot of famous libraries you know a tensorflow java i think being the biggest one but also more recently uh java cv like opencv ffmpeg among others uh so we kind of use it as like the python ecosystem you know how you know how everything interrupts value and empire arrays we do the same value of cpp's pointers so if you ever look at a java cpp project it's like programming in c plus plus because there's pointers memory free and allocation and all that so we use a lot of those little little primitives and use that to deliver optimized math code plus applications to for production it's kind of the core theme here you know so that could include serving models packaging end-to-end applications or even just running very fast numerical code or hpc code on you know in you know in a spark environment uh there's a lot of different use cases there uh if you have any questions uh reach out at hello conduit.ai so breakdown uh the problem scope so in this you know sorry an overview of the presentation there's you know kind of problem scope trends uh this will be heavy focus on ml be kind of the the general area of ml ops so i'll say what we are and what we aren't uh for those that maybe not you know are aren't familiar with the space i'll give like a brief introduction to what i'll call dl ops so focus on deploying and running fast fast neural net fast and large neural networks uh on different kinds of co-processors among other things um ml pipeline components so you know clarifying that you know you have a model and then you have other stuff what is it um performance workload trade-offs so if you're into understanding the whole ml stack all the way down to http versus grpc versus you know your own custom binary protocol stick around uh tools and techniques for optimization so if you want to apply some of the stuff like a deep learning compilers and some of these other things i'll also give some tools there and then i'll give a concrete overview of kind of our opinion on what all this stuff is as briefly as i can so problem definition so some acknowledgements uh for those that are like a little worried a bit a little bit worried about hype i always try to focus on pragmatism in my talks most people aren't deploying departing models most people you just use xg booster random forest and they're good to go they don't need all this stuff so if you want to stick around you know if you do have you know if you do need this stuff then great uh if i haven't hit your use case in this talk i would be really curious to hear more um your company probably doesn't have google facebook or gp3 size problems you probably just have you know analytics that's okay and and you know people will deploy tens or hundreds of analytics models and drive a lot more value than one neural network might uh so just keep that in mind um accuracy data quality and bias are critical to the successful model just as much as kind of the performance stuff i'm talking about this talk uh and finally the tools and techniques on this talk are mainly for inference not training or at most up training a model so like fine-tuning it or adding additional data to the model and not for training a model from scratch there's a lot of material on that because that's where you have to start is training a model but around inference when generally people just throw up on there up a server with http and they're done um you know some use cases require more than that though so what do you do when costs get out of control or you know you need to deploy a model to a mobile device what are you know what are some of the things you can do those are the kinds of areas we'll be focusing on here so there are broader categories you know labeling and data quality setting up a rest api packaging like how do you deploy an ml pipeline and experiment tracking you know which weights and biases does i think they're kind of the and i guess ml flow as well are kind of the standards in this space so if you use those tools we're not that we're kind of like a lower level component of that that you might consume from something like those tools so why optimize intro to dll ops so the main situations where people need this stuff are either transformers nlp or mobile devices you know most of the time whenever we have a phone and we want to run a model you either and you want to run it locally because of privacy issues or what have you that's gonna that's gonna fall into the bucket of vision or speech generally with ninety percent of it being vision you know taking a picture of some clothes and then and then and then and then seeing what the price is or taking a picture and you know you know and and then decide you know for a better ad for better ad serving or you know all sorts of things that the big ad tech giants are kind of doing or there's just like basic security you know who's at my door um and i want to run that on a small like jetson nano or raspberry pi uh you know these things are getting smaller and smaller and you know the more you know the the you know the the cheaper we can make computer vision deployment uh the more it'll be used for better or for worse um you know so i think most people i think more people if they're thinking about this might be just more familiar with the standard architecture or anything transformers related that's probably where you might be more familiar with some of these things or bigger language models that's where you might need that's you know the trillion parameter models you hear about are the ones where you might start asking well how do we actually run this ourselves uh you know so some of these techniques will either make you know help you reduce your cost when running these models or just in general maybe be able to run you know help you run and train some of these models on something that isn't a million dollar cluster so problem size uh so computer vision need to be compact to run on mobile or at the edge nlp models are increasing in size you know we see gpt3 we see bert um every like big tech company right now is is is actually competing for i built the 4 trillion parameter neural network i'm going to build gpt 7. uh you know there's there's a lot of that going on right now and you have hugging face directly monetizing this saying hey we'll help you deploy those we're just we're just going to charge you for it but you know we make it easy to do we'll give away the ability to do it that's kind of their their thing um you know so there's still a lot of questions around how do we shrink models uh you know that's kind of the question i'm i'm interested in how do we reduce compute cost um you know a lot of players in the space don't really have a vested interest in you know making these models smaller you know because they want to they they want to be able to say we can provide you amazing services using our own technology uh don't worry about running it we'll just we'll just use it ourselves we'll give it away but you know there's there's a whole there there's a big gap between academia and industry research labs because of that you know because of the resources required so anything in i think any any direction in the space is a welcome so anyone applying these techniques more widely is highly encouraged um you know so one of the big things you run into is ram you know gpu ram is not very big you know i think we're starting to see like 24 gig gpus but are still super expensive generally most laptops you know that are just right you're even the things you see free on collab are like maybe six gigs of ram eight gigs of ram if you have like a decent you know you're still paying for that though uh like my laptop has an 8 gig you know like a 2080 in it uh it's not a 3080 but yeah i haven't needed anything else uh that's that's only eight gigs of ram you know so you know so you either end up using multiple gpus or you know like what my colleague talked about eduardo talked about like you know the aurora card which is cheaper faster than a cpu but has a lot more ram um you know and there's other there's other players in the space like server systems you know who built the biggest i think like the quote-unquote the biggest co-processor in existence right now you have graphcore doing these kinds of things so they're with their ipu there's so there's different alternatives you know they all have their own trade-offs of execution speed you know neural networks are good at ram et cetera you know the cost so in summary these things are expensive um and running these things in production can be even more expensive um you know so some people try to minimize you know either make it run on cpu that's what most people do and say that's fast enough uh or you know if it's like if if performance is you know real-time performance is mandatory then you need to set up all sorts of uh tooling around you know being able to run a co-processor like a gpu in a data center so components of a normal pipeline so what are the components so generally it's two parts your actual model and your input pre-processing so you know what people don't think about is actually optimizing the full pipeline from data collection uh vectorization you know converting it to you know a matrix or a tensor and then actually putting it into a model um you know that you know that kind of aggregation for recommendation engines you know big you know different kinds of systems maybe you know something where kafka is involved or you know whatever buzzword you have of the week for your streaming platform your database you know you need to be able to collect that data aggregate it effectively and then convert it to a vector and then put it in the model that part um you know you know where you know the tooling is attempting to solve that but i still don't feel like has solved it effectively i feel like there's still a lot of room to make that process easier especially when you're pre-processing different kinds of data you know nlp for example has you know the tooling around converting you know uh text you know you know which could be a sequence or you know just your standard bag of words to a vector varies widely and the form it needs to be in might be you know varying length so you need masking and all these other things so you need a weird mix of domain knowledge and then knowing what kind of tensor and shape to convert it to to actually put it into a model that whole workflow is kind of i think what hugging faces made easier but just knowing how that works you know there's there's ways to optimize that as well for people so just when i say pipeline i mean everything not just the the model that you're thinking of uh so you know a bit of a primer so i already kind of mentioned it you know raw data needs to be converted to arrays could be pandas could be whatever you can think of it could be an sql table um data can come from anywhere you know there's there's a wide variety of protocols nowadays for streaming you know just a standard web or you know your sql database uh data can have different formats you know in memory you have arrow now which i think has helped the ecosystem move forward as far as you know taking a pandas array directly converting it to you know you know without copying the data or anything into something that spark understands or flank or what have you so these things are getting better the fact that like the you know formats exist where two things can talk to each other and say hey we're the same there's no copy needed uh is beyond the you know the network copy or what's necessary to run it locally or just run the compute where it needs to that's really nice so if we can reduce that overhead great um data scientists probably are familiar with grpc but it's used everywhere you know kubernetes being a big driver of that you know so grpc allows you to generate you know your services your service definitions and then you can make two things talk to each other grpc itself can speak different protocols as well so it kind of gets a little convoluted but in general there's lots of ways to move data consolidate it and then make you know make make a model score whatever it is your your raw input is uh so data movement storage as its own set of problems uh you know it is generally a prerequisite you need to get that right before you kind of get into even the ml part so you know if we think about my you know the broader topic i'm mentioning here you know this is kind of like three tiers down you have like just storage then you have moving the data around then you have the vectorization part and then you have just normal ml models which people use and then you have the like the very specific thing that we're doing with trying to make optimized models run so you can kind of see like how deep down the funnel this really is so models uh for those that aren't familiar maybe maybe if you're an engineer you don't really know what you know how a model is stored it's used using hdf5 protobuf or pickle depending on the various frameworks you use um these models are big uh you know they contain like generally a dag you know just like this you know an ad is like x plus y output c and then that that's all connected by a computation graph um there's that configuration plus the actual matrices themselves uh which can be very big so models don't have to be all like you know not every model is 10 terabytes big or something but it's it's not uncommon to see 500 megabytes 250 megabytes two gigs those are not that those are not that rare uh you you see those quite a bit so again just knowing how to store that with your etl pipeline versioning all that you know generally will people just say well just put it into a docker container it's what a lot of people do um that has its own trade-offs as well uh that's what i think it's what you're kind of forced to do today especially since a lot of this code runs in python there's still really no there's there's like some good ways to package python code but it's still very sparse um and python is a server runtime not you know a lot of people running alternative runtimes for this stuff so what do you do in that case um so there's a lot of questions just around you know what do you do with a model file where do you put it how do you how do you hook it up to a model runtime what's you know there's there's a lot of code out there now that makes just running focusing on these file formats easier uh oh you know we all we have an opinion on this as well of course but you know there's alternatives like microsoft sonic's runtime um you know tensorflow kind of did this with tensorflow surveying like you know we just have something that's like g plus c plus plus grpc that knows how to load a tensorflow model and and execute it um you know so i kind of view models as a you know like you know one one runtime may train it another may use it um so that's that split paradigm has been around for quite a while because a lot of this underlying code is c plus anyways or java or you know some you know something aesthetically typed that that's easier to deploy so what you do with the model files up to you how you interact with it will vary but hopefully that makes sense so putting them together um ml pipelines are not just models etl varies can be json it can be a mix of things uh or even within you know it can be within the model tf.data as an example of that like kind of trying to extend that paradigm and then you have your metrics and experiments um so you know model tracking platforms try to do all this at once they generally have a very simple version of it and that generally will work for a lot of people especially when you're just doing a proof of concept and you know in a lot of you know something that you kind of takes a while to learn if you're in a software engineering bubble is most models don't actually need a software engineering team you know most the data scientists themselves are kind of the consumers of these models so you know just the the bare bones things that expire tracking platforms give is probably good enough you know so the tools need to get do better bringing this stuff to and exposing it to where the users are running the experiments so some of the key performance considerations better data interchange you know faster protocols than http better in mineral memory file formats for data interchange that's where something like arrow comes in better storage you know you need data stored in ssds generally i mean not everything it depends on your frequency it depends on the use case like a data lake for example will be your standard this is where all the analytics happen this is where we aggregate all of our data your data warehouse where like you know this is like the real canonical production store and the you know that has regulations attached to it among other things um and then you know your your pre-cube you know you know one one thing you know when interacting with any sort of co-processor um make sure that you whenever you're storing your data make sure that it's easily you know you can descend it all at once you know in big batches not small batches to the co-processor like the gpu the tpu because otherwise you know typically latency hiding will be a big problem what i mean by that is the overhead of sending data can be can overcome the benefit of the faster processor in that case you need to make sure that your data movement doesn't over oh you know kind of outpace your uh you know or be the bottleneck for your compute faster models um so something people don't typically think about is when they call model.save and keras or what have you um what's in you know what can we do to optimize it well it turns out you know precision is a big one your data type you know the smaller your number is the less time it takes to compute something uh you know gpus you know have this like mixed you know you've you've probably heard the buzzards like mixed precision a half you know fp16 half that's those are just uh basically numbers that are smaller you know like long versus and doubles versus float and kind of extending that out further um so if we can you know force models to be you know you only use certain data types especially for inference you know as long as they produce something like the same result then if it's faster then there's there's no downside to that a data layout you know so it turns out cpu is faster with nchw which means you know number channels like an image is rgb right so you have three channels so you have number channel height width like the height and width of the image uh gpus are you know channels last as it's called in hwc so there's like various reasons for this but just in general just understand that you know the way you input your data matters the way you uh the way the way you save your model matters and what's in the model matters like the way you write your it's just like when you when you write a bunch of redundant code that doesn't that's not needed for training and you just put it in your model that stuff can be pruned um so there's a lot of techniques around you know even just how you program these models nowadays so as i mentioned removing redundancy so you know so some you know some execution platforms will try to say okay what's the bare minimum what's the minimum spanning tree of the operations i need to run in order to get to uh the necessary uh result um you know if you can have less parameters the the better um and again format you know you you may need to export to different formats to run on different platforms like torque script normal pi torch tfla versus tensorflow there's all sorts of various uh reasons that these that these you know tensorflow versus tensorflow lite exists for example it's just because it's a different use case and the problem is that people don't think about is the support will the support the supported operations will typically be different um or the way they implement that support for a particular op can matter um you know for example like the way you might implement say a tensorflow op in onyx um might be combining a bunch of onyx ops together to approximate the result from tensorflow it could sorry my headphones were sorry my headphones were muted there uh sorry we're good to go i just like i just had to charge them um so anyways uh so yeah um so the the fight you know so the way you know so again support support will vary um so just be aware that when you try like converting all these framework you know all these things together so just converting model converters are a problem um so tools and techniques quantization is one thing um knowledge distillation you know kind of a teacher-student model uh train the smaller model train a smaller model on the output of a larger one to approximate the same results um batch norm folding so room so batch norm for those that aren't familiar it's just a way of saying uh you know what's the running mean and standard deviation of the mini batches normalize you know in the middle of the graph just normalize the kind of the output which you know kind of keeps it from diverging is kind of the core idea that's kind of slow it has its own overhead you know there's there's what you can do is you can actually kind of bake in the the mean and you can take that mean standard deviation and just bake it into the previous layer um so that that allows you to remove some extra redundancy from from your models and you know connected layers decomposition um so okay uh so optimizing it sorry i'm gonna i'm gonna hurry a bit since i'm i'm gonna be out of time soon um so optimizing data format uh change to a more efficient binary format pre-convert your data to something like per k or numpy arrays remove layers and minimize communication overhead which you know we kind of already covered earlier so some tools to be super concrete um there's you know there there's a lot of so grappler from tensorflow is one for optimizing tensorflow models um there's simplify for onyx which just removes a bunch of the uh redundant ops and things like that so what so that's those are graph optimization and then there's the deep learning compilers that's its own whole category so tvm is kind of the main one in this space but there's also glow from facebook and mli from google and what they do is they they handle the graph they handle the graph computation they actually like run kind of a search uh parameter search to identify the optimal configuration for given piece of hardware for a given graph um these searches can take 12 hours to two days to run sometimes so uh you know you know what i would say is if you're going to look into deporting compilers look for simple wins first um you know just you know just changing the graph structure for example uh versus you know like trying to do a full like end-to-end search uh so if anyone wants to know more about deporting compilers and all that i've played with quite a few of them if you want to know anything in particular but yeah in general uh just be aware that while this is a new space it will be important in the next couple years uh tensorrt optimization so um you know one thing i would say is uh if if a vendor provides an optimizer try to use that because they're they can they could make it you know one one thing tensorrt does is it will say oh you have a convolution operation here let me replace this with the coup dnn equivalent it does like stuff like that so again it swaps the result the ops that are executed in order to you know that are vendor specific in order for them to be faster and then for the fast ai framework there's also faster ai which kind of consolidates these techniques into fairly easy to use format so now finally the graph vm part for those that are kind of curious what you know what an end-to-end application might look like of this so first of all white raw vm uh it's capable of existing you know allowing existing jvm code bases to have faster startup times that's kind of like the main use case that people pitch there's also just the single binary use case like hey i can ship a single binary i don't need a java runtime that's always kind of nice you know we tried to solve that with j-link but you know there's there's there's always trade-offs to that so if we can do that you know with completely native optimized code that you know that allowed us to kind of wedge in like other languages like ruby javascript you know you can do all sorts of like weird multi-language interop and the the cross-language support is getting better um we mainly use it for packaging but uh you know compile you know combined with other things but it does have its advantages um we use java cpp for you know all the all the libraries you can think of in python uh we take those you know we have java bindings for all of them because java what javascript does is it actually allows you to kind of declare a way to generate the jni code for actually talking to c plus so we just run it as a maven plugin and uh it's great it's easy and i i haven't had to write manual j and i code in years we just use this and it's worked for a long time now and that allows it you know and coupled with that is the ability again to you know pass around native pointers and just pointer addresses and that allows us to get you know very very like a very performant code um you know despite using the jvm and then obviously we have you know when we when we're kind of like just debugging these workflows we can just you know debug in java with its great tooling this is kind of an overview of our opinion of this you know so what do you got in here you have yaml for the configuration yes yeah more yemel but that's it works um you know then there's the models so you have a model file you have a model you know you have a pipeline configuration so again a pipeline is what i described before it's the pre-processing even the post-processing like how you return the results of the user can matter like for example if you just want to return an end user a softmax array that's probably not what they want they probably don't know that you know each you know the max of the column on each row is uh you know the the selected class they may not know that so you need to give it to them in a form that they can take like cat or dog then they can use it themselves um so that whole pipeline plus a model file generally take that door tooling um you know we have our own opinion on this our same gift framework uh in dl4j um as again kind of our opinion on running math code we we we can actually import all these models generally and then we use it for our own internal use cases for packaging training models um you know you know deploying on android uh that you know there's a lot of various use cases and then edwell's just our own production systems um we have you know and then use grow vm you know like the combo of same you know our same different time vm and bike you know are in java cpp to create like one binary so one cool thing here that's not uh that i kind of have to elaborate on is you know in in our same different time we actually have the ability to say given a model pipeline we'll look at the graph and only can create a custom binary for just the data supported data types you need and the and just the ops you need that allows you to reduce binary size and this all just gets baked for free into the the binary you you never see this uh so if that not any that's interesting please do let me know and then again you can run on different chips you know like your standard inv you know your rmg you know your arm your nvidia your intel cpus and of course the aurora chip um and then how do you use it well same thing you have your you have your configuration yaml we have a python sdk uh and then your model file and so what we do is we actually have a syphon sdk that's linked against the specific binary produced by this process uh and then all you do is you just uh you you load a trading config and then you just you pass into pi arrays and then it's all zero copy there's no overhead uh we directly can look at a numpy pointer and say in memory and say oh we we know what this is and then convert it to our equivalent and then run all the execution and then return it as a numpy array so it's a really nice way to run end-to-end pipelines and package them as a single binary um and it's all aot uh thanks to brawl vm in this in this case in our workflow we produce a shared binary or sync sorry yeah shared library with the necessary components all in there so what's crazy is uh you know in practice you know it's generally a 500 gig 500 megabyte to one gig binary which actually isn't bad i mean compared to what you might see in the python world um you know generally what we do is we just produce binaries for the platform we want to run the platform and model we want to run on um all configurable from pi all runnable from python um we have jason yaml for just running the model config and then we have a python wheel for each platform for actually integrating with everything that's kind of again our opinion on how to consume all this tooling so at the various levels you know you kind of have the you know what you might do here is you might you know at this step you might do any sort of like model optimization before you package the model you might do you might look then after the optimization you might look at the data types and and whatever's present in there and then compile a binary just for that as well as the associated libraries that's what i would that's when i think of end to end that's literally add to ad and that's kind of the the goal of a lot of what we focus on building day to day um and then again just all consumable from python just by a standard uh python sdk um so for those that want benchmarks uh just on average you know a lot of this stuff is still kind of early yet but in our years of experience you know we found that you know we're still linking it's the same c code that python is so we're generally faster you know just because of the you know you know just because of python versus job as far as speed goes so for real production applications you know you know the main the major thing to think about is just you know your standard memory memory allocation and all those other things production but beyond that you know we're you know we execute the same math code everybody else does you know so since we have our own library you know we we use all the same optimized libraries people do like who dnn etc so in practice we haven't found many performance to many of any performance differences or in general like we've been faster um if people have specifics i'm happy to talk about it in the hallway track but anyways that's kind of the the those are kind of i think the core ideas i wanted to communicate so i will leave everything open for either questions or hallway track thanks for your time you