scale.bythebay.io: Ari Kamlani, Scaling From Research to Production with Skymind DL4J and ScalNet
so again I'm re feel for the room has anyone ever used steel for J or just got - tech alright cool alright so rather than kind of give me an intro to just a general stack focus on some of the other aspects of it permanently people come in from other areas meaning you come from cares or when you're starting out just a general prototype project and how you scale that so I'm gonna focus a lot on ingestion and modeling and then scaling your network and then I have a LCM network that kind of walk through kind of putting all the pieces together go through that and then add some resources for you so this is Roya hope what I hope you take away and scale from the initial research to production but when a lot of people start out with deal for J they're nuts not really sure like how all the components map together so I hope a good idea of what these different pieces are moving up to the front okay can anyone hear me fine all right cool so if you're not familiar my we focus only on the production side which is why we don't have any Python bindings so our complete stack is in Java and Scala so we're just in a different user base completely and that's why I like all our customers in that side we don't really work with startups that much there are typically more and the big enterprises we had developed the complete stack from scratch so we actually don't have a lot of other open source ones other than maybe like Java open CV we have a lot of different segments and we worked in government finance retail telecom recently started doing a lot more in robotics but Softbank if you're familiar with them they're the ones that acquired arm Scarah so and a lot of different use cases one that's not in here is probably more on the tech side more like part manufacturing so like Boeing it's one two that we work on a lot and all right now but computer vision you know just like you see with a lot of deep learning frameworks too we do a lot there as well so this is why we do everything in Java partly it goes back to our co-founder Adam it's pretty much head started this all in Java so it's been around since 2013 and then 2014 the stack was born and so stack is deep learning for Jay and then there was Jo for Jay and that's really not just like the modeling stack it's a complete suite so that will focus everything you need to do from how you ingest your data how you model on how you distribute it so 9:1 is our latest release and that was just in August but we're doing a lot right now and we've been recently accepted into the eclipse foundation so we're working on our first release for that and so right now I'm actua pulling a lot of stuff from our master branch snap chart snapshots are currently not working for us so I'm just pulling everything from master right now so and you'll see that there's a lot of different new features that I'll be talking about as well that's on the master branch and 991 and you can see that nine one was just very recent so I'll blow this up for you so it's kind of like where our stack looks like at the very top skeleton is kind of the equivalent of Karass that's a really lightweight one and we typically just use it for the modeling side everything else kind of comes out of the main deal for J side and so that's the suite from the modeling of it below that is the data Beck which is which we call our ETL side of it and then nd for J which is kind of like numpy for JP JVM there's nd for us yeah I figure yes yeah yes yeah that one actually you can do directly from Scala right so if you want to do the skull of map function itself just like in Java right it's just that we it's kind of like a wrapper around it so and e4j you'll use that all for java but if you wanna do stuff in scala then you can use and d-4s directly yeah yeah it's that's what I mean there's been a lot of changes recently this year so if you like pull from master and say the latest release then you'll you get that and there will be a newt another release coming up soon I'm not sure one because of the whole eclipse foundation that we're moving towards that's why I'm pulling stuff from master and jumpy is kind like now you can post do stuff around a Python interface to it so it's wave just like you would create an umpire way into it so that's part of what we're doing in the migration towards right now for a Karass we have a Karass model import but we're not currently a back-end to Kharis yet so that's in the roadmap so and then that's all in the Java side and then below that that's now we move down and native and the native side as well as a java cpp to live and e4j the hardware architecture and the chip libraries come in and all that kind of comprises what we call backends all right so even the Java cp1 we cross were from the Java roll to the C+ world right we don't actually go through the j'ni it's actually it's all like a manipulate just parsed so it's almost like function pointers so we don't have to pay the overhead in that and now we actually do all this off heap so all the end e4j stuff it's all off heap so we don't have to deal with a garbage collection and there's been advances in warth workspaces so you could essentially attach a memory to a particular workspace whether you got a beefy machine or not and then you can detach it when you don't need it so you don't have to pay the penalties for that the umm yeah yeah do you not here very like yeah okay yeah I guess I'm lucky every now better now so when we go over to the hardware architecture side of it now so we pick up on GPU or CPU and I'll talk about this later but you could switch out which one that is and by default like the Excel libraries when you get out from maven and just pull those that you get the open Blas but if you want the Intel MKL then you can integrate that into the Arabic art treasure on the back end all right so this con depends what you want to include and it's just a switch for a GPU versus a CPU yeah that one you actually switch at the palm level or you could switch by environment variables so yeah so it's basically you go go into a project object module just go ahead and switch your dependency as long as we see and you're on machine that has it will always go to GPU so these are the different use cases we got coming in different types of data so kind of like you're tall you're sent your sample data for a fits in memory and then we got the next slow which where you do your batch generators so not gonna fit in memory but you still batch it in the next one more you do a little bit more custom bayside because we have these kind of built in ones and you those are fine when you just put everything on your file system but you won't move beyond that and you can do your own custom generators for that too so we give you a lot of things for depending on the type of data that you work with as well as what type of volume of data you're working with and that's where you kind of fit with these three use cases as it was what happens when you got in balanced classes and so we actually take care of that for you out of the box so it's one up so this is part of our date avec side datavac is our whole ETL so we think about when you go ahead instead of just like ingesting all in memory and all the numpy rates you just basically ingest the location your eyes just a metadata essentially and then as part of the metadata when we get down through the vectorization ingestion then we'll actually go bring these in and a batch of time so you're not actually bringing it in at that time of it so when you actually do your transformations when you do your pre-processing and then transformations we can be smarter to such those transformations happen at the batch level you don't have to pre-process it and save that off and that's where you make these trade-offs and basically we have these there's a couple of different levels so it's like a record reader and that's the vectorization side of it from that rectory the record we are basically like imagine you have the image data you have text data you have sequence data those we have all these different record readers for you and then from that you basically have these data set iterators and that's how we batch those in based on your batch size and that will pull in from the metadata and from that you can define transformations just like in spark for example you define your schema for it or if you want to augment it like an Karras for example if you want to augment your image for example right you could do that at the batch level and not have to pre-process it and then save that to disk so here's like basic one okay it's back alright so imagine you got some training data you're loading in through just a CSV record so this is like the example this one you bring in this record reader you go ahead and define a file a split of where it lives it brings in that metadata and now you go ahead and tell it whether you do in like regression or not how many classes it has and you would shuffle your data now this is just like when all your data is in there you've essentially you're you're doing everything what you would normally would have done except you're just doing it on D like almost indexes instead of the actual data because that's what this year the the fossil it does and then you do your split test train and then you would just do your normalization but fortunately doesn't really happen when you start out scaling with lots of data so this is kind of like what you would start out when you just kicking off a project but when you want to kind of scale to the next one above that you have two different other sides of it so on the left hand side you still have the same idea of you load in you would go ahead and resample it for equal weight distribution and then you might do like a transform or execute but you kind of get that back right away right and so instead of doing that and then shaping it on like your CPU and pre-processing it proposed more on the right where you go ahead and you load the metadata you go ahead and do the balancing and we have ones that would come in through like a balanced tough filters but instead of actually doing the pre-processing and the transformations there we do it on each batch so that's where we basically just say let's tell the iterator in this case like 32 about sample batches that come in and you basically whenever you get the next iterator you do the transformation at that point in time and not earlier on as well as so it's not what I want to do keep losing it back and the last talk of the day okay its back alright uh that parently does not seem to work well worse all right got to squint i'll keep it that way but the main takeaway for this is that what I'm doing here is you see that I'm basically defying a transformation process right up here for example if I go ahead and just bring in all my data and I can set the schema for it and I can go ahead and define a transformation process and take that transformation and put it in a dataset iterator so I don't do that ahead of time and now that if the data this transformation did set reader basically just wraps to iterator so now I could be a little bit smarter and how you deal with it the other thing I'm doing at the bottom here is I'm actually doing a asynchronous this iterator so you imagine that you have a couple ones whatever your batch size is you basically go in and get these asynchronously and you can still continue to start to do in your training process so you just go ahead and find your queue size for that these are the come of some of the built-in fast generators so on the left side you would see more like a more like an image type example where you would see like for example test and train and you basically separate each folder with all your labeled images into that and on the right hand side you'd probably see a little bit more what your sequential data would be your you would see you like each one in this case sees 1 through 10 are basically different engines and within each of those here 3 files are different temporal sequences so in that way you don't have to worry about all this you basically have we have a sequence record reader basically ingested it and it'll take care of it and it's just you batch data so that comes in for you you just need to find what that unique pattern is for you so we have a numbered file one that comes out of the box for you but you could just basically define any pattern you want and feed that in alright rising okay alright so this is part of that pipeline on the left side the orange one is more of the images and this is what our typical workflow would have been figure out where your data source type bring in label images this parent path label generator we define go ahead and initialize it and then we go ahead and set our record reader and then we'll define an organization scalar notice that we don't actually do any normalization here we just go ahead and tell the iterator based on what we collect the training data on and then on the blue side is more of the sequential data and that's probably what I'll focus on more here and so now we have our numbered file split but bring in through the record reader we do go ahead and do the transformation process we're talking about and it's important that we collect the metadata for it so when we actually do the predictions for it we can actually correlate what that made day what it is for whether what type of engine was for or what type of image type it was and now we go ahead and do the asynchronous data iterator and this is basically your entire workflow up until you're ready to get training so this is everything we do for data back so here are some of the more custom based ones there's multiple different ways you could do it and you can either think of it as instead of actually loading in the data we go back to loading indices or in this case you're loading it instead of having that file structure that I showed you before you basically just say here are all the metadata I get this from like a CSV I get it from my own type of database and you could just basically ingest this in so damn it I lost you didn't I it came out all right oh no not in a few min so the basically I'm giving it doing a filter I'm reading this in through spark as a CSV file and just this in I'm doing a filter on then and I'm basically just converting the RDD down to a row and getting the URIs out of that so you could think of just go ahead and reading all these infamous CSV files so you don't have to have it in this custom directory structure at all you just load in the actual structures yeah so if you put that in this case I am reading this from a data set because you know a type data frame right yeah so the thing with with our side of it is I'm reading this in from a CSV a data frame right but when it comes down to it right we basically put it down to our world side of it it needs to come in through a list or an RDD right and at this point before we're doing any of the data ingestion side of it we need to go back from the essentially the collection so goes from data frame RDD RDD back to list so we do the collection side of it all right so the one thing is the car parts really because a legacy everything we do is at the RDD level I'm just kind of showing this because of the more advanced stuff that we do now with data sets and data frames but we've always been more native so if you think of like even big deal though more native to spark in contrast to some of the other ones where they lived a higher level of serialization so that's why we all our interfaces are always going to be at the art level all right so this is another way you could do it it with this collection input split you could also just inherit from this the data iterator and go and implement your own custom based one so now once it kind of got more of the basic view on basically I've just showing you more of us you got a single output you got a single input coming in but you may want to go to more advanced based ones with different metadata coming from different sources so in this case we got multi data sets and so you either have multiple outputs or multiple inputs right so in this case here I'm showing you with multiple outputs what I'm showing like you we have as part of this we have different features and labels come in from two different sources of files and then towards the tail that we basically have two different outputs instead of just like a classification one we have one like classification and then we have another one like based on like images and then we could do more of the temporal inmate based nature of it and this way we're still working with the whole data set iterator is just how we define it so this is important for a multi task and as part of the even beyond the input side of it if you have one type of data and now you can have almost two different types of networks coming in and so it's just to find any type of data set iterators as you get more enriched metadata for it so that's kind of like the whole ingestion side of it everything you would do up in toilet or in your network and so now go over to the modeling side so it's a couple of different ways that you can do your models the basic one is you're defining your whole entire configuration from scratch and then you would sterilize that and reload it the next side is the care of smaller import that so instead of having a people implement the entire configuration with India for J and doing everything in Java or Scala they can do it in Python and they could just import it in currently we do stuff more like an MLP just a more of a basic level and an image level the lsdm side of it currently does not allow us to return sequences that are not true so we're always returning sequences and when you go from a sequence to a dense layer we currently don't support that so we're currently in the works of report working so we could do LS PMS on the Charis model import but you could still do basic ones and image data for it the next one is we have role models ooh so just like you have all these other networks for it we have some ones that are native to it so I have a couple slides on those so depending on what your use cases is typically more popular with image type data but you could just import that in and do transfer learning on that so these are the different sizes that I was talking about so either you do initialization and care of small import now it's part of the model zoo we have ways as part of a selector so you could basically define of either you know the type of network that you want or you know it's the type of use case so in this case you could say okay I give me all the CNN's give me all the RN ends right now there's like basically all of them are CNN's and there's one lsdm that we use for text so those are the main ones we use now we are working on so far it's been like a terrace mal import but we're working on as part of our ingestion that you can start importing native based ones so the native backends so the one that working on right now is native tensorflow so you can pour those directly without going through cares so this would zoom in for you can still see that but basically this kind of shows the difference between DL for Jay and carrots if you're going to implement it from scratch and so there's two types of networks we have like a multi layer and we have a computation graph multi-layer is basically just a bunch of stacked sequential layers and the computation graph gives you more control to basically do more advanced configurations so you see like in carrots were basically we have this like input and stead of basically just doing model ads we basically just go ahead and just define what we want for the lsdm and this defined what the input layer next would be so we have that same way that we do here in this case it's just a blur it's just a type of graph that we define so in this case it's the computation graph we actually define all our hyper parameters for tuning at the same level here so competition graph configuration for it and listeners are very similar to in Karis what you do callbacks but they're a way for us to monitor our network if you want to monitor it like for example the score but we have just like you have like for example with tensorflow right you have two tensor board this is we also have our own monitor UI that we could march at ease so skele meant like I said it's a really simplistic one tries to model caris it only uses it for the modeling side of it so it doesn't use it for the dead of x side of it the evaluation side of it it doesn't use it it's a very lightweight lightweight one for just MLP based ones or even convolution nets so if you want to just kind of get started and you don't want to go through the whole Karis model import right you could do something like this it follows it's almost exactly alike to the Karis world so it's just giving you another alternative if you just want to get something started quicker so this kind of just gives you an idea of what care assisted a piano tensorflow CNT K and M X Annette is in beta and we currently we don't support all of the box carries functionality but we support a good chunk of it so like you could still do a lot of the basic ones that you cut coming out more than ones that for example would that we don't do right now is you know putting your own custom layer that you would get with Karis but we have our own ways to define our own custom layers so you can just do that natively in our own but for the majority of that you know besides like lambda layers yeah we could do most of so this is the care of the small import say you go to head if on the side you see on the left but you go ahead and save it look either just save your entire network or you could save your weights in your configuration file and you could choose to go ahead and serve your network then if you're not going to do for the training on it as part of that the we recently skill has been always been in our enterprise world and over the last month we recently made a community based edition so you could actually do like in a Zeppelin notebook directly in this and you could do it like on AWS or could do it locally it's currently run as far two cent OS and that uses it like a Zeppelin based notebook if you want to use it for for doing all your notebook based stuff there which traditionally has been difficult for Ali Cisco and Java world or so if you're not going to go ahead and serve it used to go and do it for training on it then you do your import basically out of that you'll get the type of graph whether you're doing and computational which is more than functional API in carrots or you're doing the sequential with the linear model and sick either just import like I said the entire h5 file or you could split it up between the network configuration and the weight separately and whether you wanted to enforce the Train config or not is how we do the model serving or not so you basically it's a way to font for you to just did you actually import it correctly or not or if you just want to do inference so here's a transfer learning the mazu a couple of the frequently used models are the net the Google net the vgg world the inception and recently the dark net and the tiny yellow for a computer vision for object detection so that's currently 1991 but is in the master branch and so if you want to do object detection and so if you imagine before where you got the classification and now you want to do your bounding boxes and it's been difficult at least not compared to Python world to draw those and around there you could get that now with a tiny yellow so transfer learning helper was originally started part of our model zoo and we actually had part of carrots we had a whole trained models side of it so that's kind of deprecated now and we tend to use more transferred learning side of it but transfer learning isn't really specific to the whole model zoo or whether you're using it for just a convolution or net so you could use it to import your model and then from importing your model you could then just freeze those layers and touch new layers to it as you continue to train so in this way we've defined a transform learning helper for you to easily do it the right side shows that a way for you to basically if you want to use in destroy pre-trained weights since the case like an image net for a convolutional net there are trade-offs for that I'll get into those but yeah you're not always going to be able to to use that so even if you import your model but you're not conforming to the the input dimensions and you do not scale your data to those input dimensions then you're basically falling back to scratch model because you haven't tuned to that exact architecture so this shows the learning helper the phase one so you go ahead and define import your model you get a copy of the model from it depending on which step the on model you have either you're doing the inference only or you're gonna fine tune it and the fine-tune B maybe you can change in learning rate you're basically looking at the architecture from what you imported in and you're basically going to say how I want to change my current architecture so my my following layers that I'm going to train on it now I want to change those you can also override those in the individual layers themselves and you can either we have what we are dense layer is really an output layer so you're basically going to set your feature extraction and that'll be freezing your earlier layer and then you would basically drop out the output layer and put our new output layer for it keeping all the existing architecture or your base going to drop out the output layer and add new layers to it so it takes money to pass depending if you want to freeze and then train additional layers from it or if you just want to do inference and just change state the output layer for it so this follows like so like Subway type map interested in that so it's just easy for me to basically visualize it that way with the our evaluator it's kind of like a metrics so we have an earlier one so it's just a basic evaluation what we've brought ones in now for doing ROC under the curve confusion matrices much like you would see in scikit-learn or Karis so you basically defined this evaluation class and you can define a fresh one this telling us the number of classes for example or you could get it from the existing data set iterator so as you put import it in you probably just want to go in and check that you got the same validation score that you got from like carrots or your Ma's ooh and then you want to move beyond that and there's two ways to do your future ization with the freezing on the right hand side you basically want to go through and feature ice each of these data sets and the data sets we get through each day set iterator every time we call the dot next and then we want to hook go ahead and save those to disk the right the left-hand side basically says instead of saving those two discs I'm just gonna process at all at one time so you could basically use an existing one or you could go from scratch and train it all the way through and the existing one will basically allow you for reuse for later on period time so you can save a bunch of data sets to it and then you can pick that up at a later point in time and then do your evaluation and then you would use the model sterilizer is how you would basically do your civilization to load in to save and then we live later on from disk so we have a lot of things they're just a little different mappings but very equivalent to its other frameworks so this is part of the monitoring site I was talking about like the tensile board it defines a way for you to monitor your network so you can see how its training over time or even now the validation happens and so what you do before you do any of the fit stuff you could either define it as in memory or I could find it as a file so when we go to spark you always either guy do one of two things either gotta do like at a file-based level and not in memory right as it's going across different distributors or you're doing it you can define your monitor UI running on a different JVM than what it's currently turning on in that case you'll we do like an enable remote listener and give it the another IP address but you could also just do it the file based way to and split the left-hand side shows and so you would basically go ahead and define this singleton instance and in this case I'm showing it in memory and you could attach a bunch of different listeners to it in this case we define one that wait for the in memory side of it and then we could go ahead and look at the score listener for that so you can see how it's training distributive lis and then you would just go ahead and go to your training model and just go it like port 9000 and that's the default port which you could override so come on time all right let's go through so this kind of goes through that parallelization so as we go through start at the basic ones the multi-layer and computation graph then we go up to parallel rapper this is a way to do single node if you only have one node you're doing it on like one GPU all right and then you could go up through the SPARC layers and we basically that these are just kind of wrappers around your existing architecture so this is a single node one and recently we added parallel inference as well that you may actually want to do inference on a single node or not locally instead if you're doing like batch operations and I was added and I know I believe so if you or though just doing the single note even though you could use spark for this right then you couldn't do better by doing the parallelization with the parallel rapper even on the training side of it so we recommend using that one and this it uses the existing network and you basically just define how you want to distribute the workload and it's very similar with the with the parallel inference the multi Chibueze this is the pomp I was talking about where you can basically just change like this single line up here instead of native platform you can just basically define they could a platform that you're coming in on and then automatically switch to the GPUs an alternative would be it always go as long as you're on a GPU and they have the native platform it'll go directly to the GPUs but you can override this - OS environment variable as well and the higher number will win and contrast the chancellor flow it you could basically and said the same what package you install it's very similar to that or doing device scoping so we're very close to that another thing is floating precision so you could scale down instead of doing like the f32 right you may want to scale down and go to FP 16 and it's literally just a single line for that so this is way for you to kind of save on the memory side of it and since I'm running short on time I'm gonna run through this so this kind of shows you as you're scaling up that all your existing network architecture would stay the same and what we're changing here is really just this parallel wrapper when you take the existing model for it and now you're still gonna do the fit on it it's just that instead of fitting on the model you're fitting on the parallel wrapper that it took as an input the next one it's a little bit more this is the apache spark one where we basically define take your existing configuration of the network and then we define a parameter averaging this one has been in there for our legacy and we take a training master and decide how you want a parameter averaged across and the scaling out this is new and I'm one so spark may be too much of an overhead for you and so when in that case you're not using spark then we have ariane in there they use AK underneath the hood for her message-passing for low latency and it's just a different type of training master it's a void configuration basically defining the type of port for you but you see that everything all your other entire workflow is exactly the same yet so this furniture last one this kind of shows the difference between the primer averaging and they shared gradients and you can see that it's this is just a different type of training master originally we started training master with the idea that we could have many different ones for a long time it was just parameter averaging that we used in spark and so recently and this was out in here where you basically just define the UDP port for unicast and then you would go ahead and give that share training master give that as an input the threshold defined set instead of actually saying I'm not going to send it all one time you basically defined some threshold where we store those to memory so the next time those include thresholds go over and I had an example to go through but there's a network but I think I'm kind of over on time and it gets questions anybody have any questions about how to you forge a yeah I say yeah oh yeah I don't know I haven't tried that one yeah I'm guessing that it's gonna require some customization to it right but I don't think it should be a heavy side of it because we've had some customers come in and it's part of it and not using cue ball but still I think it's just an integration point and this is what traditionally our skill has been at the skill level what we did all these vendor integrations because people didn't have that Python bindings right that's where a skill level are done and it hasn't been a huge effort in the past so I can't talk about cueball specifically because I don't know anybody else if so it's a little time but basically in the github kinda find something to workflow putting all the pieces together that I kind of showed you all the way up to the inference side so it shows is like an ingestion coming in and from the ingestion then we'll take a regular Network and then from the regular Network then we'll go to a spark one and then from the spark you could basically show different monitoring side of it there's a Karras notebook in there too for you so if you're interested or have any questions feel free to reach out but that was one that I was hoping to get to but so I guess a lot and so feel free to reach out to me family any questions as well and there's a couple examples of some way all our stuff is open-source pretty much other than some small pieces on the enterprise for a skill but it's all on github deep learning for J and all are even like the datavac side of it and D for J and before us it's all up there too and we have DL for J examples directory and recently there has been in tutorials deep learning for Jay added and those have actually been exported notebooks that came that we've exported them into ipython notebooks take a look at for documentation side of it and those are actually in Scala so if nobody else has any questions I guess go dead panel [Applause]