ai.bythebay.io: Adam Gibson, Deep Learning with the GPUs in Production
Recording: ai.bythebay.io: Adam Gibson, Deep Learning with the GPUs in Production
. Okay, so I'm Adam, the co-founder and CTO of SkyMind. So, my talk today is on deep learning with GPUs in production. So, most of you probably don't even use GPUs or know what a GPU is, or maybe you know it in the context of gaming and the like. So, what I wanna do today is just walk you through how you might deploy a system that uses GPUs, very similar to what Salesforce presented just earlier, where you have ML pipelines, there's some scale and spark in there, there's some big data, you know, and all these other kind of crazy buzzwords. So, what I wanna show you today is how to kind of integrate your big stack into GPUs. So, believe it or not, it is possible. So, just a little bit about SkyMind
does not hire in San Francisco. I'm not even based here. I used to be. We actually mostly hire in Asia. We just made our fourth hire in Japan. We do a lot in China. We have four subsidiaries across the world. We're a completely remote team
So, I actually, we actually, the problems we kind of solve, I actually solve problems for Japanese banks. They still use COBOL. And we're integrating deep learning into their stack. So, it is possible to do this kind of stuff. But, you have to get in a suit. You have to do sales. Oh, God, no. Right? So, I know I'm talking to engineers today
So, I'm not gonna talk too much about that. But, I just wanna say, deep learning is not Silicon Valley, and it's not only Google, Facebook, and all these other companies. It's also not only Salesforce. You know, so, it is possible. You just have to solve the right problems. Most of the problems we solve are not computer vision related. They are not speech related. They are time series data, just like what you see with Salesforce
CRM data. Factories. There's a whole world out there for deep learning, and AI, that is not logistic regression, on Spark, with MLlib. And it's also not TensorFlow and Python. It's, again, it's, so, I'm coming from a very different world. You guys probably are not used to. So, what I say may be controversial here. So, that being said, so, what I wanna first do is define production
So, you know those blog posts you read about deep learning, right? You see, this latest thing I did with TensorFlow. Well, that's probably a startup, and there are probably only like five or six people, and they just wanna, they publish a blog post for marketing purposes, because they wanna hire AI people, right? But there's actually, and then they say, we put this in production. What they didn't tell you was say, Zendesk. had to manually patch TensorFlow and actually put it into production. That's what they don't tell you. What happens when you deploy a hairball of Python and C into production? There's a reason when companies scale, they move to the JVM. So, I'm highly biased, in that respect. But what you'll find, you know, so what people actually do is, they'll go to Google Cloud, they'll paste their Python script into Google Cloud, and then they deploy there
That's why Google's been gaining a lot of market share, is because of them productionizing this stuff for you. This is why a lot of these big companies do ML as a service for you. There's a reason Salesforce has tons of customers with all this machine learning data and all that. It's a hard problem to solve. ML infrastructure is probably one of the hardest things you could possibly do out there. And so, most people don't do production right, and they don't say what form of production they're coming from. Which is why I'm hoping to define these things for you today. What we deal with is enterprise production
So there's lots of bureaucracy. There's some COBOL in there. There's some Java. There's some Hadoop. There's some Spark. There's some Teradata. Wait, Teradata, yeah. So, you know, I'm the guy who sees the proprietary products
These companies have large market caps, and they're deployed somewhere. You don't hear about them because you're here in San Francisco. When you go outside of San Francisco to the rest of the world, where most of the money is, then you find alternative environments. And that's a lot of what we focus on. And then finally, research production. It's not really software engineering. It's just, I want the best research environment possible. Python and C, MPI, multi-GPU
A lot of the stuff you, a lot of the great stuff that you hear about. So, so research production is probably what, you know, the, say, the data scientists at Salesforce, Facebook, they all have this awesome infrastructure, right? But that's, but it's optimized for research. And it should be. Somebody needs to, somebody needs to meet DCGAN. Somebody needs to make WayGat. That stuff needs to exist. I commercialize it. You know, so there's, there's different levels of AI adoption
And we, we kind of tackle, we kind of tackle the, the early adopters. Not the innovators, the early adopters. The people who can use AI tools. And so, so when, when you think about production next time, you're probably in production at a startup. And most, I'll just say, most of you are probably not going to need half the stuff about what I'm about to talk about today. It's just, if, if you take anything away from my talk today, it's a thought exercise on, well, I'm going to do deep learning. I have some sort of an image pipeline I want to do or whatever. Or I just want to try this out
This, think of this as a thought exercise for how you might want to deploy this to production. And think about the kind of production that you're doing as well. And especially as your company scales, you know, like, most of you come from varying, varying sizes of companies. Some of you might be able to spend millions of dollars on infrastructure. A lot of you can't, which is fine. Just, just be realistic about, about, you know, how much time you have in a day. And, you know, what, what, what the return on investment is for, even attempting this kind of stuff. So, so the other, the other thing I'll just mention is different kinds of GPU clusters
So it turns out, it turns out, GPU clusters and GPU production are actually very academic. Actually, I think the, one of the largest, so there, there's a few, there's a few kind of large deployments of GPU clusters out there today. There's Google for search. Google and Baidu both, both use GPUs for like rendering. They use GPUs for various kinds of classifiers. And then there's self-driving cars. So Nvidia actually has a lot of market share in deploying GPUs as embedded chips in cars, for self-driving, for cameras and that kind of stuff. What I'm going to talk about today is a big focus on integrating GPUs into your big data infrastructure, such as Spark, Flink, basically JVM based systems that use a scheduler of some kind
Mesos and Yarn. So those are things that you can download today. Those are, those are things that are supported by your Hadoop vendor, if you use one. Maybe even on Amazon. If you're on Amazon, you're using Yarn. So everybody has a scheduler of some kind. And what I want to do is kind of just walk you through like, what that kind of infrastructure looks like today. So, so this is, this is, this is, this is, so this is what I'll call like kind of a reference architecture
Right? A reference architecture for open source big data. Right? So you have, you have, you have tons, you have, you have tons of different ways of, kind of executing big data today. Spark is the dominant one. But you also have Flink. You have various, you know, you have, what, Apex. There's all the, you know, there's kind of all these streaming frameworks. The JVM is spoiled. There's, there's, there's, there's, there's tons of great open source Apache license infrastructure that you can commercialize today, supported by various vendors, but also kind of, but, but also just, has lots of community documentation
And so, don't think about just Spark. There are others, despite, you know, again, despite the hype, there are other, there are other frameworks. So think about, think about what's right for your use case. Like if you're doing streaming, think about your latency requirements. If you have actual latency requirements, you're not using Spark streaming. You're using Flink or, you're using Flink or an alternative. Most of you don't have those kind of crazy requirements though, right? So just, just kind of, just, again, just, just think about, just think about what is this technology actually good at? Right? So, so this is, so this is, this is kind of your big data infrastructure. And then, and then a great, a great example of a scheduler is Mesos
So Mesos actually supports GPUs today. So like you can, so when you, when you have a scheduler, right? You have, you have the ability to say, I want 10 CPUs and 5 GPUs for my Spark job. So that's a scheduler. It gives you those resources and then you run the job on your distributed compute cluster. So that's, so that's, so you can actually deploy these kinds of things today. And you can, you can actually allocate GPUs to these things. And so that's, that's the kind, that's what I want to focus on today as far as your, your, your typical infrastructure goes. All right
So, so what I'm going to show next is this is, this is, this is, this is, this is how we do it with deploying 4J. So deep pointing 4J disclaimer, that's the software I wrote. So I'm going to be opinionated about this topic. But it is what we've seen kind of in production, right? You know, you can, so, so if we just think of, if we step back a bit and just think about the generic components here. It's actually very similar to the, the, the stuff that Salesforce put up earlier. On your, your, your, your ML infrastructure, you have production serving, you have training data, you have feature engineering, you have your file system, you have your storage. And then, and then finally, at the end of the day, you have your matrix compute. So that's, that's, that's something that's not talked about
I emphasize matrix compute today because it turns out neural nets are actually just a blob of matrices. Everything's matrices and tensors. NumPy. So, so if you, if you think about, if you think about where matrices run, matrices run on chips. They run on CPUs and GPUs. So when you're running on CPUs, you actually have to care about speed. You have to care about SIMD instructions. You have to care about, you know, does it support AVX or not
So, so when you think about, when you think about these things, chips have, you know, they have compute units. Right? GPUs have more of them. GPUs have more compute units. And, and they're actually great for running parallel matrix operations. So why are GPUs good at this? GPUs started as, for graphics rendering. You know, when you, when you have, when you have your screen, something, something is rendering all those pixels on your screen. It turns out that, that's actually a gigantic blob of linear algebra. So, when you're looking at, when you're looking at the stuff here on your screen, there's actually thousands of, thousands of, if not millions of matrix operations happening on your computer that you have open this second
So, that's what, that's how GPUs started. They started at, for graphics rendering. And it turns out, in the last, in the last couple years, since I think 2009, Andrew Ong published the original paper on, using GPUs as a deep learning. It turns out, it turns out it was really good. And, and so now, now, now, you know, seven years later, we have all this infrastructure and all this hype around deep learning and GPUs popping up. And, the reason for that is, because it works. But, most people haven't figured out how to use these things yet. Your boss never figured out, why should I use this GPU thing? Why should I allocate a budget for this? I don't get it
You know, and so, that's the problem we face today. How do we productionize this? How do I integrate it into my cluster? How do I, how do I justify to my boss that I need, that I, that I need this new GPU? It turns out, it turns out that this is still a hard problem today. That being said, there's a number of cloud vendors, Google being the last one that just added GPUs to their cloud offering now. So, now you can, now you can have commodity compute infrastructure from your cloud vendor of choice, Microsoft Azure, Google Compute Engine, and of course Amazon, who was the first to add these things. So, it is possible to deploy something like this today, where you have a data, you have your big data pipeline, you have GPU nodes that you add, your scheduler knows how to allocate GPUs, and then you can have your, you can have your neural nets deployed on top of a system like this. And so, in this case with, in this case with our infrastructure, we, we integrate everything via, via Spark and Flink as, as a, as a job. So, you're, so, you're, so Spark and Flink know how to use a scheduler, and then they know how to ask for, I want to run my compute job on 20 cores and 20 GPUs. Most of you probably only use one GPU though
Just, just say it. You know, that being said, you know, that you can do multi GPU. There's all sorts of crazy configurations you can do. But, at the end of the day, the only thing you need to care about is, more GPUs means more speed, if you need it. So, you know, you have, you know, so if you think about these large scale problems, like the YouTube 8 million data set, the ImageNet data set, you guys probably don't have data sets that big. You probably don't need, you know, a GPU cluster in order to run. But, your team might. So, this, so this, this kind of infrastructure is actually optimized for whole teams, where each team needs a dedicated GPU or a dedicated GPU node
So, when you, when you, when you have a company like Salesforce, they probably have 20 teams doing deep learning. And it turns out, each of those guys needs a GPU. That's what a GPU cluster is for, is for, for that, for that kind of scale. That being said, most of you can probably get away with one box with four GPUs in it. That's most, that's realistically what you're going to need. And so, so in this case, if, if you're going to deploy something today, you're not going to use something quite like this. But you can. Because it's already well integrated into the infrastructure today
Like you can, like one thing, one thing we do is, one thing we'll be able to do like on Amazon, is I can actually use, Amazon, Amazon, the, you know, their Hadoop. I can use their Hadoop. And I can actually schedule GPU nodes. And run, and run GPU, and I can actually run GPUs as part of a Spark job. This is actually possible today. So, that's, so, that's what we, that's kind of what, that's kind of what, that's kind of what we built this for. Was for commodity compute, and lots, and lots of GPUs, shut it down when you're done. When you have elastic infrastructure, that's what this kind of stuff is for
That's, you know, so you see all these complex things, you see data pipelines, you see GPUs, you see CPUs, you see all the . It turns out, it's actually this complicated of a problem, though. So, when you're building a production, when you're actually building a production system, not even adding GPUs, you still have a lot to think about. Like if you want to build an ML model that affects a hundred million users, then you need to think about all these things. Like, how am I going to scale this in production? Like, how am I going to, how am I going to have a hundred million users hitting my, hitting my model every second, giving me a recommendation, or giving me, you know, giving me like a, a classification of this is fraud or not. When you think, when you think about problems of that scale, then you need to start thinking about these things. So, all I can say is, if, if you, if you look at this, and you break it down, data scientists only think about one, one little corner of this problem. One percent of your infrastructure is actually written in Python
The other 99% is all JVM. So, you'll notice, so you'll, you'll notice like the Salesforce talk. The Salesforce talk started in Python. It was a great demo. And then you saw Spark and Scala, right? That's, it turns out, that's how, that's how a lot of companies do it. So, in this case, so that's, so that's a lot of what we're doing up here. Most of this infrastructure is JVM. And then, and then, our, our kind of contribution to the ecosystem is we made a, we, we, we wrote a, we wrote our own stack for memory management on the GPU with the JVM
So, it's all integrated as a Spark job. You don't have to know how the GPU works. And you can just use it. And that's, that's how you should be thinking, so that's, that's how you should think about the rest of this talk is, why is this so easy? And what's involved? Okay. So, this is, so this, this, so this is, this is a mini example of DL4J. So, you have a, you have a neural net architecture. And you have the error, you know, you have, you have the learning error going down over time. This, in this case, this is just lenet
And so, this, this, this is, this is what a, this is what a typical training on a neural net looks like. You have, like, I'm not gonna do a demo today, because it turns out deep learning takes a really long time to train. And this talk is way too ambitious for 40 minutes. So, I'm gonna try to cut it down as best as I can here. But if you have any, if you have any questions, I'll be in the, I'll be in the back afterwards. If you have just, if you just have any questions about deep learning or what have you. So anyways, so this is a simple CNN. In this case, this, this is just, this is just a simple, like, MNIST
Like, you, you run it, and, and you can see, like, the batches per second. You can see, like, very, so you can see the various hyperparameters. So if you remember, so if you look at all the hyperparameters up there, actually, one thing I'll just note is that a lot of these are the same ones as you see for logistic regression. So it turns out, so it turns out these, these, these linear classifiers that you hear about have the same hyperparameters as neural networks. So the same tuning ones apply. It's just, imagine, imagine that problem spread across six or seven layers. Each of these has, can have their own learning rate. Each of these can have their own gradient
And then, and then things, and then things start to get a little bit more complex. And so, that's why deep learning kind of needs, needs a special chip in order to run, in order to run effectively. And so, what you'll see here is this, so this, this will run, so this, this is running on a . And so, you could also, you could also hook this up to a Spark job as well. So you could just have a Spark job, and then you can, you can just, you can just, you can just watch this neural net architecture train on a whole cluster. But the, so, so one, one, one point I want to make is that, whether it's running on a cluster, or a single node, it turns out that the, the data scientist, this is, this is what they see regardless of how many nodes it's running on. And, and that's the beauty, that's the beauty of well set up infrastructure. You don't have to care about how many GPUs it uses, or what have you
The only thing the data scientist sees is training time, and they see how well, they see how well their neural net's learning. And that's what they should focus on. So, data scientists in Python are specializing in that kind of stuff. They specialize in the math. They know how to interpret the charts. They're not the ones deploying the infrastructure though. If they are, I'm sorry. But, you know, well, you know, that's the thing though
Like, infrastructure's hard, right? You know, there's a reason data engineers, that exist. You know, so, but you need specialists who do the math. This, and this is what you guys should focus on. This is where the money is. Because, it turns out, infrastructure engineers are everywhere. This is a problem that will be commoditized probably in the next, I'd say five, five years. You'll, you'll see a lot. It's, a lot of this is already commoditized now
So, if you think about anything from my talk, it's, infrastructure is hard, and ideally you just outsource this. And you can just think about the neural net architecture and the math. So, this is, so this is an example of kind of our, kind of our production stack. So, I'll just, I'll just decompose this a bit. So, you have, so you have a data source of some kind, right? You have your IoT logs, you have your Wi-Fi, you have your SQL database, right? And you, and then maybe, maybe they're hooked up to like Kafka or Storm, some kind of streaming infrastructure, right? So, so you're, you're doing, so you're all doing deporting on GPUs in real time, right? All you guys are doing that right now, aren't you? No. So, most of you, most, maybe, maybe some of you, so this, this is actually what a batch job can look like as well. Like some, some people will just, some people just have new data coming in from their website or whatever. They street, and then, and then they, and then they're, they have some, like Kafka or something consuming from that, right? And then, and then, and then, and then, and then they update their model
It turns out a lot of people actually have, have incremental online infrastructure in, within their companies. They have something like this. Maybe they don't have the, you know, the Lambda architecture where everything's online and everybody's always doing online learning. Most of you are probably doing batch, which is fine, but it's still the same kind of infrastructure. You still have, you still have real time data coming in. You have a data source. You have, and then, and then you have a data pipeline of some kind. So, data pipelines, it turns out, are hard as well
So, Salesforce, turns out, has a whole library to do it. We do as well. We call ours DataVec, though. Everybody has some sort of an internal vectorization DSL. Maybe it's domain specific, but they have some sort of data pipeline. You know, you define your features. You preprocess your data because all data is messy. CRM data is the worst, by the way
So, Salesforce has a very hard job. So, I understand why they built the software they did. Anyways, so, you have some sort of preprocessing happening. You turn it into a feature vector. And then, the feature vector is where the linear algebra comes in. There's probably hundreds of thousands, hundreds if not thousands of examples being into a, fed into a neural net simultaneously. So, you build your model. You train it on GPUs and CPUs
And then, you put it in production. And then, when you put it in production, that's when you're deploying to some sort of web service that talks to a mobile app that also talks to a website. Right? So, that's your scaling problem. So, the scaling problem actually comes from two sides. It comes from training, and it comes from inference. So, those are both completely different life cycles with completely different tech stacks. So, for example, when you're deploying a model in production, you're actually typically using something called a microservice. It's an isolated, independent piece of software that's just basically an API that just runs a model
That's its sole job. It turns out, the reason you need that is because it's computationally intensive. It has different latency requirements than just, say, a crudax. Create, read, update, delete. It has different requirements, which is why it needs kind of its own service. So, everybody does this. You know, usually, typically data scientists, they're probably using Flask. You know, you deploy your Flask API
You upload your image classifier. And it tells you whether it's a cat or a dog. So, even something as simple as that is a microservice. When you're Salesforce, you can't really do that. So, I'm picking on Salesforce because they were the previous talk. But, when you're Salesforce, you have hundreds of millions of users. You don't typically do that. You're typically deploying on the JVM
You're usually using some sort of like a, you know, Aka-based, just something JVM-based. Something that hooks up to Kafka. And that's where you're doing your real-time inference and that kind of stuff. So, it's probably, in this case, a JVM-based microservice. Like, okay, and just say Drop Wizard. So, we used to use, we actually, this slide's a little out of date. We used to use Drop Wizard. We actually just moved to Play recently
But, Uber uses Drop Wizard, though. You know, most companies have some sort of, you know, when they hit scale, they're usually deploying some sort of dedicated service for this. So, software engineers are the guys worrying about this, typically. Data scientists might build this stuff for demos. But, when you have a, you're going to typically have a dedicated engineering team that's going to worry about scaling your model. So, again, so again, one thing I want to throw out there is that data scientists, again, only think about, they think about one part. It's in Python. So, Python dominates that world
The rest of it, the rest of it, again, is JVM. So, you have data science and you have data engineering. Pick one and do it really well. Because both are very hard problems. Both require specialists. So, one thing I want to throw out there is, like, what, you know, some of the concerns you should be thinking about when running things on, when running a GPU cluster. One of those being memory management. So, it turns out GPUs, GPUs all have their own memory requirements
So, what happens, a typical GPU cluster, is maybe, you start with one Linux node, and then you have four GPUs in there. The GPU is probably a P100 or a K80. And that GPU only has 12 to 24 gigs of RAM. It's actually not very much. You know, when you consider the amount of CPU RAM that's out there today, where it's in the hundreds of gigs, and you don't have to worry about it, you all of a sudden have a new set of scaling problems. You have extra RAM that you need to worry about. So, when you're deploying GPUs, GPUs in practice don't have very much RAM. And so, what you spend half your time doing is figuring out, how do I shard my job across four or eight GPUs? Because you need to shard, and you need to shard when you're deploying these things
It's kind of an interesting problem, because now, on a single node, all of a sudden you have a distributed systems problem immediately. You know, it's like, how do I shard this across, you know, you're not asking that question, typically, unless you're on a cluster. Right? Because you think about, how do I shard my job across multiple computers? Well, it turns out, each GPU is a computer, and typically, each GPU, unless it has its own connector, like its own PCAE bus, you typically can't share RAM. So, even when you're on a distributed system, you have to deal with constrained memory. This is a problem that's being fixed, but today is still going to be a problem. So, this is why a lot of people, especially when they're training image net models and that kind of stuff, think about multi-GPU immediately. This is why Nvidia makes money, because they're like, we're going to keep the RAM restricted, and we're going to force people to buy ten cards per box. That's what they want to do
Right? So, that's again where something like a GPU cluster could come in. The GPU cluster problem, whether it's on a single node or on a cluster, it looks at things the same. So, when you ask Mesos for ten GPUs, it might just be because you have a really large problem, and you just need to shard your data set. So, the next one is throughput. So, throughput is measured two ways. One is latency. So, 500 milliseconds, 1 second. So, that's when you're doing training
Or, when you're doing inference, sorry. When you're doing training, you need to think about, how fast do I want this model to train? So, typically what you're doing is, you're thinking about, okay, how do I make this job fast? Either how do I make it faster inference, or how do I make it faster training? So, there's always some sort of an optimal cluster configuration. There's always an optimal throughput. Always be measuring your training time and your inference time. Like, okay, so let's just say something concrete here. Say you're deploying ResNet. Say you're deploying these crazy 150 layer neurons that Microsoft Research does. You're not gonna put that, so it turns out, you're not actually gonna put that in production
Despite the hype, what you find out is that it actually has a two-second latency window. When you're actually using the normal production, it might take two seconds for running the inference. The more complicated your model is, especially when you think about, say, attention models. again, very hard to train. GANs, generative adversarial neural networks. We're not using those in production. We evaluated trying to use those, and actually we just ended up picking variational autoencoders. Because they're actually practical
So it's not only training difficulty, but it's, can I actually deploy this thing in production? No, you'll probably use BGG or something like that. You see, every deep learning framework hypes their demos. You know, we just deployed ResNet. We just deployed, you're probably not putting it in production, though. You know, as a hobbyist, it might not be a big deal, but when you're deploying in production, and you're serving hundreds of millions of users, you need to actually think about, do I need the model to be that complicated? Let's see what else here. Okay, so, okay, a covered GPU allocation. Okay, so, one thing I'll just mention is, a lot of frameworks stress this, you know, we have a Python interface. And we're probably, we're talking to something JVM-based
We're talking to Spark. It turns out that this, it turns out that that process is expensive. You know, some folks will wrap TensorFlow in Spark, and they'll just say, here it is, guys. It's easy to deploy. It turns out that, it turns out that when you're running that in production at scale, you're actually gonna, you're actually gonna run, you're, you'll run into unexpected things later, when, when, when you actually deploy this. So, ideally what you wanna do, is you actually wanna minimize the amount of, the amount of communication between Python and Java. So, one, one great example of this that you guys probably use today is PySpark. So, it turns out, IBM published several papers on this, actually, where they measured, where they measured the throughput of Python and, PySpark to Scala, and they, they found significant overhead
And so, this, so, so, so when you're, when you're actually deploying this, your, your, your, your, your, your ops guys actually have to worry about Python runtime and the Java runtime. You know, when you ask, when you ask, for example, why did my training job run out of memory? Did it run out of memory in Python, or did it run out of memory in Java? All of a sudden, all of a sudden you have two runtimes you need to worry about. So, ideally what you do, and again, again, this is, again, this is, this is kind of how we do this. And so, the way we do it is we have a frontend in Keras, and then basically what we do is, and then we just, all we do is we send the model to Keras, to, to, to, to the JVM runtime, and then it does everything. So, so that way you don't, that way you're not troubleshooting two runtimes. So, all I'll say is, ease of use has a cost. Ease of use can have overhead. There are ways around it, but just be aware of it
And so, the way, so the way we do this is, we actually, we actually, so we actually do all of our own, we actually, we actually maintain the whole stack ourselves. We do our own memory management. We wrote our own GPU garbage collector. We wrote our own GPU, we actually also wrote our own Java based, like, GPU infrastructure for, for, for GPU memory, GPU management. So you can actually deploy this as a Spark job. You can say, Spark job, allow multi GPU, and this is, this is the amount of cache I want to use, this is the amount of RAM I want to use, et cetera. And so, if you can instrument this, if you can instrument this from your Spark job, all of a sudden, you, all of a sudden, your, your ops guys can, your ops guys can worry about the, the infrastructure. They can worry about how many GPUs is this using
They can, they can, they can, they can worry about this stuff. So that way, that way you, as the Python data scientist, don't have to worry about this. And so, so, so what, what, what, what we do is we focus on making everything runnable from the JVM, making everything run as you normally expect it on, you know, as a Spark job, as a Flink job, with everything running transparently. So, we do our own memory management, we do our own, we do our own scheduling. So, in this case, what you can do is you can, you can actually just say Spark job with four GPUs, or ten GPUs, or whatever. And this is, you know, this is, this is the amount of RAM I want to use. And just run it, you know, you can just run it as a Spark job. And so, it turns out this is, it turns out that this is a lot easier than configuring it all from Python, and then having Python allocate the buffers, and then maybe there's, maybe there's some interconnects where, you know, something Python based is something, talking to something Java based, and then there's all this overhead, and then, and then, and then, and then you, ask, why is my training taking so long? It's because a lot of that overhead
So, that's a lot of, that's a lot of what we fix. All right, so, how much time do I have here? Okay. So, all right, what do we got here? Okay, 15, all right, thanks. Okay, so, so now, so now what I'll do is I'll go over kind of workflow. So, one of the, so, so what we do, we actually interrupt with, what I would say is, I would, say the most popular deep learning library, high level deep learning library called Keras. So, Keras, I think, is the number two deep learning library out there in existence now. Recently got merged into TensorFlow. We're a back end for it
Microsoft CNTK is a back end for it. And TensorFlow is a back end for it. Kind of a factor interface for doing deep learning. So, this is, this is, I think this is kind of the industry accepted standard for what is, what, what, how does, what is defining a neural network look like? So, that's a lot of how we do it. So, what we do is work Keras back end. But all we do is we don't, we don't, we don't actually have the overhead of sending data back and forth between Java and Python to, to, to, to another back end that, that, that then runs everything. All we do is we define the DSL and then, and then we run, and then we run everything ourselves in Java. And so, that, and then all of a sudden, this stuff that I was talking about earlier, minimizing overhead, everything JVM based, you, you get the benefits of that with none of the problems of, well, we wrap TensorFlow and then TensorFlow to Java and then something else
You know, you have, you don't have three layers of, of complexity that you're dealing with. All of, all of a sudden you just have, you know, it's kind of like Google Cloud. Send your TensorFlow script to, to Google Cloud and then we'll take care of optimizing all of it. So, very similar idea. So, we, loading everything from Java, running everything from Java, and minimizing the amount of overhead you're actually dealing with. So, therefore you get faster, you get lower latency, and you don't have to worry about two runtimes. You don't have to worry, your ops guys don't have to worry about managing two runtimes, but just one. So, you know, it turns out JVM, it turns out managing the JVM is very, is very well understood at this point
People know how to tune it. People know how, people know how all of it works. So, it's very, so all of a sudden it's a lot easier to deploy to production rather than like some crazy Frankenstein, three, three, three level, three levels of complexity stack that you'd normally be dealing with. You know, one way we, so another thing we do is, you know, not everybody wants to run Spark. Maybe you just have a box, you know, 96 gigs of RAM, four GPUs. A lot of folks do that. So, we actually have, we have something called Parallel Wrapper, that allows you to run distributed deep learning, quote unquote distributed, or in this case just multi-core, one neural net per core, and then you, you do, and then you do your averaging. And so, that's actually how we run our Spark integration as well
We just have one, one, one neural net per partition, and you average the results. The only, the only thing I'll say is, there, there, there actually is different, training requirements, different tuning that you have to do, because that, that averaging is actually a form of regularization. So, you, with how you tune it. But at the end, at the end, you'll still get out a neural net that, in practice, works well enough. And so, the workflow for data engineers, though. So, you know, one thing, you know, you know, a lot of people kind of decide that, oh, well, we'll just make the data scientists a hero. We'll just, we'll make it, we'll make it as easy as possible for them, for them to deploy, you know, deploy these deep learning models, or what have you. But, you know, at the end of the day, most, most, most companies have data engineering teams, that want to deploy the infrastructure themselves
You know, they know what a microservice is. They know how to, they know how to deploy and scale your infrastructure. So, the, what's missing is a workflow for those folks. So, and, so in this case, like, typically you have a deep learning, you, you typically have a deep learning framework, and maybe, maybe it comes, maybe it's a gigantic blob of Java code, that you deploy, that, that, that you, that you have to deploy. One of the things we do is, we can run on Android, we can run on, we can, and then we can run on everything embedded. But, you also get the benefits of a higher level system with a GUI, with, that understands your cluster. Right, so, what we do is, we, we decouple a lot of these, concerns. So, your data engineers can actually deploy their microservices in, in the way they understand
As an embeddable model, with one run time, rather than, with, rather than three. And you can do it, you, you can do it all by just exporting the model from, from our, from our stack. And then, one other thing I'll mention is, ND4J, so, we, we, we compartmentalized a lot of the mathematical logic into, I, NumPy for Java. So, NumPy for Java has multiple hardware backends, CPUs and GPUs. You define what one you want to do. And you, you, and then you can use that just straight, you can just use that straight from Java, or from, or from Python if you want. And so, so, so for example, the, so for example, the, so for example, the Flink project picked us for their, for, for their GPU integration. The reason they did that was because it's embeddable
It works everywhere. And it's, and it's easy to understand. And it, and it maps very well directly, it actually maps directly to NumPy. So we compartmentalized all that logic, all that math and all that, into something that your data engineers can use, actually. So they can define their data pipeline, you can get your vector out, and then, and then you can deploy it, and you can deploy it as part of your microservice. And then you can use it however you want. So one of, so what are, you know, our whole thing is, port the Python ecosystem to the JVM, or make it easy to map it to. So another, another way we can deploy these things is via something called Scalnet
So Scalnet, we ported Keras to Scala. And so you can define, you can, you can actually almost map one-to-one Python Keras to, to, to something Scala-based. And then you can run that, it's just a Spark job. And so, so when you want, so when you want to run, like your deep running jobs, you can almost map them one-to-one to, one-to-one to Python, but without the overhead. Which is, which is again, a lot of, a lot of what I want, to stress here is, you only, you know, worry about one run time, your ops guy, and then, and then your ops guys, can sleep at night. All right. So one, so one last thing I want to do is just walk through a simple case study of this. So NASA JPL
picked us over any other framework. They had something TensorFlow-based, but decided to use us because they had a JVM-based run time. They had ETL tools, and a lot of people do, written in Java. Most of their engineers deploy things in Java, right? You know, a lot, you know, because they want control over the infrastructure. This is still very, this is still a very big problem today. Java is still the number one language in the world, and most people have some sort of JVM-based infrastructure. So they want, but they, so they want the Python data scientist to training stuff in Python, but, but then you have this separate engineering team that, that's building your app, building, you know, building your enterprise application. It turns, and, and they have data pipelines written in Java
It turns out that this is, this is actually in practice, a nightmare to integrate when you're trying to do Python, Python, and Java, and all this other crazy stuff. You need to be able, you need to be embeddable. You need to be able to fit into a pipeline. So, Keras, so import, in this case, importing Keras models, allowed their, you know, allowed, allowed NASA to run things in Python, but then, but then, but then you have your JVM-based pipeline, that you can continue, that your Java guys can continue to run. And so it's minimal, one of the things I want to throw out there is it's minimalistic. You don't need a whole runtime, you don't need Spark, you don't need all this other crazy stuff. You can just deploy, you can just deploy things as is. So you can get the benefit, you can get the immediate benefits of the JVM runtime, with none of the overhead, that you're typically dealing with
a, in this case, a monolithic system. So, one of the things we do, one of the things I like to say is, it's, we basically, we actually basically developed a micro-architecture for deep learning. So, so you have a lot of other frameworks are monolithic. They require large systems to deploy, and they, and they have, multiple variables, they have multiple levels of complexity. You don't want to, so you don't want to deal with, you have everything embedded, you have one runtime, and everything just works. That's, so that, that's the reason, that's the reason NASA picked us. Was because it's, it's embeddable, it's easy to use, and it's easy to plug in. So it's very similar to your data science workflow, where, where everything just kind of works together, everything integrates
We're doing something similar for the JVM, and for your data engineers. And with that, I will take questions. Thank you. So I think your model for being able to run Karas apps is great. And I'm just wondering whether that's like completely, does that typically work completely seamlessly? You know, when you're running directly against TensorFlow and you sort of move to your back end, does that, does that, does that end up causing people problems? Right. Yeah, no, that's, so that's the point. Like we, we, so we, we actually handle the model import ourselves. We actually take care of the Anno and TensorFlow back ends and make sure that it's seamless
So what we do support will work. And just a quick question about the import. So that is, I guess, both model and weights. So like, we could do like training on Karas. Yeah, yeah, yeah, it's, so basically one of the, so one of the, one of the great, one of the great things we did was we actually built, I, we're the first framework to do this. We actually built an explicit transfer learning API. So that allows, so that, so, so, cause what, what people typically do with Karas applications is they, they import VGG and then they add one or two, they add, they, they graphed on one or two layers. That's all they do
So all they're doing is they're slightly modifying the net. So what we do is we import the weights and then a lot, and then a lot, and then define a little detail. It actually to say, I want, I just want to add, I just want to change my output layer to three classes. So it was built basically with, it was built with that transfer learning API in mind. All right.