scale.bythebay.io: Todd Mostak, End-to-End Computation on the GPU with a GPU Data Frame

scale.bythebay.io: Todd Mostak, End-to-End Computation on the GPU with a GPU Data Frame

Recording: scale.bythebay.io: Todd Mostak, End-to-End Computation on the GPU with a GPU Data Frame

they got a turn on hold on how's that better okay everyone thanks for coming really excited to see everybody on it Saturday yeah so today I'm gonna tell you a little bit about doing end in analytics on a GPU or a set of GPUs so I don't know how much of you know about how much you know about math D but math d originally stood for massively parallel database coming out of my research at MIT it's an open source project or at least open core so you can go on either github or download our community edition that includes basically a full sequel accelerated GPU database that runs on you know anywhere from one to sixteen GPUs per server it's tremendously performant we like to think it's basically the fastest whole app engine out there meant for obviously meant for analytic workloads and then on top of that GPUs makes sense for a lot of other things so not just fast queries but visualization so you know as it may not be surprising that you can leverage the GPUs for what they originally are designed for which is rendering graphics and we can do that right on top of the data and I'll talk about that and also for machine learning so a map D is not so much the machine learning platform in itself but because we're part of this GPU ecosystem I want to tell you a little bit about this project we've been working on with anaconda h2o and NVIDIA called the GPU data frame which is a way to seamlessly transfer results between different processes running on the GPU so just quickly to set the stage you know CPUs aren't getting faster as fast as they used to so back in the day people used to just crank up the clock speeds on these things and they could easily get maybe 40 to 50 percent gains per year at some point in engineers that Intel and NVIDIA are sorry Intel and AMD basically hit a wall where they realized that cranking up the clock speeds required exponential increases and energy consumption and heat generation and so they started having to get more clever going multi-core adding bigger caches adding better out-of-order execution pipelines and essentially the growth in CPU processing power slowed down a lot and that happened right when the era of quote unquote big data was taking off right so organizations were starting to get massive amounts of data log data sensor data data coming off the cell phones social media and they're basically this gap you know cause a lot of frustration for people people had to start doing things like heavily downsampling pre aggregating scaling out the massive clusters I think until relatively recently it was very cool to say oh I had a thousand node Hadoop cluster people have realized that not only is that really expensive but it's often a major headache to administer so that sets the stage for the entrance of the GPU so I don't know how much you know about GPUs but GPUs obviously can do a lot more than just doing rendering quake or crisis to your screen they've evolved into general-purpose computing Genet leverage the same parallelism that allows them to render graphics performing Li to basically do all kinds of operations so you know started with HPC vir they're now generally programmable for the last ten years HPC scientific simulations evolved into machine learning and particularly deep learning so deep nets and AI are all the rage and most of them are running on GPUs or at least being trained on GPUs and then finally we see the final frontiers general-purpose analytics you can leverage their massive parallelism and memory down with to run things like sequel or just generic user-defined functions very very fast and so just to put it in a nutshell you know the difference between a server running CPUs and a GPU accelerated server is that you have a ton of parallelism at your disposal so CPU server you might have 10 20 some of Intel's chips you might even have 30 or 40 cores on a dual socket system in a TPU server which you can now spin up on any of the major cloud providers as well as get on-prem from any major hardware vendor you can have more than 40-thousand cords now at your disposal I don't want to say the CPU cores and GP cores are the same they're not but this should give you an idea of what the parallelism you can get that's on tap with GPUs so that's great to have this cutting-edge Hardware one of the problems has been that there's been a dearth of software that's out there that can take advantage of that can take advantage of the parallelism of these GPUs right so for select domains yeah there was specially written CUDA code or OpenCL code that could leverage these GPUs for things like protein folding simulation Bitcoin mining well now they've moved to Asics for that but some of the script money and deep learning but if you want to do general-purpose processing like running a sequel engine you were kind of out of luck so that's where math D comes in again coming out of my research at the C cell lab at MIT we've been basically working on this for the last four or five years building it into a very powerful engine that can do analytics equal in real time so the idea is taking multi billion record datasets I'll show 11 billion record data set and being able to slice and dice that instantly run queries no indexing Priora Gatien just using the brute force scan power of these GPUs to basically run queries in milliseconds and because the and because the one of the killer use cases of having this fast engine is being able to explore data right so you know if you're running if you have a classic data warehouse in the use cases to run a report a static report for your CEO showing him the sales numbers from last quarter you probably don't need a super-fast database to do that right but with that speed it really does enable a whole new form of interactive exploration of data and that's why we built at the immerse so mat the immerse is a web-based visual analytics platform that leverages not only our fast sequel so you can cross filter and drill down on anything instantly no pre aggregations or indices required but you can also use the GPS for what they originally designed for which is to render data so we can actually run the sequel query over billions of rows the data stays on the GPU and with zero copy we can actually render that using our OpenGL rendering engine that's based on the Vega rendering specification so more you know before I dive into the more machine learning angle of the presentation I'd just like to give a quick demo since seeing is believing I think I'm gonna start with this data set relatively large so this is ship a is data it's open data from the US Coast Guard basically every ship position from their GPS responder around u.s. coastal waters from 2009 to almost the present here so here you see different things you can see that you can instantly basically cross filter on anything and you can see how fast that is so again we're just scanning this database and every time I click on something all the charts update so it's a great way to kind of contextually explore the data similarly I can brush over ship lengths and we can see how fast that is this is not a feature but if you were to look at the sequel happening behind the scenes this is what we're standing to the backend and we're doing a brute force scan across 30 GPUs 127 milliseconds 172 33:56 33 26 so the speed is what enables this kind of extreme interactivity if you will and so it's fully interactive and the last thing I'd like to point out is this map there's no way you'd actually want to stream 11.6 billion records from your server to your client you'd be sitting here all day we actually leveraged server-side rendering so even though it feels like the data is in the front-end you can hover over anything basically get some kind of contextual pop-up you can drill down and do whatever you want this is actually a server rendered image right on the GPUs and so that allows us to scale past the normal visualization wall so as I drill in here to San Francisco we can actually say I want to see all the let's see you might want to look at military ops interesting but you could look at on your own time you could look at the sea where's tanker yeah you can look on the tanker and you know right here our office is pretty close to here we can actually see these tankers basically circle in the bay as they're sometimes just waiting for the price of oil to go up so it's profitable yes yeah that's a yeah I have a slide on this and there's been some independent benchmarks but I think they did there's been some benchmarks same not this data set that the 1.2 billion record taxi data set that's pretty common a lot of these kind of simple OLAP queries were taking 10 to 80 seconds so a faster system like redshift I think they had a $50 an hour cluster redshift was taking I don't know maybe 3 or 4 seconds but that's for each query right so you still wouldn't have this kind of interactivity and you certainly wouldn't have the rendering plus that's three or four times what the cost of map D would be yeah running on a single server since I'm at Twitter I figured I had to quickly give a Twitter demo so you know basically you can zoom in here this is streaming up to the second these are geo coded tweets you can play with it you can do any hash tag I was trying to think of something that was like PC to search for but I couldn't think of it so I figured we could search for more you can see the big spike about this guy down in Alabama whatever he's doing and what people are saying about him so and you can get very granular here we can again basically zoom in and see what people are saying cool that's for the next demo actually so I'll hold there see a time check okay so where does Maddie fit into your overall ecosystem so even though we are a sequin alit extension we don't generally go to market as a data warehouse replacement right so if you have a vertical or Teradata those systems have evolved they're very mature they don't have our speed but you know we don't we're basically at sequel 92 we're working on window functions some of the more advanced sequel but we're not gonna have all the features so what we normally do is kind of sit as a side car where we're like a hot cash on your store record and that could be pulling data out of a database or Hadoop system that could be pulling data out of a traditional data warehouse like a Terra data via JDBC you can pull in streaming data you saw the Twitter was basically up to the second you pull into Matt D it's gonna be ultra fast because it's GPU accelerated you can't hit third-party BI tools we have a nice db-api compliant Python connector which I'll talk a little bit more about in a minute I mean actually this is the focus of the rest of the presentation about doing kind of Indy and machine learning so hold on that and of course you can leverage a lot of our customers leverage map the immerse even though it's not as feature complete as a tableau it does allow for very agile exploration of very large data sets through the sequel and rendering capabilities and one of the things we're gonna do going forward is basically adding using these GPUs to do kind of you know you brush on some anomaly and we can do a pairwise logistic regression against all the other attributes in the dataset probably and you know a few seconds and so basically it will augment the human and kind of give automated insights into what's going on in the data so you asked a question of speed so this is this guy markula twin chick he's done a number of benchmarks on different systems typically where orders of magnitude faster than the other systems even though the other systems are running much larger Hardware footprints like a 50 note presto cluster 11 nodes spark cluster six node yeah $50 an hour redshift cluster and so this is actually a relatively small data set for us right this is 1.2 billion rows we're running queries in 20 to 50 milliseconds you've been on a single server with game or GPUs but imagine that scales right so as you get to bigger bigger data set the Vantage that we have increases by leveraging these GPUs alright so getting through the meat of the talk one of the exciting things we've been working on with again Nvidia anaconda is shuo John Owens lab out of UC Davis the gunrock folks graph history and a few other players is this GPU data frame and it's part of a larger initiative called go AI GPU open analytics initiative and basically it was designed to solve the problem that all these people are doing cool things on GPUs right we were doing more the sequel and visualization piece you had obviously all the deep learning frameworks like tensor flow piano torch doing neural nets you had people like h2o actually pushing a lot of their non deep learning ml to GPUs like gln and things like this and you even had people like continuum now I'm akanda pushing numba where you could do general-purpose computer on GPUs the issue is that all these different platforms are relatively siloed right I'll just skip ahead to a slide here in that if you had data on the GPU with Matt D and we had a result set you basically had to send it to the CPU and then you have to send it you know if you wanted to do some compute and numba and python on the GPU you have to send it back to the GPU and then if you wanted to do some kind of deep learning you'd also have to send it back to CPU and back to GPU and moreover you probably would have to send it over the network even if it was on a local host the way most of these systems were set up so that was a rather inefficient state of affairs and basically any kind of real-time miss that you could achieve out of all these platforms learning on GPUs was largely destroyed by the overhead of marshaling data and sending it across the PCI bus from GP the cpu effect so just some motivating use cases that we found and that our customers use this for so the first thing you don't even need necessarily this gdf for this is we have a lot of data scientists using us for interactive feature in chair the fact that it's so fast and you can slice and dice so quickly people really like it to kind of discover anomalies in their data and discover kind of visually discover correlations we have a number of hedge funds using us to basically when they get a new data set of say a new credit card data set or something that they want to try to mine to look at for trading insights they'll use both our query engine and immerse to basically look for interesting features in the data also look for anomalies you know the second one is interrogating black box a models so I think you hear a lot about this in the news where people say hey these especially deep learning models are very powerful but we have no clue what they're doing under the hood so we actually have Volkswagen using map D to basically take their AI models and then put the results back into math D and just do systematic comparison of the inputs and outputs both other ribbon out there algorithmically as well as visually so that they can show their bosses and say hey there's no kind of correlation that's gonna get a suit here between say a protected class and whether they're gonna buy a car or something and then finally enriching data so this is something I'm really excited about is that you know a lot of this stuff a lot of the heavy-duty deep learning it's actually kind of hard to put in a databases at UDF and so basically we can have the NL DL as a sidecar alongside map D and that we can actually say hey give me a sentiment on this text or basically predict whether this person has the flu based on their tweets or something like that and we can actually roll that they'll roll that prediction right back into our data tables and then you can use that for further analysis so when we all got together the idea was you know we're all running on the GPU let's have a zero copy framework such that we can pass data seamlessly between these different processes without the overhead of marshalling again our going to the CPU and because we didn't want to reinvent the wheel we actually picked something that was already out there as a base Apache Aero I'm not sure how many of you heard of this but it's kind of an up-and-coming let's see if I can put this right basically it's an up-and-coming framework for both doing kind of IPC between different processes as well as just a common in memory columnar representation it's such that you know these different processes can just seamlessly hand data off to each other and they can all speak the same language in the same format of data so we use that as our basis Wes McKinney what PMC on arrow and at 2 Sigma has been pretty heavily evolved and this Apache we basically added IPC support for it so that on the GPU you know if I want to give you my data if NAT D wants to give a show its data it can basically write through this arrow format and then just use kudos native IPC capabilities to hand that data off to the next process and it's completely zero copy so not only do you overhead the void overhead of copying the CPU but are you actually avoid overhead of copying on the GPU itself and you can actually use this even inside different doctrine since it's three minutes okay well okay and it's great so you can do you know you can take data out of math D pass it through the data frame pass it to something like anaconda or these other processes I'm going to see if I can give a really quick rough demo so even though you can do this programmatically through our Python connector try to do it visually here so this is a very much work in progress and we were kind of working on it last night to get this going so I can make no guarantees if this will work well but hopefully well so this is flight delay data I'm just gonna give you a quick quick example so imagine I wanted to look at JetBlue here I have average departure delay by origin state I'm gonna click on California and now I want to see what the relation between are I already picked this arrival delay and departure delay and so here I would do train and then it finished I got an r-square of point eight four which is pretty good but you might expect that there would be a good correlation maybe I want to say I actually think air time is a good feature as well so I could use those same filters and click train and it's about the same so did it have a yeah but obviously this is kind of the workflow that a data scientist might go through that they could quickly apply filters and kind of clean their data you know you could just clean the data through adding filters and then see how they can improve their models before they actually put their models into production so interactive feature engineer great so probably at one minute I want to make sure I have time for questions just a quick thing is that arrow or the GBF is not good just for outputting data so we actually put it in our import path as well so when you push data into map D if say you have a panda's data frame in Python going over thrift which is our kind of default IO mechanism you know row-wise 42 seconds we went to call min or five seconds by using arrow and doing IBC we got it to 42 Millis out of 420 milliseconds excuse me for about three million rows but you can see the dramatic improvement of using IPC so I won't touch on the use cases we go across the number of verticals telcos financial services what we do with the hedge funds I'll just say that if you're excited about this and want to get involved we are open source get us in github download our community Edition and Lenox CentOS or Ubuntu works out of the box some people have gotten it working on a Mac but it's not officially supported you can spin us up on AWS cloud we have a community dish in there obviously an Enterprise Edition and yeah we're a growing company so if you're interested in working kind of cutting edge problems on GPUs we'd love to chat with you so thanks so much have you take questions Oh it's a great question so one of the things I didn't get to hear is our memory caching technology so usually we're running on more than one GPU first off so a lot of our customers run like Verizon I think has four nodes with eight p-40s each which is you know and they're moving probably to a terabyte of GPU Ram we're a calling store so we actually compress the data heavily and the nice thing is that we have this three-tier caching mechanism where not everything has to go in the GPU so everything by default unless you define it as a memory table will be stored on disk as you query we basically percolate the data up the through CPU ramp the GPU realm it's kind of like a pyramid here and so the hot data the data that's actively queried and we define that as things that you're doing active calculations on will sit in GPU memory or will be streamed through GPU memory ideally that can be cached across the multiple GPUs and we find that you know I think Verizon on two nodes had us running on a twenty billion record log data set so pretty pretty sizable with maybe 20 hot Collins and you can obviously scale out as you need to but what isn't actively queried or even just columns using for projection can sit and CPU around CP Rimes typically much bigger so it gives us a bigger footprint and we can actually execute and CPU if necessary if it doesn't all fit on GPU will use our LLVM compilation engine to actually run on on CPU we can target CPUs were quite fast they're obviously not as fast as you use so it's pretty flexible system right happy to chat with anyone after thank you [Applause]