ai.bythebay.io: Arno Candel, Deep Learning is Like Water (it's Everywhere!)
Recording: ai.bythebay.io: Arno Candel, Deep Learning is Like Water (it's Everywhere!)
you [Music] so this is a team effort by the h2o team nicely aligned as a TI realized today we're about 65 or so people growing nicely and the company is a start-up in Mountain View and it's a software product company so in our services but we're building actual product and the product is pi/4 enterprises hey I for business and but it's also usable for everybody else like my mom anybody can use it it's very easy to use as long as you have Java on your hardware on your on your bare metal laptop or cluster or whatever you have an arm device or a raspberry pi you can run it and it's open source you can download it right now it has a bunch of algorithms that operate on these distributed data frames that are in memory and compressed and all so you can make operations throw slicing columnar transformations joining all kinds of stuff and all of it is done on the back end in a distributed way and then your client experience is done because on your laptop you're just typing some strings your laptop is always nice there's no cooler blowing or anything on your laptop doesn't get hot doesn't use battery there the cluster does all the work and in the end when you build a model which is that a core part of h2o you actually deploy those models with a single button click of a button write the code to deploy auto-generated in Java or C++ it's a package if you get that you can put your models into production you don't just build a model but they also make this package and as you can see h2o is somewhere on this map who can spot the yellow spot is what's a CB insights which is one of many players in that space but we really go after the big companies a fifth or so of the fortune 500 are using us and mostly sintex insurance health care and that shows up in the gardener magic quadrant where we made our debut this year as a leader actually is not this a visionary almost a leader I always mistake this one the line here where we have to cross this is a little above us that makes us a leader once we get across that so the execution has to go up and be the completeness of vision has to go up as well and we are working on that very hard on both fronts so the division is like automatic machine learning smarter stuff stacking ensemble in deep learning AI reinforcement learning all this smart stuff that brings us to the right side and getting even more customers will get us top this was TechCrunch San Francisco shout out to us and of course you see some logos their cafe and makes men tend to flow i'll talk about that in a minute fortune spoke about us just recently i think forbes as well there's a book written by the community that came out in december and it's actually a very good book so if you are interested in it you might even be able to get your discount so ask us or just go buy it it's like thirty dollars and so but if it's a really good thick book that explains everything about age to go from start to end for r and python and all that customers love us because we love them so we always care about the customers first since the company was founded in 2011 customers were instrumental feedback was critical so adp progressive capital one market chair kaiser zurich insurance and comcast and then there's hospital Corporation of America McKesson Macy's transamerica paypal ebay cisco and nielsen catalina for example nielsen matches up what you're buying with your credit card and what you're watching on TV and see if there's any connections between the two things Cisco's building 50,000 models in like a model factory to figure out how much sell each item for each country in each store and so on there's a lot of use cases from a fraud detection to sepsis defection the HTA you don't want to get sick when you go into hospital it's still the leading cause of death at a hospital is actually sepsis getting poisoned by something else that wasn't your illness and we're working hard to make everybody's lives better so the h2o engine itself is basically able to read data from anywhere it's agnostic to the data source it just has a bunch of connecters mostly just flat files from where Hadoop for example arrived three it reads the files in in parallel from all the different worker nodes as we call them the cluster the h-2a cluster consists of worker nodes or cloud nodes or whatever to call it but they all suck in the data and parallel every threat and every one of those machines is reading data so we're really fast in parsing probably the fastest person overall and then once it's in in h2o it's actually compressed each column is compressed by the types as you mentioned Alex the earlier we don't store doubles if this number is a 1.3 we just thought a 13 and they say / 10 later if all the other numbers are also in that range right you can do smart stuff run-length encoding for sparse stuff floating point tricks so we are only storing one bite or even bit sometimes if it's a binary column then once the data is in memory its distributed across the cluster every row and every column entry is only stood ones on the whole cluster and when you run your algorithm you send the algorithmic code there to all the other machines so not you of course will need it all that so you press the button and it runs on all the notes basically the same logic applies to different data just like a sindhi instruction and regular chip you run in parallel it was a reduction in the end you get the overall result and this paradigm is very powerful it's what's underneath spark and Hadoop but we wrote our own real fast engine that's more like seed and Java but it was actually implemented in Java but we're using all the tricks to make it fast like tungsten for spark if you want just a few years earlier than them and then the algorithms that can do whatever they want you know deep learning gradient boosting random forest logistic regression clustering PC a you name it matrix factorization in the end you deploy it as a standalone package in Java or in C++ as I mentioned earlier and of course you can embed it in any streaming engine or whatever you like the AP is that you can talk to this back-end workhorse are our Python and flow and also Scala for the sparkling water integration and you can see that autumn Python of course that's what data scientists use but you every time you type something it really just makes a string and sends it to the server that's it somewhere else usually it's only rare that you have both on your laptop that's when you're a calculator for example you can't afford a cluster so this is my life's demo for a distributed deep learning use case here let's go over here let me make this as thick as I can this is a nice resolution here so let's connect to the Tendo cluster which isn't up yet so I need to start it so I'm going to say here see this for loop you probably can't read it but all I'm doing is I'm saying loop over my host names and start a cluster and now it's launching 10 Java dejar commands here with h2o jar which is my way of launching this cluster now we can connect to it because every node in this cluster is actually also a web server so I'm just asking the first one to give me a website response and one of these 10 Java process that are running right now says okay i'll be your web server and here you can see the depend outs these are the the utilizations of all the cores on all the 10 machines so there's 320 course overall and green means it's busy someone else is probably developing something right now this is in our mountain the office and just like hijacking that for this demo but it's okay we don't really care about the speed or anything now we are going here and I'm going to say this is flow this is a nice interface by the way you have these models you have data connectors you have some stuff to score make partial dependence plots you can look at the loss and profile it you can have all the stuff but what I wanted to show you is there's a pre-defined flow as we call it it's like a little workbook thing and i'm going to say here run this again because i did that earlier now it's parsing a hundred million rows from a file system that all 10 machines have access to the real reading data right now and parsing the comma separated files values and it's already done took eight seconds you can look at the frame that frame has 116 million rows it was 12 gigabytes on file is compressed down to 2 gigabytes or so and it's basically because it has all these compression schemes in there see it's like one bite beats two bites there's nothing bigger than a 2-byte integer because there are also small numbers like significant digits that you need to represent them is not that many in the distribution across the 10 worker notice like this each one has about 11 or so million rows so now that we have this frame we can split it so either like click here on split and I type in the two numbers and they say go or I just reuse the existing cell that was created earlier when I did it that has the names 100 million and 10 million as the output frames so right now we just did a distributed split operation is traded to new frames and now I'm going to build a deep learning model on that so that's the easiest way is to just leave on the script and now it's running deep learning and you can see here it starts pumping all the 10 notes have a little model that they start with i'm doing a for hidden layer noodle night so a regular fully connected neural net with for hidden layers and these are all randomly initialized in the beginning these weights right and each of the 10 notes gets these weights and then they all work over their local data and as they go through their data they're updating those weights and once after so on so many iterations of training samples that FC and they coordinate and they make one model that they all average their own model to so they basically reduce their model into one model and then they broadcast exact for everybody so that they all have the same model and then they continue and this kind of stringing together those tender workers helps with the overall efficiency to be high because they don't diverge they all stay together but they still see different data so there's some variety so you can see that this is working together and one of the nice things of h2o is you don't need to wait until this is done you can just look at it here you can click on view and as it looked as its training you can say they fresh and you just look at this for a while and we will see that it keeps updating as its training and you can look at the feature importances you can look at the aoc curve on the validation set I did a strange split earlier so that I have a holdout error and this is all nice right you can look at this real-time you can even profile it if you're a developer so this is the power of this back and this Java back and you can see what's going on and you'll get the full transparency and of course you could have built a different model like gradient boosting models on the same data set here training hundred test pin response is better i'm the late or not for example you can do all kinds of fun stuff that this airline data set it's kind of our basic demo workhorse and then you can say go and now it will build another model on gradient boosting using gradient boosting i can of course go out here and kill the other model i can said I want this to be refreshing and I don't want this to be trading anymore here cancel if you don't want to go up and down so fast you can just go here it's faster this is the outline and now we can watch this guy converge right the same thing there's also going to be a convergent plot the gradient boosting machine is also going to use all the notes both are busy training and you'll see this this was the first tree with when there was zero trees actually it says this is like the random guess if I had to guess everyone to be the prior wouldn't know anything about it let's wait until the first tree comes in so it's scoring it right now it's scoring on the full data set because I didn't do anything special so it's pretty expensive to score on all ten million or 20 million rows and also on the training data sets of scoring a hundred million rows right now in memory you see here and already jumped up to 68 something similar that we saw earlier on the deep learning model so just to give an idea this is basically each devote a classic CPU cluster age to hville all right now we can just kill this cluster boom oh and my body's can play again all right now XG boost who has heard of actually boost here okay so extra boost is another implementation of gradient boosting using C++ as a back-end and because it's so modular and not embedded in this big h2o system it can also do other stuff really well it can work on sparse data easily it because it just takes a matrix it converts with GPUs is it has other like things in it like it can go deeper instead of doing a whole depths at a time you can just go down in one corner of the tree there's some smart algorithms in there and so we chose to integrate extra boost into h2o so that you have the power of actually boost and the power of h2o combined so the benefits of h2o of course as slow as you just saw and it's the real-time feedback it's the ability to manage data in parallel and produce these for example input matrices for extra boost in a parallel fashion so it's much faster now to create a D matrix that you need to build any when you wanna call extra boost this pre-processing can be done by the Java engine and of course the production izing of these models is the same way as all the other h2o models it just works so now we should show you a demo of this because we said we were going to do live demos so this is this machine here i'm using a GPU box even to show you some cool real-time GPU usage of this of this model let me see how i can move this back here this is work ok i want to have this window here where I anybody knows this nvidia SMI club to see the the GPA usage so this is my box at the office in mountain view it has two GPUs and right now it's not using that much because nobody is sitting there doing anything but that will change soon so we are going to load a data set I'll do it from scratch so you can see everything I'm going to find a caracal data set and it's called pareve it's a data set that was a Kaggle i'm going to import it i'm going to parse it it's going to get some column types i'm going to say the target is actually classification so i call it an enum the idea i'm going to you know what I'm turn it into the string I don't need to look at the ID everything else looks good so how many columns something like a hundred thirty or so you can go through and look at all of them see that there's some in arms you can c d and so on o us all right let's parse it and this is none of my on my server there it's not a big data set if you look at it it's only like a hundred my exes or 100,000 rows hundred columns perfect for extra boost its really fast on let's say medium sized data sometimes it's also good for really big data but it depends on the structure of the data h2o is really meant for large datasets scales really well but for something really small x equals can actually be a lot faster especially if it's sparse ok now we're going to build it'll split here again let's call this train and valid 7525 and let's build a h2o GBM model first as a baseline let's call it h 20 GPM so we remember it later and they use the validation set just to score we're not going to do early solving or anything they're just going to run it for 50 trees steps 5 just a default so this is not so hard to use as you know what a gradient boosting machine is and I'm saying this is the response and don't need the ID in there because that would be over fitting and all the rest is just defaults and this will run quickly and will not use the GPU here though but if you do a pop you can see here is using the CP already gone okay so other words again it only took 12 seconds these are the cpus I'm not doing the power perf bar thing I'm just using top here ok so it's each door uses all the course and now we're doing the same thing but first let's look at the model so the model has 79 a you see on training and the 74 six on validation ok and these are the feature importances and if you wanted to deploy it you can actually just click here on the download model deployment package or on pojo there are two ways to deploying it is click on the pojo oh it just downloaded it there's also preview actually the previous even nicer because they don't even need to leave the browser so this is the scoring code that's auto-generated you can see all this java stuff this is all the work of all the trees and each of the trees are also explicitly unrolled in this code all this work you don't need to do anymore although the categorical levels and so on a lot of stuff they even stopped before the trees came this is a lot of like headers is like what are the categorical levels and so on but let's go back and build another model this time an extra boost model because I told you that that's now working as well so target I ignore the ID everything else is default but you can see that there's actually boost etta and so on these are actually boost conventions if you use the extra boost you can just use the same names for the parameters you don't need to learn h2o conventions for example call sample by three and so on there's also the the lost guys hist method which is like GV m there's the approximation exact all kinds of stuff like really cool this is the full integration of extra boost and of course the back end can be GPUs now as well and if I say go this should now use some cpu but also nvidia GPU see this here this number at the top is now no longer 0 it's 95 if you can see as I'm sorry I can make it a little bigger but I'm sure you believe me that this actually was using the GPU and now let's look at the model this wasn't scored in the middle we scored every every so often in the first four seconds in and after that the only score ten percent of the over all times and I'm going to spend all our time scoring that's why it's like disconnected but he can set it to be scoring interval of like five trees then you get it every five trees they don't have to have this jagged line there it's just the default this was so fast that it just basically didn't need to score so also 7574 and a half validation AFC so now you have two models and you can look at the variable importances and you can deploy the same thing right this also has a deployment package here when you click on the download deployment package its downloaded and if you look at this file what is it it's a bunch of bytes some domains the categorical levels some experimental details and then here in any file that explains like what's the model right it sexy boost its license Apache liat license too sparse it was enabled blah blah blah this is all the settings so this is very easy very transparent you can run this with ease so now you're saying I don't like floral and Judaism Python I'm a programmer I'm not just a clicker all right I get that let's do that let's go back to the Jupiter interface and I loaded a simple little thingy it says connects to my HTML cluster load the I ish to a package first and then connect to it I launched it 40 minutes ago before my talk it's using 40 course it's our lives and now I can just get these two models that are built earlier and I can just print it I think maybe I didn't name the the extra boost model extra boost is that right I probably called it something else yes I should have called it extra boost but I was lazy so that's the wrong one I should actually go to this one you have to know the name of the models and you get them from Python otherwise it's not going to work so I can go here and can either the list models and I will see this is the one ok now I can put that in here and say it gives me that guy alright and you see the two numbers that we saw earlier the validation a you see some asking the model for its aoc on validation by default it gives you the training error which you don't care about as much and now I'm saying fetch minute frame of course the last that doesn't do anything it just has a handle to it and it has the server to do all this right nothing has left the server yet my laptop here is not doing anything even if this wasn't hosted in Jupiter on my server it still would be not doing anything in Python it's just a pointer to the state in Java on the distributed system now I'm saying predicted both models GBM and extra boost and take half and half and take the average that's my new prediction it's just a little better hopefully see if it's true I don't know 74 9 yes nice let's hurry to blending this is simply an awesome method of purest kind to just take both and you don't even need to guess how much you just take them half so this is an easy way to put two models in production gives you a better model than any one of them of course tuning those would be even more valuable than just blending them but an expert will do both alright that was that so let's go back to the presentation so you saw that this cackle challenge cannot be done with the two methods in the h2o framework from Python or flow using gpus and the whole h2o infrastructure so now let's move on to deep water that's over here for I know you as a hungry but for now all you get is water stay hungry they say right it's a good thing all right so h2o deep water is an integration of MX net tend to flow and cafe into h2o so now we have three more GPU thing is running inside of h2o you know that Nvidia 2d and n leavitt is really fast Intel nkl is also fast and if you have a xeon phi or so it can be very competitive in terms of performance so don't get me wrong i'm not saying intel is slow i'm saying if you take a laptop or your desktop at home and you just to keep install or something it might not be as fast right so you have to first stick it out to get em que el installed and all that to make intel faster but it still do not be as fast as a nice $400 GPU let's see that's really the strength nowadays of GPUs is to do deep learning that's right the nvidia stock is going up so now the use cases that are possible with this method called deep learning on GPUs is that you don't have just these four hidden layer simply connected fully connected neural Nets you can do complicated convolutional little nests and so on and these use cases of course go into unstructured data and all and just to give you an idea how these graphs look like the passive of the left side they just have fully connected layer layer layer that's it they all talk to each other with dense matrices now you have these convolutional operators and pooling and average and all that stuff so there's real tensor flows that the name tends to flow but we don't do just tenses what we also do MX late in cafe they each have their advantages and people might be used to like a cafe modeling that they have our ID and they can just provide that graph protobuf file and say run this in the h2o engine and it will then call cafe with that photograph violence that you do it because age doors not really good at convolution little next at h2o is written in Java is not meant to do descent linear algebra on the GPU so it's calling this c++ and back in but it's not calling it from python is calling it from java so it's really fast it's from java multi-threaded data preprocessing handing it down a c++ process that has to GPU and of course the whole thing is also deployables as a mojo as we call it the model object or something java object and for example that you can do is you can take a state of the art in which classifier is the dress that inception that came out of rest net and inception investment was microsoft inception was google and put them together they got the best of best basically and this architecture is something like few hundred i don't know few hundreds i'm sure at least 300 or so layer steep but it's always repeated so the yellow and the blue and the pink parts are just repeated the same building blocks over and over again so it actually pretty boring but once you know the structure of course everything is boring the graph itself i can show you let's go think you might have in life somewhere let's see there's a bunch of example scripts and they're actually in github you can get them right now if you want that in h 2 o 3 example sleep learning notebooks and there's a lot of them and this is the one that I'm showing this resonant we inception v2 is actually a graph that's defined by these several pages of Python code not trivial to come up with this but luckily there's open source contributors and I just copy pasted it from the internet this is the MX net implementation of that graph i literally just took it and brass a safe ass around it and dump this to a file loaded it back into h2o and set go we do this live demo as well I think this one here and again you should see this number here go up soon so let's connect to it okay 47 minutes that's my time this is the graph we don't need to look at it let's define the graph okay so this is Python to pit the notebook defining the method that says this is how you make a graph now let's read the frame that I want to use for training and this frame is simple it just has a column full of paths for JPEGs and another column labels so this says cats out mouse this is the jtech that's the easiest way to do it there's no binary data or anything in h2o but once you give this frame to h2o which is a valid CSV right then it'll say oh I see this is the JPEG so I'm going to get the image reader for that and I'm going to reach those JPEGs all in and parallel scale them all making the right size for this graph that you gave me and then I'm going to hand these float erase that are well defined with three color channels and so on down to the GPU back end and we do all this for you in our rapper so you don't need to worry about anything you just say this is my frame cat dog mouse and if you want to see cut off mouse you just ask pandas or something some image library here image in ipython to display some random five images right and so this is a cat actually almost two cats with the mirror dark and then maybe hopefully we'll find a mouse to okay perfect kind of an ugly mouth but I'll entail so this is the graph this graph when it gets me this is life like display of the graph just like 10 support that's the MX net way of plotting it but you can imagine it is a long thing here full of like connections so that's that's deep learning for you in a nutshell the whole thing is complicated but once you have it and this is the method update i defined and it says save myself to this file and then i just passed this file here as a network definition file I'm also helping out with the size and the channel so i know this graph once 299 by 299 pixels and I wants three colors RGB channels so once that's done I can just say go and now it's training but you're normally used to waiting here until it's done but I can just go to flow which one is it this one and I can say get models and they will be one more and this guy has now a refresh option that I can just look at it and this will now show up something after a very little delay i guess yes your first points already shown up and this is the life model training right now you see the number on the right here the ninety percent so the GPU is busy it's running the inception the resnet v2 graph on the GPU and it's training with your csv that has images and labels you can also look at the log files for example if you want to see the logs oh here it went up first it sometimes jacks around with what it finds a local minimum so let's watch this evolve so here you see GPU staff native back in Java blah blah blah cat dog mouse gets the confusion matrix you see what's going on m and you see that it's training at about 37 samples per second the same thing is probably also going to be shown here yeah oh yeah see now it got better over a while I'm pretty aggressive here i'm doing a learning rate of 0.05 which is the default which is like a little bit high for this image model stuff so it's like jumping it out in space until it gets lucky at some point and it's okay I'm stuckey this is using stochastic gradient descent with momentum so it's not the best method we're actually adding rms prop I don't prop all these smart optimizes it's a tentacle integration so that you don't have to worry about specifying the learning rate and all that but you can see it's converging is the confusion matrix only thirty-seven percent error instead of 66 if you were guessing blindly and those classes were balanced which they roughly are so it's already better than random is slower training data of course it will over fit on a small data sample data set because you can see it is only 267 images and obviously the model just memorized them all based on some color of some background or something it's not actually learning the cat dog mouth but I'd you need a bigger data set you need to train for a few days or whatever it's like this is not saying you can memorize cats and dogs now it can memorize these cats and dogs now okay so this is still a data science problem in the end it's not going to be solved by the naive user clicking on it but I want you to know that I didn't have to take a two hundred image data set I could have taken a much bigger data set for example that's cars that were totaled and cars that were not totaled and I can train for like 10 hours on this system and enrage right and then see how it converges and at some point it will actually learn to generalize and not just oversee especially since this graph is able to do that this graph was meant for for a thousand different classes to distinguish things so this is a good model and for this good model the 200 images are way too easy like oh yeah I got that no problem so see here the errors down to seven percent oh so that's it this is the border actually we do have a minute we could probably show you the we have to cough Avon as well if you want to see both gpus getting light lit up I have yet another notebook that I saved on the million song data set I think this one is MX nice and the next one here is cafe so if I say go you should see both of these light up with a hundred percent very soon so we have also multi-gpu training yep here now there is one hundred percent so there's cool stuff going on in the deep learning space and we are part of it and we are making your life easier fully and there's a lot of things you could do especially if you know your data right if you have a lot of data that's really the value algorithms are more a commodity but knowing how to use them and knowing how to have a framework to do all this stuff that still it's still not solve there's not just an easy button unless you go to the cloud and use maybe Google Cloud computer something but then you have to go to the cloud and you're restricted to what they offer you so it's not quite salty at this whole problem and we're hoping to help you take your problems into production with whatever it takes AI is not just the blurring its knowing what to do in data science right in the old days it was a statistics expert and now it's just a bunch of calculus clicking buttons but maybe that's dangerous so you need to be careful about those so for example the CEO will say we need to do this AI transformation right then the management will say oh who does it I don't have anyone give me somebody and then you can see this you're right obviously the senior guys will say oh me and I've been doing Python and are for 10 years ago or 20 years I've never done these learning I really should start and all the kids are doing it already and the rest of the world has no clue who they say what the hell is this this AI stuff isn't here to take my job of a right and it is actually happening like the driver truck drivers the number one job pretty much in every state in the US and driving is not that difficult for a robot as you know so our professors tell us start with interpretability so this is the the trilogy hear the voice a co-teacher ami the center professors and when they meet us in our offices every so often they they literally say like what's important is not to just run stuff through their GPS but to understand what you're doing of course if you do both even better so which platform do you want to use for AI which programming language which hardware is even a quantum computer in the bottom right you thought I'd write it is a quantum computer doing deep learning because it optimizes some system in physics right and it can also optimize the state of a neural net but they also want to use neural nets to optimize the quantum computer so they both go hand in hand and this is a cool exciting space then there's analog chips that may make the brain fpga stuff that gets made custom for deep learning but the rest of the world is still using either intel or nvidia chips but then the ioc space that has embedded devices at the edge they don't want to use a big easy on on that little handheld device right so there's arm as well so they're all playing and everybody wants peace learning of course then you wonder on premise on on the cloud where should you supply the stuff who is doing it is it my people it's your people you need to grow that knowledge or not it's a lot of problems so I like crowdsourcing I like a go because you learn a lot you force yourself to learn a lot unless you don't like losing or whatever if you got it wrong but if you like losing then you shouldn't go to kegel if you like winning then it's a really good exercise because you will not win and you will try harder and that's why Carol is good and you don't want to trust your genius unless you really know that the genius it's better to ask a hundred people and see if one of them gets lucky right and you can do part time crowdsourcing of data scientists and all that kind of stuff so the data science service will modernize as well they are startups doing nothing but this cloud service for data science but of course an internal bake-off is still a good ideas you want to keep your people a happy and keep them getting smarter so that's what we usually do and in the end you still need to wonder what are you doing right is it is it really necessary to throw all these images into a GPU or should you just get a new feature that says how many times you had a transaction the last week that's your turn predictor you don't need to look at their face as the piping on their phone you just need to ask your common sense questions and once you have a model what you do with it just press the button say predict or do you actually look at the statistics so you understand the model and so we're working hard on the feature of importance but also the interpretability of a per observation scoring right then you're saying why is this person rejected you don't want to just say well because the neural net had a million multiplications that left to it you want to say well because of this is this so this is basically it I think in the future you will see more of these things going back to the drawing board but you also see more of these things where anybody tries to automate everything right and Charlie Chaplin did it under the years or so almost ago and you haven't seen modern times now it's a good time for it because it never will be as modern as today so thank you very much [Music] I'm just wondering is the layer-by-layer visualizations through the back propagation or other techniques available in deep water or is there any intention adding that if not you're asking about the visualization of the neural net asset training yeah or after training just as a back propagation of each year of the network what that looks like wood it picks up so the h2o deep learning the CPU based fully connected neural net you can extract all the ways of every connection between two hidden layers you can get them as matrices you can do whatever you want but we don't have a built-in like graph that shows you these hundred million or whatever or ten thousands or whatever this is it's always a big number no matter how you put it right so you need to know what you're doing is it but we also will integrate the tents aboard visualization somehow but it's not quite clear how but in any event all our models that either read or train will be saved as native sense of flow or cafe models so you're not losing anything by going to h2o you just get a float above file back that you can take into a your cafe system right once you're done training in flow let's say so you're not you're not making an our custom proprietary java thingy so it's all still these three backends so we're trying to stay as open as possible so any visualization is they have you can still use let's enjoy your lunch [Applause] [Music]