Scale By The Bay: Sky Thomas, Street fighting techniques for multi tenant machine learning...
you okay so it sounds like a lot of people I've been to a lot of sessions here everybody you know gets the kubernetes 101 I mean I'm not going to do that today I'm gonna try to there there are a lot of things about Big Data workloads that are very different than the traditional workloads that are running on kubernetes I'm gonna try to I'm gonna do a demo but at the same time I'm going to try to explain a lot of the pain point you'll run into and some of the techniques for working around those I'm sky or Skylar I'm really excited to talk to you today things are changing very very rapidly in how easy it is to push these environments out into the real world today I'm gonna talk at the beginning a little bit about some of the problems I had a couple of years ago why they are different now the different weapons that you can use in kubernetes to help you win your street fight techniques you can use to do data science at scale I'll demoing building a data science environment throughout this presentation I'm just really trying to get you to understand the basic pattern of these environments because there there are very basic patterns no matter whether we're doing PI torch or tensorflow or spark or some hadoo there are certain ways you want to build these environments and there are certain things you always want to do and there are certain ways you want to treat kubernetes so I wanted to start off and talk a little bit this is really personal to me a few years ago I lost my little sister to heart disease and medical mistakes and I decided to build a startup and I wanted to take fitness tracker data combine that with deep learning and see if there was any correlation between your heart rate on your basis watch or your Apple watch today or your Fitbit how much you're walking and whether that that could tell us anything about your body in your health could we could we create a check engine light that says hey you should probably go see your doctor started to think about it like with all of the advances and deep learning that were going on at the time it looked like the perfect time to do something like that and so I started trying to build environments to experiment and play around with things and this was around 2013 Hadoop was still a really huge deal docker was just appearing on the scene SPARC was young moving fast tensorflow was totally brand new I think I started doing a lot of this right after the tensorflow announcement Hinton's image net results were out there and lots of recent papers and it seemed like the perfect time to experiment but it was a nightmare I banged my head into wall after wall after wall after wall and I I think that's not going to be the case today but the the big issues that I ran into at the time were no great notebooks for data exploration no book frameworks at the time were kind of finicky lots of Python version woes each frame work was complex to install get running MapReduce mahute SPARC tensorflow torch ability the experiment was limited and deeply discouraging I was forced to own both complex environmental engineering and data science that's something you definitely won't don't want to do you want to play on one side or the other it's hard to do both 2015 and 17 got a lot better exploration on a single machine was pretty good notebook technology's got better tensorflow got much better creating training jobs under tensorflow simpler spark grew to 2.0 lots of new features still limited by yarn at the time many components became containerized but orchestration was still a challenge today we're at a point where containers are ubiquitous kubernetes is insanely powerful at orchestrating things SPARC has native support for kubernetes coop flow is beginning to make deployment of ml environments and kubernetes simple and in summary things are now orders of magnitude nicer for an IT department wanting to support a company's data scientist so how do we do this so there are three main steps you got to procure an environment in many organizations you'll have one huge Boober environment with tons and tons of machines it'll be on premises could be in cloud doesn't really matter where their environment is anymore if you if you're running kubernetes the challenges on AWS versus GCE versus an open shift cluster that you built on premises although all of these things are very very similar all we need is really a set of machines with kubernetes installed in the cloud it can take minutes to procure on premises that could take you months to get the prm's approved but I'm going to i've already procured an environment I'm not trusting the demo gods today to create 50 cloud servers if we look let's see okay if we look real quick in the development environment I did run in it I ran it this morning it took about four minutes and 14 seconds to procure about 50 servers look out at Google right now it's this GC cluster I've got about 800 virtual CPUs I've got five about 5,200 gigabytes of RAM decent-sized cluster these clusters bill Mascara Nettie's past a couple of thousand nodes but building a thousand nodes takes about the same amount of time as 50 notes it just depends on the cloud vendor in his environment nothing really special here about machine learning going back to the slides so I already did that but I want to first create some storage now I work for a storage company I'm not gonna I'll talk a little bit about why storage is so important to these environments but I'm not gonna focus on that but I am installing our map our storage si si and storage environment inside of kubernetes so if I go to here I've already procured the nodes right here I'm just going to tell it to go ahead Oh before I do that at all I'll show you the 50 note environment let's see here let's go to pods actually this little tool is a little bit shy when I don't use cube CTL in a while so I'll just give CTL get pods so hopefully I'll connect to my kubernetes environment in a second okay so this should be a little bit so right now you can see I just have three namespaces the default cube system and qu public if I go out and go ahead and tell it to install the map our components in the environment all my storage is getting installed right now and if we look I got two new namespaces a namespace for the CSI components and a place to put my operators so I'm gonna be installing a lot of this stuff via custom operators today anybody familiar with building operators or usually will talk about operators as we go along in the presentation today but these are operators that I send it a bit of ya Milland it builds out a really complex storage environment storage cluster so I have the operators installed let's see o cube CTL apply - f so this is my custom resource for my map our cluster that's going to spawn off all the the storage so if I go and look now in the environment it started creating this full example cluster and an LDAP namespace if I go into full example it's creating hundreds and hundreds of pods now we're creating storage on every node in the cluster there daemon sets their stateful sets or zookeeper there's there's lots of different things but I'm not going to talk about that today the final thing I want to do before I continue with talking about things is I'm going to install coop flow how many people have used cube flow in Coober Danny if you if you have it it's it's a really cool set of technology so it bundles up a lot of machine learning technologies like tensorflow like pi torch it creates you eyes etc we'll talk about them in the presentation as we go along but coop flow is actually where the real machine learning stuff happens I'm going to install the coop flow components in its own coop flow namespace now when I create jobs I'm going to be creating them in in user namespaces but let's go ahead and install coop flow so it's that's going to start up and I'm going to let that run all of this takes a couple of minutes to install not not a long time at all but I'm going to go back to the presentation for a minute and so we created storage we did that we created the data science environments we created Jupiter hub tensorflow Argos elven core we'll talk about those as we go along so kubernetes is up it's got our components in it kubernetes is really great we can run warehouses full of workloads and everything's wonderful kubernetes has this notion of pets versus cattle as a deployment model before kubernetes the idea was that we had these really important servers that we baby he'd we kept the os's and tip-top shape we got our vm's working well and kubernetes kind of turned that on the head and said let's not care about these they're just a bunch of boxes let's number them and brand them if something dies we'll just get rid of it and throw it away and that works really well for certain types of apps the problem is state the traditional problem with these types of environments is I need to store data and in the kubernetes world kubernetes by default if you're not using a stateful set or a daemon set kubernetes doesn't care where your pod runs it's gonna it's gonna stop that pod with it somewhere else and continue on but data science workloads don't work that way they're they're the worst consumers of stateless data that possibly could exist they need massive amounts of training data you need to create models you have shared notebooks all of these things are very hard in the traditional kubernetes environment but we shouldn't give a give up kubernetes has rapidly gotten support for stateful applications built in there are still some things that would be really helpful if they'd get out of beta but the main components that kubernetes now has to help us handle this excuse me smoke staple sets daemon sets persistent volumes persistent volume claims the new CSI driver for storage and all of that has made kubernetes a much nicer environment so the rest of this presentation is really about what are the problems and what do we get out of kubernetes and how do we use it to its biggest advantage so kubernetes is all about pods pods are the unit of management in kubernetes pods get spread around on all of these nodes what what how do pods help us in building real-world data science environments well it's critical to understand the nature of work for data scientists they're called scientists for a reason they do novel things one of the problems that people traditionally have with data science environments is that data scientists want to put whatever they want on the machines they do not want some cookie cutter website that that corporate approves they want to use the tool that they're most familiar with they're most comfortable with or is the most appropriate tool for the job and that's traditionally very hard for IT to deal with pods and their levels of isolation help us now pods aren't perfect pods are not virtual machines pods if users can put whatever they want in a container and they're running along with the accounting software at the same time that can be a problem better pod isolation the G Weiser project that was just announced these are things that are very important to think about when you're building these environments if you're gonna let data scientists be data scientists and you really need to you also need to think about isolation from other users you can use taints and toleration zazz another kind of stopgap until you have more true isolation so kubernetes namespaces and we're in resource quotas so the other thing that's really typically hard in data science environments before kubernetes is the idea that we have training jobs that can last months we have we have many different data scientists that are trying all kinds of different things kubernetes has namespaces and resource quotas to help you keep users from taking over resources from each other and you really need to pay a lot of attention to how you're setting up the namespaces what tenants have what resources all of this is really critical because most if you go out and you start downloading pods or start playing around with any software that's designed for kubernetes today people do a very poor job about isolating it setting our back rules etc this is this is one of your primary tasks as someone that's managing this infrastructure you have to think about this very very carefully you want to figure out how you want to divide your projects and tenants how you want to budget resources for them how you want to give them specific namespaces you want to specify as many resource quotas as you can because if it can be taken and away from other users it will be taken away from other users so don't just think about CPU and memory that's a good start but a ephemeral storage local storage extended resources and GPUs GPUs are absolutely critical for a lot of these workloads so figuring out what departments and what namespaces have access to those resources and what that overall pool is how you allocate or over allocate those those are all critical questions that you need to think about but kubernetes does have the tools to help you do that persistent volumes for system volume claims and CSI drivers so the the main thing to point out here is that most of the if you're coming from the big data world and Hadoop most of Hadoop is based on this notion of HDFS my company ships an HDFS implementation but in the machine learning world most of the accesses are POSIX and so if you're going to be running tensorflow if you're gonna be running PI torch if you're gonna be running a lot of these types of workloads you need a POSIX driver that's integrated into kubernetes you have to figure out which one you want and CSI is the latest and greatest standard for that it just actually got to 1.0 last week the problem is it's 1.0 for kubernetes 1.13 so we still got about a month of slack till CSI but that that's really great news for for statefulness you need to move stayed out of pods now you don't have to write to persistent volumes and therefore through the CSI layer you can call out to a stream via Kafka you can talk to a database you can you can talk to an s3 type of API in a vest all of these are options but it's really critical for a lot of these applications to talk POSIX so weapon number three is kubernetes aware storage I kind of took a little the Thunder on the last slide the one point I wanted to make here is copying data sucks this is the only plug for map bar in the in this that's that's one of the hardest things to do is if I'm spinning up a bunch of tensorflow job with a bunch of workers and each of those workers needs to be talking to huge amounts of training data right now if you don't have some sort of storage solution that's moving the data across the entire environment what you have to do is copy files that takes a long time it consumes a huge amount of storage so that's something you really want to think about readwrite access to your data sets is a requirement even if you must copy so it's not enough to have just a read-only copy of the data somewhere that's 3,000 miles away almost always you'll have to put it really close to the the containers themselves data security and isolation at the user team and application levels is big so you're gonna have a lot of data scientists dealing with a lot of privacy sensitive information that how you store that and where you store that and what your storage does about isolating it is a big deal that you you need to think about but it is very solvable now okay so we created all of that stuff let's let's get out of the slideshow for a minute and just make sure everything went okay so we've got a bunch of storage pods that are up and healthy we've got coop flow a bunch of coop flow pods that are up we've yep so everything looks like it's running okay there I'm going to go ahead and start out some processes so that we can show that we can connect to coop flow if I go over and this should be the main dashboard for coop flow the main components that I'm going to show off today are jupiter hub for four notebooks and the job dashboard for tensorflow jobs so right now I don't have any jobs to show let's go back I'm gonna start a notebook up so what i'm doing here is and i'll talk about this in just a minute when you a lot of data scientists live in their notebooks and so one of the problems that you always have is they've got to spawn notebooks and notebooks are containers so they're spawning their own containers you've got to have some user interface to go in and say hey where's the docker image that i want to launch that contains my notebook and start it up that's what jupiter hub does here in this case but i'm gonna go ahead and spawn a jupiter hub instance it may take a little bit so i'm going to go back to my slides so so data science patterns so basically what we're doing most of the time is we're exploring data we're training data or we're deploying our models those are the there's data cleanup there are all kinds of other priest ages and you could divide these stages up in different ways but you're really thinking about these three things and the one that you're probably most concerned about when you're scaling this stuff up is the training side of things so I'm gonna start in the middle and I'll go back to the notebooks and talk about that but basically the idea is you need to train on lot in lots of different ways there are a lot of spark users and this audience you can run spark on kubernetes now you can run tensorflow all of these follow a single basic pattern it's very important to understand what that pattern is they use an operator I'm gonna do a slight digression what is an operator an operator are just a pattern they're they're not anything that there are two different kubernetes types of objects their custom resources and kubernetes that's the plan for what it's going to build and a controller which is the thing that looks at the plan and builds what you tell it to operators manage a higher level of granularity in kubernetes than simple controllers like stateful sets or deployments that you might be familiar with kubernetes operators allow you to install update and orchestrate containers based on your own syntax and your own needs and there are tons of operators out there there's a there's a github site awesome operators that updates every day that has new operators out there but operators tend to fall into two categories those that create a job and we're going to talk a lot about those today and those that install or upgrade products and keep a product healthy so I said all of these things whether it's tensorflow or PI torch or spark all of these follow the same pattern and the pattern is I build a job operator I submit a Hamill to that job operator they spawn a bunch of containers on the node they do the work and then they finish so they're in queue flow there are a bunch of operators just for different types of deep learning workflows so there's tensor flow PI torch spark is separate but it has its own operator to tip hyper parameter training chainer MX net MPI message-passing cafe - and pachyderm those are the ones that are in there today they're more coming along every day and the idea is that your data scientist picks one of these ways to write his deep learning job or his just statistical job but whatever he's trying to build but he selects one of these operators he creates them Yambol he fires it off and that that deploys all the containers in your environment so just to kind of to highlight that again you've got a cube CTL client you pass some yamo from it to kubernetes the controller is watching for that yeah mole in this case we're sending a tensorflow job the job tensor flow job controller sees that yeah Mille you've specified what the job should look like in that yeah moe and so it starts launching containers in the environment for you the containers are going to have your training code if this is a training job and it's going to create a generated model it's going to talk to training data in the tensor flow world there are three main types of containers a master workers and parameter servers Argo is a work flow operator it a lot of people use it within coop flow to build containers that actually run in later steps Argo follows the same pattern it has a gamble that's defined by Argo it gets passed the Argo controller and the Argo controller launches a bunch of these containers that's the pattern it's it's a pretty simple pattern everybody does it but it's really important to get that down in your head if that's what's going to be happening so each data scientists will want to run their own favorite training framework Lisa likes PI torch Mary likes tensorflow encourage that hair urging it Jannetty in your environment it'll make your data scientist happy try to run the job operators in a single namespace in your cluster what I mean is this the operators themselves should live in their own namespace you don't want to clutter and have every data scientist run his own operator so that saves you on resources but run the actual jobs in user tenant namespaces that way you lock down the resources that those tenants are using it requires a lot of our back to make all of this work but if you do that it really makes your data science environments a lot better technique to schedule your pods smartly consider spark so the kubernetes scheduler is amazing you can schedule based on dozens of things so we're putting all these containers on all of these nodes kubernetes is pushing those out there based on a request well these operators can use a number of tags and annotations and labels to make those pods live on the right nodes if they need GPU you need to annotate the pods that that they need GPU and so that that's a lot of what you're doing is you're trying to figure out what are the right annotations for my pods and what are the right annotations from our nodes spark so I mentioned all of these follow the same basic pattern spark is probably the most advanced in thinking about all of these things spark goes beyond the basics today even more than tensorflow in allowing many types of worker pod labeling and even has a custom scheduler to launch pods based on HDFS fragments so it's it there's some really cool things these operators can do if they spend a little bit of time and energy and in launching them in the right places technique number four avoid making data scientists hand roll containers this is probably my biggest pet peeve in the community today one of the uglier patterns I see over and over is encouraging data scientists all data scientists to go in and build their own contain now it's true data scientists need to have whatever libraries they need they need to they need to build things the way they want them but usually in most organizations the pattern of the way things get built get built is very very similar so rather than encouraging a hand-rolled container type of pattern the the better idea is to create mount points in the pods create standardized data science pods for training and create mount points to pull in the things you need like training data training code models libraries etc and pulled us in as volume so that everybody's using similar containers but they all may be using different code that's running inside of those containers SPARC leads the way with this spark has this notion of a resource staging server today there's some debate about whether that thing lives on and how they may do it in the future but but sparks really thinking about how to how to avoid this hand-rolled problem so this is something I expect tensorflow and some of the other operators too to kind of catch up with soon here's the SPARC operator again similar pattern you send a spark the amyl it looks a lot like a spark submit so if you've ever done a spark submit the animal looks like a spark submit it gets sent to the spark controller the spark controllers launches a spark driver which is equivalent to tensorflow master that spark driver talks to a custom scheduler that figures out where these workers should launch the executor 'z the executor is get launched on nodes the right node to talk to the right HDFS instance the resource staging server can push the training code into each of these executor x' so you don't have to have a new container for each bark job and you get a generated spark model at the end that you can use in streaming cetera okay so technique number four separate your compute from storage so you saw I launched a storage cluster I also launched some others some coop flow stuff you keep those things separated you you want to you want to scale your compute jobs or your train jobs separately from storage and that and that's something that's tricky when you don't have kubernetes remember that you can do that in kubernetes kubernetes operators so I said they fall into two categories other category or product operators a lot of the stuff that you'll be building will require support infrastructure things like Kafka things like mice SQL air flow etc all of those have operators that make it easy to install those products for for the machine learning environments we have our own operator that that's what I do in my day job but a lot of what these things are doing is you have lots of really complex types of containers and you can't just put them all in one stateful set so you need multiple stateful sets things communicating take advantage of those types of operators if you can technique five when you have when you have the complete complexity in your own environment think about building your own operator operators are really good for handling hard tasks there are alternatives you can build helm charts you can build case on it files I would really recommend taking a look at operators because unlike the other solutions operators allow you to handle the whole lifecycle they're not just about install and uninstall you can do different things depending on upgrades etc okay so we've talked about training let's go back to the first exploration users need these data science containers I'm actually running out of time so I'm gonna move kind of quickly I've got I may need to launch Jupiter our studio Zeppelin all of these different types of notebooks notebooks are a secret weapon you'll want your data scientists to launch as many notebooks as they can the notebook spawning problem is a is an issue you need to figure out how you want to do that that's something that you should spend some time in it jinan early on is how you want to spawn them Jupiter hub an ambassador is one way you can build your own and their reasons you might want to think about that other types of UI you'll have to deploy you have central dashboards tensor board for like history of tensorflow jobs art history models workflows etc ensure users can monitor their own job status this is something I see people make the mistake over and over they say well we're not going to make our data scientists have to pay attention the things that are running in kubernetes that's that's not their their job that's the kubernetes administrator no not really when when these data scientists are launching really complex training jobs that require tons and tons of memory and run for a long time they need to be able to see if everything's still okay they need to be able to see when it's causing memory pressure at different places so they can scale the memory down or scale it up you you need to really provide insight for data scientists when you're you're building the environment for them so I'm going to finish these slides and then I'll go back to the the demo models can be embedded or served by applications everybody always forgets after training how are we going to use this model so if you're going to be embedding it in an application that's one thing you need to you need you need to put it in your application but if you're going to expose it externally and you're going to expose the model over time as you make changes the model you need to think about what kind of micro services infrastructure you're going to use for streaming that model within coop flow there are a lot of different ways to do it the coolest is Selvin core it allows you to to create all kinds of different proxies to do a be testing and multi-armed bandit set it role with your models it's really cool but you also have tensorflow serving nvidia has the tensor RT inference server there's batch predict their spark streaming but you you need to be able to serve your models and so you need to think about how you're gonna do that the final slide on techniques that I wanted to make sure everybody heard is how do you handle tricky things so yarn is a big deal in the data science world we all have tons and tons of yarn where clothes our company has customers that have thousand node clusters of yarn today what is that going to go away what do you do well you you have three options yarn doesn't play well with kubernetes that's just a simple fact the first option you have is replace it so do you have something in kubernetes that with what time and and thought is easier to use and faster use you can replace it maybe drill for Impala sparklin so the next option you have you don't want to throw everything away though the next option you have is rebase that tool spark on kubernetes is a really good case study and rebasing the technology on something a little more modern Haemon spark could be another way to do some of these big data and workloads the final is coexistence so it's possible to run containers of running yarn in kubernetes it's just really ugly but you're a yarn container is basically a container that starts out empty and then starts filling itself up so it's not locked down at container design time so it's very different than most types of containers but that works and you can use it in very specific circumstances it also may be a viable option to just keep running those those young clusters outside of kubernetes that that also makes a lot of sense but that's that's the end of the presentation I did want to just show you guys that some of this stuff came up let's go I did want to show you a spark or a this is a tensor flow job Yambol so if we're going to send a job to tensor flow this is what the amal is it says we're gonna create one parameter server these are the arguments to the tainer this is the where's it yeah this is the image that it's going to use the the docker image these are the workers there are their parameters that's that image and if I go and I go over here let's see and you tube CTL apply yeah I'm pushing out that tensor flow training job and it created that so if I go back over our jupiter notebook container came up that we launched before i'm gonna go back and go back to the coop flow go to the tensor flow dashboard and here's our tensor flow job that we just launched its got fifty forty replicas and one master image or one parameter surfer or like we specified so it's very easy to launch this you want to make sure you're launching it in the right namespace that's that's the key to that that's one of the key points that I want to just emphasize this we're wrapping this up but thank you very much I appreciate all the support [Applause]