scale.bythebay.io: Anya Bida, Just enough DevOps for data scientists.
Recording: scale.bythebay.io: Anya Bida, Just enough DevOps for data scientists.
you hi so this talk is for you if either maybe you're a data scientist but your team actually needs infrastructure has that ever happen to anybody it hasn't happened to my company but this talk is kind of like what I wish every data scientist knew about DevOps or maybe you just want to know what your infrastructure team is doing behind the scenes so that's what this talk is about just enough DevOps for data scientists first a little bit about me I'm Anja and my twitter handle is here I'm a senior member of technical staff I'm on the SRE team at Salesforce the site reliability engineering team I support the Salesforce Einstein platform so you might have heard about that a few times at this conference so you can talk to any of us about the platform remember what it's about we are hiring hint I am also a cool organizer of the SF big analytics Mena we have almost like 5,000 members we're growing really fast so that's a lot of fun too I've done some work around spark tuning there's a cheat sheet that you can click on I'll post my slides later and I've done some talks around that so come talk to me about those things if you like so what I'm gonna talk about today first what is DevOps just so that we're all on the same page tell you about Salesforce Einstein and how we're scaling I'm gonna tell you about our goal from the infrastructure perspective and I've kind of distilled down some best practices that I've found as we build a scalable and machine learning platform so I'll give you those top 10 tips and then think about what's next ok so what is DevOps I consider DevOps to be kind of that intersection between software development infrastructure we build and release and also some networks and security so that's kind of my working definition of DevOps so we're kind of at that sweet spot right in the middle maybe you think of data scientists is way outside I hope not maybe some people do so when you go to data scientists you think oh I want to run my machine learning job I've got this awesome library I've got Optimus Prime and I want to run it on spark mo okay so hey in front can you just give me some clusters I need a spark cluster oh oh and I need some microservices okay so we need to deploy a cluster maybe we got a cluster running let's deploy those micro services in some containers so maybe this brings you a little bit closer to your DevOps team I hope and so we at Salesforce we're scaling like crazy as you probably know so I need to think about all these requests from my data science team at the level that Salesforce is scaling and Salesforce Einstein is is no exception they're serving up four hundred and seventy-five million predictions every day so we're definitely needing to build and build and build but what is my goal from the infrastructure perspective as I increase the number of predictions that's my blue dot I want to scale the number of predictions as much as I can but my infrastructure costs should not scale at the same way right so I want to keep those in check so this is my goal and I hope that we can all kind of agree on this this unified goal so if there's any people in the room who still have their data science hats on go ahead and take those off because I think in data science school they teach you if you build your model and if you keep working at it keep training then eventually your models gonna work consistently but in the infrastructure world I'm running thousands of servers and I know that some of them are going to fail even if I do everything right they're gonna fail right so I need to plan for failure so that's kind of this tip number one plan for failure I know failure is gonna happen I actually need a way to dig in and troubleshoot quickly so I need a dashboard and the dashboard in the car is a awesome and it's been that way for a long time because it's simple it's got my key performance indicators it's got my speed amount of gas in my tank that way if there's a problem I've got a simple place to look and I can dig in later and dig in for more details so what does the dashboard look like from an infrastructure side I stole this dashboard from Yoast Bauman's whoo on this hills forest data management platform so this is a very simple dashboard it made it look simple at first but I'll go through it we need a dashboard like this to give me information about every machine in my system so here he's got he's got a service that's this is a micro service that's running and over this two day window I can see the the line that's fluctuating up and down let me point to it see if this will work you can see that yeah okay so my line that's fluctuating is gonna be my number of requests coming in and then the line way down at the bottom is the number of errors that are coming in right so my error rates so that's that's the first thing I want to see on my dashboard like the speed on my car I need to know how many requests are coming in and how many errors are coming in secondarily I want to know the response times so I'm building a micro service I want to know what type of latency are my users experiencing and I don't want to I don't want to know the average latency because I that's that's like saying I don't really care about half my users right I want to know the worst latency I want to know the the highest response times that people are seeing with my micro service so he's got that plotted the yellow line is the upper 99th percentile of response types this is this is kind of a key performance indicator for my micro service for something really handy the other thing he's got plotted in the vertical lines are events so let's say there's been a deployment I can see that event and I can see oh this is how the response times changed after that the particular deploy maybe my data engineer wrote some code and it wasn't so great so I can see how that's related so the point here is that you need a dashboard like this that shows me my key performance indicators I want to I want to deploy that dashboard and I want to collect those metrics for every single machine but I don't want to stop there like when you're driving your car you see oh the engine light came on so I might see that on my dashboard but then I need to dig in more I need to open the hood there's a chip in every car that you can take to your mechanic and you can find out tons of metrics about your car and the same thing applies here we want to collect every single metric possible Staff D is a great open source tool that lets you couldn't collect metrics so I would recommend trying that one so the first tip there is just a plan for failure and know what you're gonna do to troubleshoot the second one probably everybody's familiar with this may be data scientists alright so I'll just go through this quickly Bluegreen deployments so I've got my users and they're talking to my service that's running on the current machine which is blue let's see oh no this machine is having problems it needs to go under maintenance I need to spin up a new machine my new green machine all new connections should go to the green machine right and the old connections will just continue connecting to the blue machine until they drain out so this is the concept of Bluegreen deployments ok so that's pretty simple and the next one assume people make mistakes and I mean people I mean me and my team and you and your team we are gonna make mistakes I kind of Group the catalyst takes into two categories a technical debt and like overusing resources okay so technical debt I think that every manual change that I make is some technical debt that I have to go and fix later other things like if I'm building my dashboard let's see I made duplicate metrics I need to go ahead and clean that up so we actually plan that into our system so we have quarterly debt days so the SRE team goes into a room and we close the door and we just handle our debt we don't take any new requests right so we handle all of those manual manual changes then add those to some automated tool in some way so handle your technical done the other ones scaling down resources so this is kind of a no-brainer spin down your unused instances if you're using AWS janitor monkey and is a great tool for that Netflix came up with this whole simian army but janitor monkey is a great a great tool it says it says hey I've noticed that this machine is idle let me send an email to this particular data scientist who's responsible for that machine I mean if that data scientist sends an email back to janitor monkey basically extending the lease on that machine ok that machine can stay up otherwise that machine's gonna be spun down so it's been down unused resources kind of a no-brainer and that really helps you understand the cost per job when you're thinking about cost to serve and whatever service that you're providing it really helps you understand that cost per job okay so assume that everybody is going to make mistakes and then the oh that kind of leads me to my next point all of the changes should be auditable so mistakes or changes that we made on purpose they should be auditable everything from network connectivity to infrastructure to everything so here's an example Q who's a SRB on my team he built this tool it's not open source yet but I just recommend to build similar tools so so what I'm going to tell you so this tool is called shaper it compares schemas so this is one example of making sure that every change is auditable so let's say I've got a database on the East Coast and I've got a replica on the west coast I want to make sure that the schemas and both replicas are the same so let's say let's look at my schema I've got a create table statement so my schema looks pretty similar between East Coast and West Coast I've got my list of conferences the columns in my schema the name of the conference the city the the date where the early bird discount applies very important so it looks like my schemas are this time they're identical and so shaper will actually continuously run and Aude it on my schemas Schaefer will generate a report if there's any difference so if there's a difference boolean and what the difference is and that report can actually send an email to to me or to a listserv and say hey there's a change and then we can talk about the actions that op shaper will initiate then let's see what happens so oh no there's been a change on the East Coast somebody's added a discount code I need to know this this is important so there's been a change shaper will say oh there's been a change and it will send an email shaper can also enforce that the schemas should remain the same so cheaper can actually say no you can't you can't add a column to that to that particular schema in that database because it has to be the same so the actions that shaper will take are configurable to the particular schema and cue actually wrote this wrote shaper to be generalizable so shaper can be used to compare schemas for elastic search or Cassandra or MongoDB so it's generalizable across databases and you might be thinking wait MongoDB is skip Ellis so so so cheaper actually thinks about indices and compares the indices across the the MongoDB database examples the other thing about cheaper it's modular so it's plug-and-play so this is just an example of one of the tools that we've built to make sure that all changes are auditable and I would just recommend that you think about these types of tools and your sisters too so we talked about a lot of changes there's all these configurations that happen so the next tip is to use a configuration management tool an example might be console from hashe Corp so where am I getting all these configurations well let's say data scientists and data engineer says hey hey infrastructure I need you to do something I need you to I want to run a new spark job and I want you to spin up a new micro service for me no big deal right well I've got maybe 20 parameters I need to configure for network connectivity I've got maybe 50 parameters to configure for user access which I probably should use Jeff for puppet 4 right I want to deploy a cluster so maybe that's 20 more 9 different parameters so all these parameters they really add up let's say I want to build this in three regions and I want to collect 20 metrics per host that really adds up right I got 6000 metrics it's just 6000 different configurations that I need to set up I need a tool to do this right so use a configuration management tool and to help you with that pick a Nieman convention and stick with it so for example a naming convention might be service environment region host name and metric it's just a naming convention but it's so helpful if you want to create some templates for automation if you want to do service discovery create your dashboard so I can I can filter my dashboards this is everything related to product this is everything related to dev really helpful if you actually want to query your logs so that naming convention comes up there too and then for cost analysis if I want to know how much is the scheduler costing me then we convention as hell for there so I've got my server that I've been asked to spend up I need to think about the users and the permissions that are going to access that service so if you are using AWS this is kind of a paradigm that you can take home and use AWS things about permissions in terms of I am roles so identity and access management it's basically the concepts of authentication and authorization so I've got my user my user has an I am role and that allows the user to access the service ok great may be the service is the scheduler and this scheduler is going to create a spark job that's going to run on for example an EMR cluster well that scheduler has its own I am role and the job the spark job has its own I am role and the EMR cluster has its own I am role I hope you're getting the point that we need permissions specifically at every level of at every step and these I am roles then determine what that user or service or job what that job can write to or what can read to so I can read from the input bucket and I can write to the output bucket so the point here is apply permissions at every level ok so I've got my permissions I want to run my job ok I need to now understand resource allocation and this comes up so frequently so I've got my spark job and I want to run my spark executor so on the right here I've got a diagram that I stole from to folks at Duke University which I really like this diagram shows the spark executor and how this memory is handled in the unified memory pool so that's awesome for understanding the resource the memory allocation within the spark executor but what I want to talk about actually is outside the spark executor so let's just look at the blue box the container let's simplify things this is just my spark container it's 8 gigs it needs to run on a particular node so my question as a data scientist I want my job to run can my container launch on my cluster ok let's see so my cluster is three nodes and okay great each node is eight gigs awesome oh no I see that four gigs are already used on each note of the cluster so can my cluster can my container launch what do you think No right the answer is no so I might think my container could launch if I'm monitoring my cluster at the total amount of resources at the level of the total amount of resources so I've got four node I'm sorry I've got three nodes I've got four gigs per node hey I've got twelve gigs of RAM available michaelis my container should launch right well though the answer is no there's no there's no individual host that has eight gigs available so my container is gonna be pending until resources become available so this is just very helpful and understanding like as a data scientist are your jobs gonna launch okay and then this is another this is another tip sorry there's like a hodgepodge a laundry list of tips this one's another one that I really like so the point is to monitor multiple viewpoints so here I've got a camera most of the cameras just have a single lens well this camera has sixteen different lenses and with this camera I can use different focal points to understand my environment right so that's how we should think about our environments as an infrastructure team okay so this is an example of a tool that we've built to monitor multiple viewpoints vy5 is another member of the SRU team and he built this tool so it's a connectivity diagram so every circle represents a host and every arrow connecting them represents some connectivity so if I have an arrow from one host to another that means that connectivity is allowed on that particular port from that host to another port so it's really helpful and you can see I've got three clusters there my dad of environment my staging environment and my prod environment so it's great that I don't have any connectivity from prod to death because if I did that would be a red flag I need to look at that and I also see there's some individual nodes kind of flying off by themselves maybe those are some knows that somebody's funding up as a test I probably should spin those down and if they're not being used anymore so this isn't this is one way that we look at network connectivity and if I click on a single node I can I can say ah this is what connectivity is allowed from this particular node or I could color the the blue circles as the host and the orange circles as the load balancers or as another viewpoint I could I could limit I could search based on all the the nodes that are used for the scheduler service so the point here is to monitor multiple viewpoints okay so what if we said today plan for failure use Bluegreen deployments assume that people are going to make mistakes and have a plan to fix those all your changes should be auditable use a configuration management tool pick a naming convention and stick to it please and I should say then you don't have to actually stick with your name you can create a variable and pass that in so but just tick with the convention apply permissions at the user level the service level on the job levels understand resource allocation and monitoring multiple viewpoints and I hope that I've convinced you to deploy your infrastructure as code that's kind of the summation of all of the the items that I've talked about today so did we just automate ourselves out of a job well no I don't think so I mean now we have time to take out new projects and to grow and at Salesforce that's always going to happen so if you want some next steps did you want to dig in deeper I would check out a couple of books there's a great book from Google sre how Google runs production systems if you're curious about sales force stuff check out James word gave a talk carriers are engineering an open source ambassador and then also Yost Bowman's he showed his dashboard I think he's given a lot of great talks on the goal that I shared you want to scale out your key performance indicator but you don't want to scale your infrastructure costs so US has some great talks there and if you're curious about spark operations the book high performance spark has some really really good useful tips there there's a whole chapter on joins and then these are the talks that are given at this conference for sales for science day and so to check those out if you have questions about those talks you can ask me too and I thank you for your attention I'd be happy to take any questions and I'll be around Thanks [Applause] shout it out about example point number four about keeping your code sane I'm East Coast and West Coast okay that's it isn't that just source code control sure um so I think the point there is that let's say there was a change that was made outside of your standard deployment system let's say you're trying to replicate your databases but it didn't actually work or um you might actually want the data in Europe to actually be different than the data in the u.s. maybe companies in Europe need to keep their data in Europe so it's a it's a schema comparison tool that basically says oh these schemas are different or other they're the same so yeah it's it's kind of like after you've after the fact yes you're so Skoda you're building really cycles should usually have consistent scheme us maybe you didn't all the time hi great talk so question regarding some of the things you mentioned in terms of figuring out which containers are available and managing all so do you do this that sounds course manually or do you have some sort of single system we never do it manually here so so so are you asking how we manage our permissions so I guess like you mentioned a whole bunch of things and permissions resource allocation you have one single system used to manage all that or is it sort of well I in my opinion any tools should do one thing and do it well so I don't have one tool that manages configurations and users um there's you could use you should could use chef or puppet you know what I would recommend is to make sure that your systems are automated and if you have the same naming system they make schema or a naming convention for each everything in your system then all of your tools can work together thank you so much thanks [Applause]