SF Scala: Tim Kral, Enabling Big Data and Machine Learning for the Masses
Recording: SF Scala: Tim Kral, Enabling Big Data and Machine Learning for the Masses
[Music] like I said I wanted to thank everyone for being here and I want to give a little background on this talk I give this talk like a sort of modified version of this talk maybe six or eight months ago to the internal company and if you ask you know there's various medium employees in and about the crowd and if you ask them what they remember about that talk they'll tell you that I introduced many of these concepts using internet cats because I love cats and that's mostly the reputation that I've gotten at this company so when I was putting together this presentation I asked my wife hey should I include cats do you explain these concepts and she's like yeah I don't know so but we decided I would pull the audience so do people here like cats I mean is that like it's not like a good area cat sorry well maybe you guys will see some cats in its presentation all right okay so what's on the agenda I did include some comics because everyone loves xkcd yeah so I'll try to not show you random slides and have them make sense all right so I'm gonna sort of reiterate what I've talked about that medium is doing give you some technical overview about how we're solving our big data problems an overview about our Big Data Platform version 1 version 2 what we learned in between a little bit about our ML platform built on Scala all all the big data platform ml platforms built on Scala what's in store for the future and I will do my best to answer any of your questions if you have a question just shout it out and I will stop talking in try to answer all right so what do we do at medium all right well so like I've said there are people out there that want to consume written content form and people who want to produce written content and medium is sitting in between these two trying to match them and on the backside we would like to pay riders for their actually produced content so where does Big Data and ml fit in all this situation so first like F was talking about we have recommendation system to match readers with the content that they'll like from our writers we have pipelines to automatically annotate and categorize content coming onto the platform and this is kind of where we mix machine learning and human intelligence together and we have ETL platforms or sorry ETL pipelines to support some of our external facing these cases such as paying out writers and we have a bonus use case where we're using ETL pipelines to measure how well we're doing this is a cat representation of Gustave mediums head of product science I did get his permission to post his cat likeness on the slide so no worries there all right technical overview so here we go so at a 10,000 foot view this is what we're doing it medium we have spark code and I'll get into that in a moment why we chose spark and our engineers will submit that to our CI server that will build the code post it to our hosted file store actually I think I have yeah there we go submits it to the CI server it goes on our hosted file store we have a cron that kicks off and submits a job to our spark cluster and then our spark cluster is reading and writing to our various data stores that we use all right so why did we choose spark so actually Alexi asked me this question in the little interview that we had before this talk and really the answer is that first bullet point there you know I know medium has a brand and were you know we're known in the world but we're actually you know still a fairly small start-up and so we're having to rely a lot on our in-house expertise so we had people in-house that new spark and so we had to rely on what we knew so so that was one of the big reasons you know and then the following that both points after that are you know also true it's a generic framework that covers our use cases it's the industry hardened and it has integrations with the data stores that we use so that's great ok so people always ask us you know why did you choose Scala but it's really for similar reasons we had in-house expertise on Scala so we chose to use it and of course it's the first classes on spark which was a choice that we had just made so in-house expertise great IDO yes so this was a third choice that we made to watch our stuff on AWS in EMR can anyone guess why we did this manager yeah that's true but in-house expertise but yes so and then all the nice things about Amazon low boost rocking low bootstrapping costs available support and we run a ton of stuff on AWS so we really like it DevOps loves it so so yeah that's why we chose the EMR another interesting choice that we sometimes talk about is we decided against using streams at least in v-0 and that's again for business reasons not necessarily technical reasons bulk turns out covers most of the use cases that we have at the company or at least gets us 80% of the way there we are on occasion running back fills of our data which can be difficult over streaming we are recommendation system like it was talking about is not quite to the Facebook and Twitter scale yet so streaming has less value for us and like that last bubble point set we just don't have the engineering resources to go tack that problem right now although we are hiring so maybe that will change okay so taking all that together we set about to make a data platform and here is the best xkcd platform cartoon that I can find which I thought was pretty appropriate alright so one thing that we did was we combined our CI and cron server together and that just made it one less moving part we use Jenkins again in-house expertise and so that's great so we submit our spark code the Jenkins Jenkins serves as our cron server as well it's reading from our stateful cron database posting to our file store which is s3 I can tell you and then kicking off our jobs in our spark Buster and I also point out for our MVP just in case people are curious we actually were cut out EMR altogether and we were just running spark our spark culture locally on Jenkins so it was actually quite a beefy beefy worker cluster there all right in terms of the spark cluster word that we were using we set up what we term now is static clusters per environment so we had three basic profiles dev pre-release and prod and so we just had static EMR clusters that ran and we submitted jobs over Jenkins and via cron so that was all great and of course this is a Scala meet-up so I would be remiss if I didn't put some Scala code into my presentation so what we do we relied heavily on utilities and mix-ins so we treated each job as a separate application so you can see this is an object that medium job base class actually had the main method and we had a very small small spark wrapper around spark submit those ardent Python so the cron would run the Python which would call spark submit with the our spark jar on its class path and it would call the our job our Scala job so this little example we are matching cats to people who love them and want to adopt them so you might see this example and other slides in the presentation okay so mix-ins provided functionality through inheritance we had utilities available to help extract and load the data frames that we had okay great so as you can imagine there were quite a number of problems so I'll highlight a couple of them one we did build in a mechanism to create dependencies between jobs so pipeline would you know extract and load and then a downstream pipeline would extract from those results and load again and sometimes we would go to three four levels deep into that dag but we forgot to build a visualization tool to manage that which turned out to be pretty bad because as you can imagine something upstream would fail that would have cascading failures across our system and we know who knows going on until we dove into Jenkins and look at the cron so not a good situation and as you might imagine because we were running these static clusters and we had lumped our jobs together we were all running the risk of cascading failures because we didn't have enough isolation in our cluster setup so this was also a problem in terms of Scala we had some more problems one as you guys might know spark has this pretty nice feature where you can run locally and and it makes it very easy to set up the unit testing but since everything was happening through mix and inheritance it was basically impossible to unit test any of this stuff without an actual database connection so we're sort of flying blind as we were developing these things just submitting it to production going through logs trying to debug and then rinse and repeat okay so that was a big problem - we found that people were constantly reinventing the wheel when they were going to extract data from our data stores and this was particularly bad because medium is not a scholar shop by Nature we're actually most of our people know JavaScript so people were hunting about our jobs and doing a lot of Cod copying and pasting which sort of was bloating our code base and the data platform scene was sort of losing control of what was going on and we're having to support all these scenarios that people were hacking together so this is really really bad okay so we sort of gathered up all these problems and we made a spreadsheet and we went for a version two and this was maybe about six months ago okay so version two I'm gonna this is gonna get deeper into the Scala code here - okay so one thing we did was we started doing better isolation on our spark clusters so we sort of defined some verticals we had Tier one vertical we had ml vertical we had a dynamo vertical we were doing extraction and loading from our dynamo instance and we set up clusters around that so that's great so more isolation means that we're less at risk of cascading failure great we also turned on a dynamic feature in EMR so that the clusters could elastically expand and retract based on the usage and so that just made the whole thing more stable and we're dealing with a lot less you know pager duty so that was great and we actually went one step further and we turned on what we call transient dynamic which is another feature in EMR which allows us to spin up instantaneous clusters per job so now we have ultimate isolation we're really not afraid of cascading failures anymore so that's great and what's even better is as Amazon is releasing newer newer versions of spark we can safely spin up transient clusters to go test to test features in those versions so this has been really really great we have actually a couple of pipelines that are running at older versions of spark that we can't quite migrate yet because of some spark bugs and this feature saved us because we can isolate those totaling to their own cluster and it's it's really fantastic the one downside of this is it can be super slow to create these especially if you're trying to rapidly iterate for development purposes and I'm going to talk about that in a little bit we have some plans to address that actually okay so like I said we didn't build any dependency management or visualization which I would recommend against but in version 2 we sort of got a little more serious about this so we built an in-house visualization tool to examine the execution DAGs and this has been a lifesaver because we can anytime a jobs failing we can go view the execution dag and see if there's upstream jobs failing and if that's having a cascading effect so that's great and we even have a little static checking at the CI level so we will make sure that your DAGs are correct before they go into production and we have cascading failures indicated great okay so so we set up some of that stuff and that made things a lot more stable but we still kind of this problem where we had non Scala experts wanting to write the Scala code and sort of doing the copy-paste thing and losing control of how the system we're as the Data Platform seen we're kind of like we couldn't keep control of how the platform's being used so what we did is we sort of sat down and we examined like okay what is the anatomy of an ETL job all right well just the the a command ETL sort of lays it out for you like first we're extracting then transforming then loading and more to the point extraction loading have configuration that we need to inject so we said okay as the data platform team what if we just sort of commoditized extraction and loading as services so that those are generally available to everyone and then what people and the company will do was concentrate on just the data transformations which one you think about is really the meat of what they're trying to do the jobs that they're trying to get done so this kind of provides this nice dichotomy where the platform's team is doing just sort of what's the word that I'm looking for boilerplate work and people don't have to like worry about tuning spark and configuring spark and and how all that works and they can really zoom in on like the business logic that they they want to produce to get the job done that they need to get done and so sort of to wrap this up in a statement as a data platform scene we would provide extraction loading as services instead of those be like controlled by developers in the company okay so so to do this we sort of went from the inheritance model into the composition model and we did it with dependency injection so here is our cat owner matching jobs sort of reimagine with composition we have separate entry point for job execution and I'll tell you why this is important later so all your extraction and loading functionality is provided through composition by injected services so that's great we had this idea of doing self documenting mix-ins for configuration taking advantage of polymorphism here and so that made it that made our data extraction and data loading very very typesafe so that was awesome and it had some other benefits which I'll talk about in a couple slides but most importantly what this did was it made the methods that we were producing as a data platforms team extremely easy to understand so when a non scala engineer started typing in their ID and said data loader dot and it would it would autocomplete with extract of three extract dynamo extract redshift this made it very very clear what was going on what methods they need to call to get their work done and then in the day loader service we're actually loading we would allow developers to specify all the destinations for their data so and we would handle that all at once in one shot so this was great so in terms of so when we started diving down deeper we wanted to make sure that we built a platform that was extensible because we we couldn't anticipate all the use cases and all the data stores that we were going to use so we decided to do this through plug-in architecture so if you consider the data extractor and data loader as services that were gaining inject into your job behind those services we were going to build plugins so these are actual plugins that we have today on the platform we might I think this is pretty much it we might have one more loader plug-in actually but the data platforms team actually produced all these plugins which which enabled this functionality to all the engineers across the company but more importantly it also sort of provided a an extensible framework so that if there's a business case to add another load destination or another extraction destination or source that was very very easy to do so there are 18 point-to-point combinations we don't use all 18 ok so what did that look like in Scala code so we started sort of at this service definition level by defining a Scala trait here and like I said we concentrated very hard on naming the methods in a way that would make sense to someone who wasn't familiar with big data or familiar with Scala so here we go for creating the data loader plugin we're gonna load an RDD it's going to have a destination an RDD based on protobuf messages and Save mode pretty simple to understand and here is the example redshift implementation for that plugin this is actually code that I copied out of our code base but I tried to obfuscate some of the implementation details and that simply implements the plug-in and then one of the things we used was this these Auto bind annotations and so what that did for us was it allowed us so the whole thing is based on a Google juice to do to run the injection and these annotations is like a very thin layer over juice that that allows for dynamic classpath scanning so when the application starts up the application being the detailed job itself when that starts up will scan the classpath that will inject all the services using juice and actually I have it in the next slide and so that injects all the services using juice and you can check those directly into your job and use them and what's great about this if you can see we can inject because the pipeline's themselves run at separate applications we can inject pipeline level configuration through juice as well so you can see like a pipeline name and the import ID is a sharding mechanism that we use at medium for our pipelines so right so at the service level how do we use these plugins we define our here's our definition of the data loader again we're exposing the singleton using this annotations and we're supposed to them using the data loader trait so that's great and abstracts away all the details of the data loader from the job author and then as we saw in the slide before that gets directly injected into the job itself so very very easy for someone to write a ETL job okay so let's do a time check so one of the big problems we had like I said was testing and one of the awesome things about doing composition with dependency injection is it totally opens up our ability to test this stuff so since the extraction in loading pieces are now platform pieces we're telling job authors you can trust these these are well authored stable pieces that you can rely on so let's divide and conquer as a testing strategy so we're actually going to isolate your data transformation piece and we're going to surround that with what we call a mocking data extractor which means we're going to mock out the data through spark data frames that your job uses you can run that through your real transformation code and then we will give you what we call an asserting data loader which will simply assert your transformations at the very end so this is a kind of a dense slide but this is what it looks like in Scala code if we were testing our cat owner matching job oh yeah we wrote a little wrapper around that so okay we're oh yes so here we are mocking like I said we're mocking out our our data frames we're saying that the mock source is s3 so that the job recognizes it as such and then we're going to assert the loaded results in our serving data loader and the a dependency injection so what this line right here is saying is we're going to we actually create a service injector we do the classpath scan and we grab we create the actual graph of our services and then we override the extractor and loader within that graph so we can like we can sort of pinpoint the services that we're trying to mock in the graph this is actually a key part of this testing framework because it means we don't have to mock every single service in the graph like we can rely on juice to create the full graph and just mock the services that we care about in the test so okay what are the what are the kind of the general results of this well we started to see more and better testing and we started invest more and more in testing jobs before they got to production so we're going to catch logic errors and before we even release them to our CI server because we can run these locally and certainly our CI server acted as a safety net back up so that we weren't going to release regressions into production so that was great number two as a data platforms team we started to get tighter control over the elements that we were producing so we had like you saw in the trace that we were defining we had very specific contracts with job authors this is what how you how you use these services this is what you can expect for these services and so when there was a change of those contracts those changes were managed and we can communicate those out effectively to the rest of the company so because of that we started to see more engagement across teams one of the best compliments I've ever gotten in my entire career was one of our engineers looked at me and said you've made spark programming fun which was something I'd never expected that I would hear to hear but but I was thankful and you know and my next one-on-one I was like telling my boss I was the greatest compliment I ever had in my career but but I kind of agree with him like if this is kind of made spark burning fun because we've taken out all the all the nuance spark stuff that you have to beat your head against the wall to get right and we've allowed people to just concentrate on doing transformations and solving business problems and we even found that there are some non programmers that started getting into the act and like you know playing around with the with the with the framework that we had built and you know Gustav was was one such guy and and so we were very flattered that that we were influencing even non-programmers in the company to to try stuff out so kind of once we started settling the big data question especially around Scala we decided to take it one step further because like I was mentioning we had some ml use cases that we wanted to go address so we sort of took the same approach okay whereas before we were talking about the anatomy of an ETL job now we're we were thinking through the anatomy of supervised classification so kind of at a very very high level supervised classification you take your training data you extract all your features you son mold them into a nice vector and you build a model then you take your unlabeled data you did the exact same thing you apply your model to you're assembled feature vector and that produces results then you evaluate you can evaluate those results with machines and humans and your from human evaluation you can create a feedback loop into your training data which in theory will make your model better which would make your evaluation better and the whole thing proceeds from there so we kind of went through the same exercise like I said and we said hey maybe people shouldn't have to worry about how to assemble their future vectors and evaluate their models and build their models like maybe that's something that we can provide for them as a service and instead they can concentrate on finding data that's going to be representative of the problems they're trying to solve and and extracting their features and this one I kind of it's sort of half purple half orange because there's some infrastructure work that I'm about to show you that we did to sort of make their jobs easier but in the end really what we're asking people to do to solve these types of problems is find training data and decide what features you want to extract and the platform kind of takes care of the rest of the rest of the problem okay so this is kind of a high-level view of how the feature assembly works like I said that your spark table is going to extract your training set we have what we call a metadata store which injects metadata about your entities into a series of feature extractors those extract features into their data frames we join then we join against your training set and we have a labelled feature vector that we can pump through a model builder so looking at the Scala code your this is going to look very familiar because you know the the pattern we were just reapplying this pattern that we had built with our ETL jobs so let's define service for each extractors this is a global definition and then what we said is ok even though we have this global definition of a feature extractor each job is going to have very specific needs for the types of features that it wants to extract so we're going to allow for a local definition of a feature extractor that's a tie to a specific job this is real code that I've copied out of our code base this is the feature extraction definition for our intrinsic quality ranking job which as you imagine tries to rank posts based on their intrinsic quality and then we implement our local feature extractor using a local service implementation so in this case we are going to extract intrinsic features from the title of our post and that's going to become part of our vector which is going to be used to build our model and then that whole loop proceeds from there as you can see the the annotation that the auto bind and annotation slightly different were Auto binding this and a set of services instead of as a single pin so that set will get injected into the juice graph and then we'll run through all the future extractors extract health features build a vector etc ok so similar thing with future similars the global definition a local definition which basically just sort of defines the entities that we're trying to classify and injects all the feature extractors that we've built on the previous slide so this is the definition of the intrinsic quality feature assembler that we have and then there you go our ranking job can use our future assembler and actually didn't put the call in here but there's a very simple method call that just simply says assemble your features and the system will run through all the future extraction feature assembly what I don't have in this slide we actually did build a model we use these same similar concepts to build services to build models as well and that's basically a thin wrapper around sparks @ml lib okay so how we do in that time no pretty good okay so I just have a couple slides to talk about what's in store for the future and then I'm happy to answer questions okay so like I said in terms of our cluster we've gotten a lot of mileage over our dynamic clusters and even our transient dynamic clusters but one thing that we the data platforms team has done is partnered with our DevOps team which is evaluating kubernetes right now and we want to do what we were calling transient dynamic clusters with pump priming which means that we're going to keep a floating pool of free resources and we can instantaneously sign up a cluster the hope here is that our clusters are much faster to sign up and the development iterations can become much much faster so we're going to invest a little bit more in dependency management this is a really really important piece so I talked about sort of the upstream views that can show cascading failures we're going to be adding a downstream view that shows the impact of failures downstream we're adding manual retries for our DevOps rotations so that just a single button press when your page at 3:00 a.m. we're doing stuff like better access to logs and I should say this is this has been built in house kind of as a bare-bones thing to get us off the ground but we're also interested in perhaps implementing a system that does you know real real serious dependency management stuff like air flow out of air B&B so that's in the future potentially one of the things that's near dear to my heart actually is this idea of Auto deploying machine learning models and sort of the the rough idea would be once you build the model you sort of you you run the model on a subset of your unlabeled or you run the model on a subset of your data you evaluate the results and if the results are good enough you ought to release the model that's sort of a very high-level description of that but right now the way we release models is is extremely manual the way we select features is extremely manual we'll program them we'll run the model we'll look at our results and we'll kind of make a judgment call whether those results are good or not and if they're good enough we'll release the model so we're hoping to sort of tighten up that loop and make this a more automated process and finally one of the things that we're sort of dabbling in is this idea of rapid prototyping of ML models as you guys might know as I'm sure you guys know Python is like a preeminent language in developing machine learning models and sort of one of our theories is where it shines is in this idea of rapid prototyping so you can get in there you can sign your features you can build a model you can test it very very quickly so we've been sort of thinking about and Prout typing ways to maybe do a little bit more science and python to allow us to get to a result faster and then productize that in Scala to sort take advantage of all this platform stuff that we've been working on so that's really end of my presentation the you know I kind of want to highlight those future slides sort of the things that we're working on over the next quarter and the next year because medium is hiring these are some of the open roles that we have we have we have recruiters in the audience and we have Brian is back here and gyrate waving his hand and John over here these are heads of engineering so I'm sure they'd love to talk to you if you're interested in coming to work for medium and I am happy to talk to you as well if you would like so with that I will open the floor to any questions I will do my best to answer them I will say that I finished this slide deck at two o'clock this morning so I'm a little little tired and when I finished the only person left awake in my house was my cat so I took a celebratory selfie with my cat to celebrate the end of these slides and so I'll show that picture to you now there it is in his name is Sam thank you very much for this as you can tell he was quite pissed so I understand that you can read formats different formats but I saw your s3 plugins and that means that and this data step can-can with director professor right so what's in the extractor though right so got it so if I can summarize your question like sort of [Music] the summary to your question is or to summarize your question what what are we adding on top of spark within that extractor so the answer is configuration really right so I mean we needed a way for people to easily and smartly configure like for example the bucket that we are going to go fetch the data from right we actually run dev pre-release production instances so we needed a convenient way to like I actually can get into configuration in this presentation but we wanted sort of like a one-stop shop configuration marker exactly yeah so exactly that that's sort of one thing that falls out of this and you know it's sort of a it's a separation of concerns right like if we can if we can abstract away all the configuration people can't just concentrate on the job on the date of transformations and and then the and benefit like I talked about with testing is now you can inject mocked datasets into a test and run through the transformation to test your logic looks like that for a purse party come easily these are for max parties and as really you're different in assortments really just said different dollars you can't easily dataframes our beliefs or so it's and really as you said the configuration the confirmation is not a part targeted relational so look so that part of the final standard writing write a plug-in just for that my second part of the question is you write all these and funky who's that turning user are you expecting the data science analyst or he'd just write a secret so I could use that one oh that's okay so I'll answer your second question so the question was what is who are the users that we're targeting here so the people we were really targeting are not Scala people like and the plugins are actually abstracted away from them like those are things that the data platforms team is working on so the idea being that the not someone who doesn't know Scala and just wants to run a query or do some simple data transformation all they have to do is inject a service and call simple method enjoy are they analysts or developers that they're developers they are developers so gustave the the cat representation of Gustav he's our analysts so he is writing a little bit of code here but we're mostly targeting our product teams developers on our product teams people when people so they can write stocks opposed [Music] I thought right yeah so and so the answer to your question the question is why aren't we why isn't the data platforms team writing all the Scala code and the analyst is just running the configuration so the answer to that is because we only had two people on the data platforms team and they we just didn't have the resources to support all the use cases that were being thrown at us across our product teams so that's why we created the platform to try to enable them to write their own pipelines that that was sort of the genesis of that decision so we use the we use the basically the single split right I mean it's just that the twofold validation yeah that's where we started I believe one of our engineers looked into like I know spark supports k-fold we looked into that and I believe the answer was at our scale there wasn't enough of a performance boost ongoing with K greater than 2 so we just stuck with the regular the cross validator the train split I forget what it's called does that answer your question yeah did it did I repeat the question I did not repeat that sorry the question was we're using K fold evaluator in the machine evaluation you talked about there was a self document who makes him that he said you're taking avenged oh sure so with the yeah it's the the extractor has very specific methods so will say extract from s3 or extract from redshift or extract from my sequel and each of those methods takes in a configuration mix in specific to that data source so a job has to be has to extend the correct data source and if it doesn't then we'll get the compiler okay so I I think I forgot to repeat that question to the feature stores we store the question is where do we wear our future stores we store ministry oh yeah the metadata also in s3 so the question is have we evaluated systems other than spark to that take advantage of plugins the answer is no we have not [Music] sort of relationship you sort of mentioned you guys it's just curious why he did that as opposed to using out open source right that's a good question so the question was why didn't we take advantage of an open source dependency management system like air flow and so the the the the answer is we did put an engineer on specifically air flow trying to roll that out for a week he couldn't quite get it to work and so reserved time we just time box it and we it since sort of the need was very dire to have insight into things that were failing we just sort of rolled something very simple the stuff that we that we rolled out took you know 24 hours to produce and it's actually something that I really want to go and do something about because I tend to agree with you that there are totally yeah 100% agree so the question is can i elaborate more on the otter deployment of machine learning models my short answer is no because this is sort of something like that's in my head and and I should also point out I am I am NOT a machine learning engineer by trade so there may be people in the audience that know more about this subject than I do and I'd be happy to meet you afterwards and learn from you but sort of the very generic idea is that it it's very manual and and labor-intensive to evaluate models right now and decide if they're worth deploying and we want to create a system whereby machines can help us do that so we're setting baselines and for exceeding the baselines through like a K folds evaluation that we're just going to auto release the model into production yes I'm sorry how do we test our work yeah the question is how do we test our work and in terms of the Scala yes yeah so so we actually do two things one we write the test the end end tests like I talked about in the presentation but we also have the ability to submit to our spark like we run Def clusters if you notice and so those clusters exist to be able to submit jobs from your laptop and so sort of the general cadence is you create that you engineer the job you'd write some test for the job make sure those are working properly then you submit the job to a deaf cluster make sure it works properly there then you send it up for a code review I think it's reviewed and deployed to production last question so the question is are there any plans on open sourcing this work the that has been broached and discussed the short answer is no there are no immediate plans but I am an advocate of that so so we'll have to see what happens two points [Applause]