Scale By The Bay 2018: Tim Kral, Enabling Big Data and Machine Learning for the Masses...
Recording: Scale By The Bay 2018: Tim Kral, Enabling Big Data and Machine Learning for the Masses...
Yeah, so um I have to admit uh I'm on after rotation this week at Medium. And so, come Monday morning, nothing was happening and I decided to rewrite these slides to make them more code focused. And then, of course, by Tuesday, everything had completely blown up. So, uh this is actually the first time that I've been running through this iteration of this talk and I'll do my best to keep it under 30 minutes so we can all get to lunch and we can address questions, though I'm happy to stay after if people have questions. Um So, I titled my talk Excuse me, I I titled my talk enabling big data and ML for the masses. Um And uh here's kind of what I had in mind to talk to you guys about. Um talk a little bit about what we do at Medium. Uh give a little bit of a technical overview of our big data setup
Although, I wanted to focus more acutely on the Scala that we had written. Um talk about what we had in our old platform, what we rewrote on the new platform, and uh some of our ideas for extending and testing under the new platform. I include a use case about how we created ML jobs on this platform. I'm not sure if I'm going to have time for that. Um but I'm happy to answer questions or, you know, if I talk quickly. Okay. So, what do we do at Medium? Oh, by the way, I really like cats, too, so you might see some cats in this presentation. Um So, there's a bunch of people out in the world that like to consume written content and there are writers that like to write uh content
And so, Medium is the platform and um it's the platform that's connecting these two groups of people together. And, of course, uh I think it was early last year that we launched a program by which we pay writers for their content uh if they so choose. Um So, where does big data and ML fit into this sort of business case? Uh on the one hand, we have uh recommendation teams that are trying to match readers to the content that they want to read. Um we want to automatically annotate content coming onto the platform from from the writers. Um We've sort of been launching a curation process, but we definitely are at a scale that we can't do that all manually. Uh and we have some ETL pipelines that support product facing features such as paying out our writers. And uh of course, we have a product science team that's trying to measure everything. And so, uh we're building out ETL pipelines for them, too
Um and so, what we found was we had a very, very small data engineering team. And in fact, for a while, uh we had no data engineering team. And uh we were trying to push people more and more into a data focused uh you know, business decisions and measurements. And it became just too much for the data engineering team to be writing all the pipelines. So, um this is kind of the solution that we had to come up with to sort of spread the love around to the rest of the company and enable individual teams to write their own pipelines. Um So, this is kind of an overview of our entire setup. Uh an engineer will write some Spark code and submit it to our CI server. Uh we have a cron running that's checking the state of the jobs and um submitting requests to our Spark cluster
And then, um our cluster reads from uh the artifact that the CI server publishes to S3 in this case. And um runs the actual ETL job against the data in our data stores. Uh so, when I previously gave the talk, I talked a little bit more about this 10,000-ft view. Um and I put a link to that video if you guys like to hear my voice. I think it's pretty terrible on video. Um but for this particular talk, I want to focus right here on the kind of the things we did in the Spark side of the world. All right. Um so, what were we doing before this rewrite? Uh so, this is kind of an example
Uh we're trying to match cats and owners. Uh maybe we're like we have cats out on the street. Um So, this pipeline uh that we've written, we're treating each job as a separate application. The uh there's, of course, a main function that the whole infrastructure is calling. And we're treating each individual job as a individual application. Uh we're using mixins to provide functionality for these jobs uh through inheritance. And uh the team wrote some utilities that support the various extraction and data loading uh tasks that need to happen. Uh So, how would we configure these things? Um We used uh Medium the Medium job trait was kind of the central trait that we would use
Um and it was a mixin, so it would provide kind of this functionality. So, we were using just um command-line flags that were being parsed by this little Twitter util library. Um And then, we would pair the job the Scala job with a YAML file. And so, you can see here's an example of um some job-level configuration that we'd set up. Uh we're big believers in Protobufs at Medium to describe all of our schema. So, in this case, um we were uh Oh, I kind of messed up this example, but like, for example, if you wanted to inject a Protobuf class into your job, this is how you would go about doing it. Uh and then, we had various other uh sort of what we considered platform-level configuration values. So, for example, um what's the Redshift schema that I'm loading my data to? Should I truncate columns when I load? And then, what the S3 bucket uh means is we're actually writing temporary results to S3 to do post-processing during the loading step
Um And then, there's this sort of second level of configuration that we were in the middle of doing and that was all injected through environmental variables. So, in this case, we would allow pipelines to see the pipeline name that we were running. And the import ID was kind of how we segmented the data. So, it would be, you know, year uh month, day, hour. And we were running constantly running bulk jobs to load to extract and load data over these uh what we called import IDs. And then, uh those were available as um a mixin variable through the Medium job trait. Um Oh, sorry. Uh so, they were actually available through the platform elements, in this case, the Redshift loader and uh to the individual pipelines
All right. So, that was kind of the setup that we had um going into this, but we realized that there were a couple of problems as um our data team got fewer and fewer and we were having to push more and more of this work out to individual product teams. Um For one, there is actually very, very little testing done, or automated testing, I should say, done with these pipelines. Because of the mixin structure, uh we basically required you to have a database connection to run any sort of tests, which means uh when people were writing, they're having to do a lot of manual testing, a lot of manual verification, which was very cumbersome and slow. Um Because uh our product engineers weren't familiar with Spark or Scala in some cases, they were doing they were constantly reinventing the wheel. They were looking at what code already existed. They were copying and pasting huge swaths of code. Um they didn't really have a sense of how to tune things or what things meant
They just wanted to get it to work. Uh so, they'd get it to work, they'd push it up, they'd do some manual testing, and then that was good enough. Uh so, it meant our code started to get like really, really ugly and unmaintainable. Yeah, like I said, we were copying and pasting paradigms all over the place. Um and like, the platform team kind of realized it sort of lost control of our of our data platform. Um In terms of configuration, um there's just a bunch of stuff that we could infer based on the environment that we were pushing data to. So, we have three main environments at Medium, a dev environment, staging, and production. And so, we were making um these product engineers go through a lot of these configuration steps and provide all this configuration data that they really didn't need to uh because we we realized we could just infer it from um the extraction loading environments
Um and in some cases, like in this S3 bucket case, it wasn't really clear what we were configuring. Are we configuring this Redshift loader mixin or we configuring this Redshift connection mixin? Um it was just very very confusing how everything fit together. And then in terms of the the environment variables, specifically with the import ID, there's no type safety. So um there's like that we're this is all string-based, so we're like kind of awkwardly using uh these things as strings all over the place, uh which didn't work and we're having to wrap it a lot to get information out of the import IDs. Um and of course when you use environment variables that makes testing very very cumbersome as well. So those are kind of the problems that we were encountering. And so we sort of set out to rethink how the data platform could look so such that product engineers at Medium could write their own pipelines and have them reasonably tested and sort of maintain control over the important pieces of data extraction and data loading at Medium. Um so what we did is we reimagined sort of the structure of the ETL job
And of course just the acronym itself ETL provides the natural road map to do that, right? There's data extraction, data transformation, and data loading. So what we decided is that the data platform would concentrate on data extraction and data loading as platform-specific pieces. And all we wanted the product engineers at Medium to do was concentrate on the data transformation which would support whatever product initiative that they had. Um so this is kind of how we laid it out to them. We will provide all the services for extraction and loading. You guys just concentrate on the transformation. These will be services that you can consume. All right
So and uh this is sort of our vision of the new the new world. Uh this is our cat owner matching job rewritten. Um so we had to maintain this application model cuz we didn't want to rewrite um all the the in that 10,000-ft view slide, we didn't want to have to rewrite everything around that. Um so we kind of had to maintain this one application one job paradigm. So we did that by splitting out um the actual job from what we called the job runner. So the job runner carried the actual main um or main method that the normal platform could execute, but we split out the data transformation piece into its own class, which means we can instantiate on its own, we can test it on its own. And um we can run dependency injection. So we decided that we're going to sort of throw away this idea of inheriting all the utilities that you needed for extraction and loading and we're going to move into a composition through uh injected services
Um we decided that we were going to redo how we configured these jobs. So um instead of there being sort of this uh this unclear link between how you're configuring things and what the job needed and where all the configuration lived, we wanted to make it very very clear. Um for example, this cat ownership matching job was reading data from S3 and pushing data into Redshift. So we want to make that very very clear and uh so that the product engineers at Medium could understand kind of what was going on and what they were programming against. Um and a nice thing about that was we kind of got this this type safety uh which I will show you in a little bit. Um but you can kind of see uh because we're using polymorphism here, uh we can push the instance of the cat ownership matching job directly into our um extract S3 and directly into our data loader as the main configuration uh object, which was kind of cool. Um and then of course uh we worked hard on developing uh really clear, easy-to-understand contracts for our product engineers. Uh so that when you looked at dataextractor.extractS3, they knew exactly what was going to happen uh when they looked at dataloader.loadRDD, uh they knew exactly what was going to happen and it became much more obvious how to uh use these tools
Um so here's kind of a look behind the scenes at uh our runner trait. Um we uh we set up this uh implicit Spark session uh that can be used throughout the platform. Um Some people don't like this this um feature of Scala, but I actually kind of like it. Um I and you'll see in several of the example code um that we use this all over the place and it just means that we don't have to pass this Spark session around, which I think was um very useful. Um we used a library called AppBuilder, um which is a microservices library built for Java and Scala. Uh it's open source written by yours truly, so there's a little bit of bias there. Um but uh what it allows us to do is uh set up uh package scans for new services to add to the dependency graph. And I'll show you in a couple of slides um why that was important
And uh finally we added the actual job class that we configure in the job runner. We add that to the dependency graph, so all our nice platform services get injected into uh the pipeline job class and uh they can be used by our engineers. Um so like I kind of mentioned before, we also reimagined how we configure these jobs um through polymorphism. So the source configuration would be attached to the data pipeline and we just make that itself a data source that can be passed directly into the data loader. Or sorry, I messed up these slides. The that's supposed to say the data extractor. I got these backwards. Um and then of course the uh the destination configuration gets added to the pipeline, which itself is a data destination object and that's pumped through the data loader in this case
Um what was cool about this is uh it meant that we could use these pipeline objects as actual connection configuration objects, which meant we could pass them to other services. Um and there's an example of this when we started to get into the ML platform stuff, uh this became extremely useful. And it was kind of neat because we sort of came up with this polymorphism idea independent of wanting to do the ML use case and we when we got to the ML side, it was just all there and it worked uh beautifully. Um so this is how we started configuring jobs in this new world. Um we started using uh convention over configuration. So you can see here um the the cat ownership matching job uses the S3 data source and the Redshift data destination traits um but but uh there's this source environment and destination environment. So we just ask our engineers, tell us the environment product, staging, or development that you want to extract data from, tell us the environment dev, staging, prod that you want to load data to and we're going to infer a bunch of configuration on your behalf without you even having to think about it. Um and then of course just some good programming, we introduced these hierarchical names uh so that it would be very clear how we're linking all the configuration to um this code
Uh so again looking at this example, what did this kind of look like? Um we still supported the job level configuration values like I talked about before. However, um we just required our product engineers to tell us the environments like I mentioned, we did a bunch of inference there. And then it became very very obvious where uh specific uh data extraction and data loading configuration mapped to. Um so this cut way down on the amount of configuration we had to produce manually for these jobs and it just helped speed the process up. It helped provide clarity in the the whole platform for the product engineers. Um in terms of environmental variables, uh because we're using dependency we the job runner would read uh the information from the environment variables and then we would bind that directly into our dependency graph so that anytime anyone wanted to write a pipeline or a new service, they would have um that configuration easily bindable into or easily injectable into their pipeline or service, whatever the case may be. And what's great is we were able to provide some type safety specifically around this import ID, um which meant we could do a bunch of things like extract timestamps. Um we could build table names based on it um that were that were safe
Uh we started doing a whole bunch of stuff for that. Cool. So, um that was kind of our vision and that's sort of the world we moved to. Um but in terms of extending and testing new data platform, uh we did a lot of thinking about that, too. Um so, we decided that we wanted to have a plugin style architecture, which would make it very very easy to add new sources and destinations to this platform. And so, this is uh a pretty complete list of everything that we've built today. Um so, there's even some some cool things like uh if product science dumps things into just a Google Doc, uh we can write a pipeline against that, uh specifically a Google spreadsheet. Uh we can write a pipeline against that um without having to like port the data to a database
Um And uh the other kind of interesting one that people might wonder about is the SQS plugin on the loader side. Um we wrote that to support our curation process, so we run a curation over a priority queue um in SQS, and so we built um a data loader plugin as part of this platform to do that. And so, if you multiply this out, there are actually 35 point-to-point combinations that we could write. Of course, we don't use them all, but um but you could if you wanted to. Um So, this is looking at our the data loader service that we created uh with the dependency injection. Uh of course, we defined a trait to represent our data loader. Um and we bound an implementing class to that. And uh then we wrote all these plugins and just injected them right into uh the loader class
Or the you know, the the data loader implementation. Um And then in terms of the plugins, it was a kind of a similar pattern. Uh we defined uh these plugins and you can see that um the plugins take a destination, which is typed. And so, that goes back to our cat matching pipeline, where we could pass the actual pipeline job itself into the data loader and the configuration would be available for our loader to to uh to use, our loader plugin in this case. Um so, this is the beginning of our Redshift implementation. Um we bind it as a singleton within our dependency graph. And uh you can see that uh the environmental variable configuration is available to this um very very easily. Uh and you can kind of see like I don't have it listed out here, but you can kind of see how we started breaking these things out
There's a Redshift connection factory, which itself is a singleton bound into the dependency graph. Um which makes things very very easy. Um and the the data extractor service is defined in a very similar way. Uh one of the differences is we kind of broke out the extraction methods, uh each one, so extract Dynamo, extract MySQL, etc., etc. Whereas the with the data loader, we sort of decided that we wanted to do it all in one one go. Like once they produce their data that they want to load, they could just list out the various destinations, um which was done in the config file, and that was kind of all done by the platform in one one go. So, in terms of testing, uh we sort of did another reimagining. Well, we took how we had reimagined this ETL job, which quite frankly wasn't much of a reimagining
And uh you know, if you can say that extractor and loader are platform pieces, we can sort of divide and conquer this problem. And so, we produced what we called the mocking data extractor, which would send uh mocked data frames into the transformation code. The transformation code would run, run the load, and then on the load side, we wrote what we called the asserting data loader, which would just assert all the results of the data that we had transformed. And kind of what this means is that the end-to-end were writing were focused very very much on the data transformation piece of this pipeline. Which was cool because that was the piece that the product engineers really really needed to test. They didn't so much care how the extraction or loading happened. They just They wanted to focus on the transformation code that they had just written. Um so, yeah, like I said, we're isolating the uh the transformation piece of this puzzle
Uh so, here's a a look at uh our uh sort of a abbreviated look at our mocking extractor. Um So, uh within the tests, I have an example in a in a later slide. Within the tests, the test writer would just uh tell us what data frames do you want to serve up when the job asks for them? And then we would sort of break we uh broke up all the uh all these mocks by a a mock data source. And the framework is actually the testing framework in this case is actually smart enough to match the data source with the uh data source being used by the pipeline itself. So, for example, if the pipeline requested uh data from Redshift, uh but the next mock was attached to Dynamo data, the testing framework would sort of catch that and say uh you know, throw an error and say uh you're not providing the the Dynamo data that the platform is expecting. Um uh similar for data loading, we wrote this asserting data loader, which provided the ability for our product engineers to write asserting callbacks. Um Which would simply assert uh run asserts on the RDDs uh that we got as they were being loaded from the pipeline job. Um and uh and finally, we wrote uh what we called the I don't know why we called it this
We called it the real Spark service injector, which was uh um this was a uh a wrapper around um around the the Juice injector. This whole thing is is built on Juice at the at the core of it. And uh what it allowed us to do is create this uh this uh the same dependency graph, except we bound our mock services into the graph. And then there's this little feature in the app builder library that allows you to override um the normal injected services with your mock services. And what that means is um we set up this injector, we would extract the the job from the dependency graph, uh and then we would just run the job because we would know the dependency graph held the mocking data extractors and the asserting data loaders in it. Um So, yeah, here's an example of an end-to-end test uh with our mocking our mocked out data frames, our asserts for our expected results. And then we would just dig out our job class, which we know we had put into the dependency graph, and run it. Um And so, this is kind of what we found when we um when we put out this new framework
First and foremost, we got much more and better testing around our pipelines. Um and uh number two, like I said, we got much tighter control over the platform elements uh themselves. And I put in an example uh in the Spark world, they're very concerned, especially around J- JDBC, about overloading the JDBC um or sorry, overloading the the uh uh database with JDBC connections if you have too many partitions in your data frame. But because we had isolated the data loader, um and that was being owned by the data platforms team, we were able to to solve that problem and ensure that we didn't do that without the product engineers having to worry about any of it. Um We got more engagement across our teams. I think we had uh three product teams actually actively writing pipelines. Um and at one point, one of one of them came over to me and thanked me and said uh you know, Spark programming is fun again. Instead of copying and pasting these large swaths of code, they were using this very easy to understand injection framework with these very easy to understand contracts
Um And in one case, we even got non-programmers into the act. Uh our product science had Gustav here. He He himself wrote a pipeline, which was pretty cool. Um so it kind of validated our our choices here a little bit. Um so just sort of a couple of takeaways from this whole experience uh that I'd like to share with you. Um convention over configuration. And by the way, I I don't think any of these are specific to this particular project, but it's sort of things that we realized after the fact. Uh convention over configuration was very important, trying to limit the amount of configuration these engineers were having to do
Uh certainly composition over inheritance uh because of the ability to write tests uh against the code being written. Uh dependency injection was sort of the implementation of that composition. Um uh by writing the testing framework and shipping it along with the new platform, we were able to enable our product engineers to do the right thing and write tests. And of course, we had written example tests for them to follow through and understand too. Um and finally, I don't think this is a surprise to anyone in this room, but developers really like patterns and good names. So the better you can name things and the more the the more patterns that you can show them that are easy to understand, um the more successful it was. Anyway, uh with that, I think I'm pretty close to time. Here's some links uh that I threw together
Uh like I said, I gave a similar talk at a Scala meetup at our offices at Medium, and so that's on YouTube. And Alexy actually interviewed me for that uh talk. Uh so that's also on YouTube. Um and I threw a couple of links in for GitHub for the some of the libraries we were using. Uh a word of caution on this app builder library, it's still kind of in beta. Um it's just me and and one other friend doing it. Uh so it's a little light on documentation, uh but certainly if people are are interested in that, um send me an email uh because you know, I need to I need to convince my wife that I need time to work on these things. So if people are emailing me, I'll be like, "Yeah, see, look." Um anyway, so and I will close with a picture of me and my adoring cat
THANK YOU, SIR. UH WE'VE STILL GOT A COMFORTABLE AMOUNT OF TIME BEFORE LUNCH. Uh so there's plenty of time for questions. Or I'm happy to I'm happy to stay after if people want to come up and ask questions. I have a question. Sure. Uh for the data structures, you have many like Dynamo Yep. spreadsheet, etc
You transform the data in a way I call representation, and then you have a single implementation and then you would bring it over to your I think data frame. Yeah. So that's that's a good question. Um so what we did is um we followed what I believe is called the the robustness principle, which is um be liberal in the in the inputs that you allow, but be strict in the output that you send. So uh in this case uh with our extractor, we had a very loose definition of what you could extract, in this case a data frame. And for our loaders, we required you to be very strict in what you loaded. So we used um we would require you to write a RDD specifically around a proto buff. So your your transformation code would actually take in a bunch of unstructured data
We would allow all the data coming into the pipeline to be unstructured, but then through the transformation process, we would force you to structure that data into a proto buff and create an RDD, which you could then load, which allowed us to sort of maintain some semblance of sanity around like table creation and everything. So that's that's one of the things that we were thinking about in that. I've got a question. It's more of a provi- a provocation than a question. Your your title was uh uh big data for the masses. Right. And the approach is was uh to make it easier for developers. Right
Um there are a lot of people like uh like Gustav and on your slide um who who can't aren't developers and can't really code to save their lives. Sure. Uh is there uh more of this you can see where where people are either writing much shorter code snippets that get compiled into uh a block constructed kind of in your framework or even uh looking at sort of what Hive does, you know, give them the ability to write SQL or other kind of data specification and then compile that into chunks here. Yeah, no, like uh that's a that's an excellent question. Like this was sort of like maybe a half step along this journey, cuz we were we were trying to push this out to people who were engineers. They just weren't as familiar with the framework or Scala in this case. But I think if we were to keep going reasonably, we'd have to pursue the some of the things that you're suggesting. Like our product science team isn't they're they're not going to invest in like yeah, in Scala
And And while Gustav did for in one particular case, and that's great, that's not a sustainable uh you know, a sustainable model. And what they really know is SQL. So Okay. um So yeah, I think the next step is going to be something like what you suggest, which is like give us the SQL and we'll load that into the platform wrapped around all these services and we'll execute. you'll be like say where the data's coming from and where it's going to Exactly. Right. So so it would be like it would be maybe a almost meta programming language, but it'd be but it'd be much simpler, right? Like much much easier for a non-developer to understand. I mean, you've removed so much boilerplate and so much cruft
But uh for these people, it seems like we need to really Yeah, even more, yeah. I agree. I agree with that. I agree with that. Did you mention the many folks who don't know how to write code for scalability So did you try to address that also? Uh sorry, can you So did you at all address folks who won't know how to write scalable performance code or to abstract them out? Yeah, uh so we were we were doing some of that, like I mentioned with the JDBC connection example, uh that wouldn't be something that people would really understand. Um so by by abstracting away the the loading abstracting away the loading and extraction, we were able to sort of process some of that. Uh I don't have a good answer for you in terms of like within the actual transformation code itself, it was a little bit of a wild wild west, because we just didn't have the resources to review everyone's code. And so some of that we were trying to make up on the operations side, like we would spin up just larger clusters if we needed to do that to run people's code
And we sort of monitor the the cluster size and how much money we're spending and try to deep dive into particular areas from an operations side, rather than like from a pure coding side. Yeah. Yep. Um how did your the blog I mean, people who are writing these translation pipelines, what sorts of test running were they writing? What sorts of framework were they using? Yeah, so they were they were writing tests similar to this one I had on the slide. They were writing And actually, there were kind of two there were two sort of varieties of tests. Um they would write unit tests on Yeah, maybe Yeah. Um