Scale By The Bay 2019: Nick Shrock, Dagster: a Framework for Data Processing Applications
Recording: Scale By The Bay 2019: Nick Shrock, Dagster: a Framework for Data Processing Applications
[Music] so with that all being said my name is Nick Shrock and I'm the founder of a company called elemental that is new and we are the stewards of an open-source project called daxter and here to talk about that today so what is daxter it's supposed to say workflows for data applications but I guess that's even more concise and pithy but but we're effectively are we're an open source Python API and if you're familiar with a system like air flow which is used to orchestrate a dag of compute right how many of you are familiar with air flow example okay so most of the room we are you know at first blush we look quite similar to air flow but as I think if you go through the presentation you'll see how it diverges from that system substantially let's see how this formats here so you know you hear all these terms ETL Elte ml pipelines data processing etc etc and one of the things that I find strange about this ecosystem is that in my opinion and in our opinion all of these terms effectively comprise they're all the same thing right if you look at the ml pipeline actually if the previous presentation all these great slides about how you know very little of the work of an ml pipeline is actually you know the actual training right all the work is all the stuff leading up to it which is effectively an ETL pipeline and in the modern world the term ETL doesn't even make any sense because what I order the term ETL come from right came from way back in the day when you had one transactional system then one transform step and then you'd load it into a data warehouse when people use the term ETL today it looks nothing like that at all it's a very complicated graph going through multiple different systems with multiple stages of materialization and it's incredibly complicated and there's tons of different people involved right it looks nothing like these old ETL systems so all this terminology feels very outdated and then you have this discussion of like oh is it extract transform load or extract load transform I can think of a like it's hard to think of a more meaningless discussion because this is all just like a caching decision right like we're engineers like you're not--' right that's all that's all that discussion is so what we think of these are instances of what we like to call data applications so how do i define a data application how do we define it there are graphs of functional computations that producing consume data assets and I mean functional computations are kind of a big sense meaning that a spark job is a functional computation right a data warehouse transform job is a functional computation if you system if you use a system like paper mill a jupiter notebook is a functional computation and what a data application is is a dag that assemble these together in order to orchestrate some something meaningful so what are the problems that we see in this ecosystem one is I think the developer lifecycle is in development life so it was fundamentally broken in this system so my background is I was a facebook engineer from 2009 2000-2007 teen and while I was there I built abstractions in infrastructure for our product developers and you know so the the thing I'm most known for is creating a technology called graph QL and effectively the team next to me was hey thanks and the team next to me created react right so I don't anything to do with react I call myself react adjacent and you know when you go from that world you know I was like present at the creation of the full hipster stack the if you go from that world old to this world and you see someone working in this you just it's like appalling like kind of the developer workflow the lack of testing the lack of aesthetics there's all these individual brilliant systems that have achieved massive technical achievements in terms of being a little scale out like big data is solved it's amazing you know but if you look at the developer lifestyle it's broken we mean that one and comparing this to airflow say you don't use the same tools in production that you do a local development like local develop with air flows not thing and testing is also not really a thing like testing is really difficult in these systems and so frequently people can only test these things of production these systems are fractured and poorly understood meaning that they're typically composed of a different bunch of different technologies a bunch of different type systems a bunch of different metadata systems and as the data flows from a data Lake my favorite term know the analogy betrays what a bad idea it is you pour all the data into a lake and then you can't organize it it's perfect it's the perfect term so you know as the data flow from these different processing systems right you effectively lose all context of the data and if you coming up to the system are inspecting this graph it's actually very very difficult to understand what's going on right and these systems are typically authored by a bunch of different personas in this ecosystem data scientists state engineers normal engineers all participating they're using different tools there's no good integration layer to combine them and then there's just tons of systemic repetition and this is kind of difficult to express but reuse is very difficult in those systems and also you just have this feeling especially we go to a conference like this that everyone is rebuilding the same infrastructure over and over and over again and you know that's kind of these are the problems we see and so what do we think are kind of the values that have we have that it will help kind of move the needle all these issues one you know predictably that we think that the dag the graph of computations should be participating in the full developer lifecycle I mean that you're modeling this this app and structuring it into computations that should be how you work typically right how you work today is you're reading your individual job and then only in the last second you plop it in this dag that will castration through the compute and then you kind of like hope and pray it works that doesn't make any sense when you step back and think about it you should also be testing these graphs as you go along to most of these workflow engines kind of conceptualize these things in terms of a dag of tasks right meaning that they only encode their execution dependencies tasks a then tasks B and C then testy they're not modeled as functions we believe in functional data engineering meaning that these tasks should Express and give guarantees about what their inputs are and what their outputs are and then that embracing this multi-tool in persona world I don't think the entire world is going to converge on flink I don't think the entire world is gonna converge on spark I don't think the entire world is gonna convince converge on one computational runtime the heterogeneous world it's gonna continue to be heterogeneous you have to embrace that and figure out a way to have an integration layer that incorporates this multi-tool imper sona world so this is one of my slides that got totally messed but the other heat another huge problem is the siloing aspect of this typically in this multi persona applications in a lot of works there's a strict dividing line between data science and data engineering I mean like I'm the data scientist I work in this tool set and I'm creating models and if I need data I have to go over here and talk to the data engineer this is what causes what some people call data breadlines and that's not a particularly good situation where you want to get to is this kind of state where there's some sort of overlap between them enabled by tools and abstractions this state of the world action reminds me I was at Microsoft 15 years ago and back then there is a split or there's software test engineers software development engineers the developers write the code and be total piles of crap they throw it over the wall the testers and lo of a whole to beat bugs and this is crazy feedback cycle right looking back on it it's actually kind of crazy now what a developers do we have unit test framework we write tests we're accountable for things that's the way the world should work in the same way data scientists and and other what I'll call data domain experts should be responsible more for the end-to-end capabilities rather than having these like multi you know instead of having the capability owned by a bunch of different people where this kind of a dysfunctional relationship so the claim is that the tools that we have are not equal to the complexity that we face that's the claim and this is like what DAGs end up looking like in these systems crazy it falls apart so quickly it's always amazing we just see these type of tags over and over again and then you know you know I don't want to pick on airflo too much because like I know I know the creator of it is good and like that's that the system has served us well but it's also time to kind of and they were trying to address this within the context a lot of these things that we're saying in terms of functional data engineering etc are trying to be addressed when the context of that system but it has like fundamental properties I think are difficult to recover from so what we want is a you know what I like to call the full entire development lifecycle so without further ado I'm just gonna hop in and just start demoing this thing and we'll see how far we get here there we good to go there all right so let's talk about develop here let's just [Music] all right so let's go to like hello world here and I'll make that bigger one second here all right so this is like computer computer science blinking light demo right we're just like loading in some pandas dataframe and doing some basic math right so we have a few nodes of computation we're doing some math so each solid as we call it this is kind of the unit of compute in the system if you look at the graph every single node is called the solid alright and each one of these things this is a function right declares the inputs the cores two outputs we have our own type system we'll explain why we need that and then this is just doing computation right and these are stringing together doing some computation in the end we kind of do you know combine two things and then this is all strung together using this pipeline syntax so this is this DSL right that this doesn't actually do the compute this actually just builds a dependency graph right and because it's functions we can just call them like functions functions are amazing they like compose you can pass things to them this is this whole abstraction that works really well so the moment we have the computations encoded in this format we can load it into this tool that we call daggit all right yeah all right so now we can go and we can check out what this pipeline looks like right not surprisingly it's a dag viewer but I think the big distinction here as opposed to like dag viewers you've seen the past this has a lot richer metadata associated with it so it's not just you know it's not just circles and lines right you can actually inspect this thing and at least begin to understand what's going on go back here so now we want to execute it so the one of the distinctions here between other systems and kind of the core difference between other workflow systems is that our DAGs are parameterised right you see here like the first act takes in a path and spits out a panda's data frame now the issue with that is that it's not directly executable you need to provide the parameters for this in order for it to successfully execute and so we made the explicit decision to separate the structure of the graph from config which can get complicated so we have a strongly typed config system and help so you kind of construct these computations for you so we go to what we call the execution pane here let's imagine I hadn't written this now we have this kind of auto completing auto completing config editor here where it gives you like nice error messages you know because what we find it's fine what we found is that so often these pipelines are just super complicated functions and the end of it is taking like like balls of huge amounts of config and it actually ends up being really complicated to manage them and that's usually one of the things that prevent them from being testable because in order that you have a testable thing you generally have to kind of parameterize in complicated ways so now instead here because this thing requires one input at path we can say hey I need the parameter is this thing comes from this input this thing's really fun to use actually especially when the config gets complicated and now I need to say right and the you know this doesn't just provide value in this fancy-pants editor it gives you super clear error messages so even when you're doing unit tests and whatnot you can actually like debug this stuff super super quickly all right now we execute this thing and now we have this like live updating executor right where all your logs are right here and this thing is really fun to use actually and so this like serves as a local development environment you'll also notice that our logs are not just raw strings I mean you can just log a lot rostering if you want but I like to call normal logging just developers thinking aloud right and that's not nearly as useful as logging semantic information that is parsable by tools in a runtime and so we'll go into how we do that but like for example one of the values of our system is that it does a type check which is the function but you can also yield meta information about the value that passed through it install user pug bulb and this in this example we do something simple where you know let's say like oh I want to click on this mulch solid okay we just filter down the events to that and now we can see like hey at this input it has two row counts and wick we can see what columns that have and once at the output have these columns cetera et cetera so this is really useful for understandability and debugging here so let's imagine I'm developing this about zoomed in stuff but let's imagine I'm developing this and I screwed up right so there's dag of compute right and I can't spell either all right let's go back into this thing how this is too small arena all right let me part in the CSS books so now we go in here is that big enough alright so now let's react to keep this thing so now we're gonna get an error right and so this was a dag so we just displays in linear order it's easier absorb but this one failed therefore this one was skipped you know you can go in here and like quickly get your stack trace on one now all this stuff works the same in prod two which is sweet so let's one of the things it's nice is because we're aware of the inputs and outputs and we can take responsibility for shuffling the intermediate values from computation to computation we can also do incremental computation which is super nice so here we can say like actually what I want to do is try to retry this starting from where the steps have failed and skip the ones that succeeded but shuffle the old values to the new run right so boom now we do that now it's only going to try to execute two things right specifically visit specifically these two ones alright let's go back to the runs go back here now let's say I go back and fix this now here we try this and lo and behold it's fixed right see how this new the buggy flow yeah what's up correct yeah so one of the things that we shuffle intermediates and it's pluggable two so in a cluster old store n'est three or an object store but locally we're doing on the filesystem right okay here so we have all these fun development tools here now let's talk about testing right I showed you the UI code I'm not gonna explain this slide all I'm saying is that oh god this got messed so to ignore this code no the whole point is that you can actually unit test these pipelines like there's a Python API where you can parameterize them and execute them we can plug the environment so that in a unit test environment we're not persisting the runs meta information to a database just like an ephemeral thing you know contrast is the airflow we're like in order to what run air flow locally like you have to like be touching the database the entire time it's actually really difficult to run tests in isolation so productionize see how bad this is so this is probably not visible really the point of this though is that the the goal of this system is to make it so the people writing domain logic are effectively just just running their business logic and then these units of computations these functions are aquaria Balai an api meaning they're metadata and operable by an API so that you can write an entire suite of devops tools around this so all this is doing is actually like mooing up an ec2 instance for you mooing up a database on your behalf and then uploading this thing to AWS but this is just the beginning of our journey to really operationalize and make this kind of a programmable infrastructure to deal with all your DevOps issues I'm gonna skip this for now and then we also provide a scheduler so you can define your schedules in Python you know and I should probably not picking on airflow too much but you know what airflow does every five seconds it evaluates a file and constructs every single dag Minh decides to schedule it or not we take a different approach where you know we use we use cron cron works it's worked for I don't know since before I was born and all we do is say we have cron point to a function that says construct this pipeline provide this config and then run it and then we also provide a Python function so you can filter out events I'm like English bank holidays or whatever and it's a very simple reliable system it isolates the pipeline construction from other pipeline constructions it ends up being just a much more reliable way to structure your computations alright let's talk about testability for a second and what do I mean by it so oh god this got totally messed up so the you know this is what we see in the status quo today we're very little of these computations can be tested in a unit testing context and then maybe there's some integrations have seen but really it is kind of like pushing pray usually maybe of a staging environment if you know your DevOps people are really good and you really take time to do that so you know the one thing I think that happens at testability people come in here and this act my triggers and people this audience but you know they often come in or they're way too ideological about testing and you start getting lectures about mocks and stubs mark stub fake like this please can we move on to something else but you know the idea here is that we want to take traditional testing techniques and apply it to this domain and this domain is hard right you don't have control of your inputs typically you're just getting it from somewhere your business logic is typically encoded in centralized services that are impossible to mock locally right like spark you can't mock like that would be an entire company like spark like you don't want to do that but there are tons of stuff in these systems that can be caught by unit test and they should be caught by unit tests right dag structures configuration parsing there's some business logic which can it's just a it's just a question of how to restructure your code and but the corollary that is what I explained you before is that don't attempt to unit test for errors that you can't and you know striking this balance is difficult but I think it is critical to moving the needle on these systems and like what the attainable goal here is to make it so you can kind of move a bunch of the testing upstream and but this shifting the curve to be like this represents like an order of magnitude increase in profit in like productivity because the more errors you can catch up the line the faster your feedback cycles but you're always going to have some testing and production as an example because some some errors in this domain cannot be caught until production for instance if the inputs change underneath your feet so like part of this is that you have to have testing the production to test your data quality you think of this like a factory and like in any stage of a factory there's QA all right I have five minutes so alright so airflow as I mentioned before doesn't really address develop and test only like production i.g and scheduling these things so what do you need to actually make these systems testable one is the actual system environment needs to be pluggable meaning that the primitives provided by the system like if you have a dependency and production on s3 to do intermediates shuffling and you have to figure out a way to make it to its executable on your laptop and the csv pipeline you also want to provide an abstraction for pluggable user environments meaning environmental context that the system cannot know about that only the user in their particular domain can know about and then you also need parameterize computations I know that was a lot but and I'm kind of jumping around here so let's talk about metadata so as I said this is our system and just to contrast what this dag looks like in comparison to here so it's tag here is that this has like you know descriptions you can kind of get a sense of what's going on we actually have a first class compositing abstraction because meaning that is the function so functions can call the functions this whole thing you know contrast that to DAGs and airflow or cube flow or any of these other systems and there's generally like no meaningful metadata there right it's just like boxes and arrows right all it's expressing is the how of the computation in what order do you compute rather than like what that computation is doing it's kind of the difference between the systems it's like this is why we like calling an application layer yeah so yeah the the outputs are optional so what you can do is that you can think of think of the outputs like an event and like sometimes an event doesn't fire and then that tire downstream dag is skipped so we do have some we do have some dynamic control flow constructs implemented through that mechanism but it's not fully turing-complete or something all right oh you're right I need to switch to my other deck now yeah this is a whole whole mess so just like summing this up quickly you know and this is all compressive weird right that's fine so you know typical data applications here are cross multiple system barriers and then frequently have multiple different constituencies right so you may have data engineers running in Scala spark job then the analysts if you're lucky are writing sequel statements to do their business logic maybe in a tool like DBT you know DBT here anyone DVT is a great tool everyone check it out and then data scientists maybe pick that up yeah Dexter simply effectively wraps these computations in this unified metadata layer and then if you kind of go full on meaning like you're you're starting to use more the daxter native constructs in your Python code and all the computations are structured as solids so we're building the ecosystem here so there's a center graph which is the center of everything below it is arbitrary data processing tools any storage any language any runtime by me any language like the Python can shell out today right on top of that we have daxter native tools built over an API spoiler alert that API is graph QL but that doesn't matter right we have these nice we have these nice tools on top of it and because in API you can also build your own tools right we provide local development monitoring the fit editor all that stuff right and then also on this tool we have integrations with existing infrastructure tools so this is a pure software abstraction it's meant to be able to be deployed across any arbitrary execution substrate so it's not vertically integrated in Tights a kubernetes like you flow right so you kind of have billions ecosystem here we have arbitrary computations underneath the abstraction that represent different constituencies they all collaborate to build a single graph of compute that's accessible behind an API this is how we build our local chewing doing our local executors but we can also do things like oh this is supposed to be a task but we can compile actually to other task formats so we have like an API format that allows you to in your airflow tag like just call a function which invokes one of our pipelines with a configuration it constructs an airflow tag on your behalf we can also do the same thing with like tasks for example and see where I'm at here so anyway I'll close this out but you know this is an open source project it's on github you can check it out we have a slack you know all that stuff so if you want to if you want to experiment with this let us know sign on and thank you very much [Music]