SBTB 2023: Pete Hunt, Your Pipeline as a Function.

SBTB 2023: Pete Hunt, Your Pipeline as a Function.

Recording: SBTB 2023: Pete Hunt, Your Pipeline as a Function.

I'm not going to talk about AI or or large language models or anything like that so uh I guess sorry in advance um someone said thank you um I actually uh I'm I'm a CEO of a venture back company don't worry you're not going to see any advertisement for our product not even going to mention the API or anything like that um but I got a a compliment from an investor the other day he was like you guys don't have the words AI on your homepage at all like that just projects so much confidence anyway I thought that was pretty nice um all right so hey I'm I'm Pete uh today I'm GNA talk about um your pipeline as a function um just a just a quick snapshot of who I am um I uh started my career at Facebook worked on um reactj s which is a very popular front-end Library the reason why that's relevant to this talk is because at the time we kind of saw that as a way to um backdoor functional programming into the mainstream and like by and large it it worked and we were really pumped up about that um left Facebook started a company called Smite we did um realtime stream processing to you know identify Bad actors on the internet um eventually uh sold that to Twitter Hey pis nice to see you um and spent a couple years there working on a variety of stuff related trust and safety and data um and now I'm at dagster working on dagster which is a Next Generation data orchestrator and that's all you're going to hear about it so what are we going to talk about today um in my opinion there is a deep correspondence between functional programs and data pipelines and I think that this is not necessarily obvious at first glance we'll get into to why that is in a second um and I think if you you adopt functional programming Concepts you can dramatically simplify um how we build and operate data data pipelines now with that said you're not going to learn anything that's like super directly applicable to your day job today um like I said I'm not showing you any new API or thing that you can put into practice immediately uh but this is more just kind of thinking about hey like how can we take these functional programming Concepts that we've known about for like decades and maybe them in a new domain and like can that change the way that we think about how we architect things so um with that uh I'm going to dive in um wanted to to name check Marius Ericson here he wrote this amazing paper called your server as a function uh and I'm very much ripping off a lot of that and applying it to the data domain but um you should go check that paper out it's really awesome um that ended up becoming finagle which is this uh really cool distributed systems framework in Scala so uh in my opinion functional programming has largely revolutionized how we build software so we see this across a bunch of different domains sometimes it's called functional programming other times it's called declarative programming I'm not going to be a super stickler for terms here but you see it on front end with react where people are building they went from this world of manually mutating the Dom uh based on events and network activity to this very declarative uh kind of functional approach for composing uis you see it in terraform where you write this magical configuration file and you run terraform apply and it does some crazy mutable stateful thing under the hood um and converges the desired state with the state that you've described in your config file um you know I'd argue that's a that's an application of of functional programming uh philosophy kubernetes similar thing applied to to container orchestration and even in in the world of data analysis right like SQL you know it's it's got a little bit of a clunky syntax but in many ways you know it resembles a functional programming language so uh I'm not going to I don't really take like an orthodox approach to the the term functional programming and so you know you might have a bunch of different definitions but broadly I think in the real world when when functional programming actually lands with mainstream people it's really about two things the first is it's about minimizing side effects uh specifically like observable side effects um and the second is minimizing a mutable State outside of the function boundary so what I mean by that is like I think if you were to kind of take a look at some some folks that were you know writing JavaScript or C they would maybe do some local mutation within their function they would maybe have a four Loop or they would append to a list or something but you know I think generally best practices these days is to treat any values going into functions and coming out of functions as as uh immutable or at least to highly discourage that mutation uh contrast this with what you generally do when you're building data pipelines now data pipelines are you know you can call them elt or ETL pipelines they're basically moving data from one place to another and then transforming it and then making it available for a business stakeholder they're really all about side effects in mutable state right I mean they are constantly talking over the network to thirdparty apis that are um item potent or not depending on what the that SAS vendor does um and you're sticking data into a data lake or a data warehouse you know like S3 or snowflake or something and that's just a big pile of mutable State and so uh what you often see when you're when you're working on like production data pipelines is they tend to be really difficult to like test locally to understand what's going on um to uh to to run them uh you know in a in a reproducible way on your laptop those sorts of things and it's really because they're they're they're this integration point that touches Network Services and mutable state in a major way so let's see if we can tame some of this complexity uh anybody here ever use DBT at all okay um this is like the hello world uh project from DBT it's called a jaffle shop apparently in Philadelphia or something there's a sandwich called a jaffle uh my dad's from Philly he had never heard of it so I don't know if that's actually true or not but supposedly there's a thing called jaffle this represents like a kind of super simplified e-commerce data platform so we've got these kind of raw data sources which are coming in from uh somewhere on the internet from a source that we don't control then we stage them in our S3 power data Lake and then we uh produce the data products that are used by you know whoever to make business decision so we have a cleaned up customers table and an orders table um this this making sense so far all right we got some nods great cool let's zoom in uh to one part of this data pipeline so if we were to implement this in Python uh we basically have a function that you can call to materialize this data asset this raw payments asset and you know what we're basically doing here is we're like reading from a CSV file uh this could be an API call somewhere or whatever um and then we're staging it or we're copying it into our data Lake and that that means we have basically a consistent snapshot that everybody can work from and we can process you know do do Transformations and processing on it cheaply um couple of notes here like this assumes that the data fits in memory uh we'll talk a little bit about scalability later um but you know there are ways to to to write code like this that don't load all the data in memory so hopefully you can like you know give me a break on on scalability and performance at least for a while um so if you remember that that kind of uh slice of the data pipeline that we're zooming in on there's now a staging step right so we take the raw data we land it in S3 and then we do some normalization and pre-processing and we call that the stage data so what this function does is it calls its Upstream raw payments materializes the raw data then we read it back out of S3 and then we we're using duck DB here uh which is really sweet and everybody's heard of it um but uh if you haven't you should check it out uh but this could be any sort of data transformation this could be in Python this could be like in a data warehouse whatever um and we we run some some data clean up on that data frame and then we store it back in S3 as a as a CSV file as staging payments. CSV and then finally at the end the end data product that we want to produce for our customer um is the orders table and so that depends on the Upstream staging orders and staging payments data um which we call up there to materialize them we read them back out from S3 and then we write them then we write the derived data back to S3 which joins the orders and payments data together and does some pre-processing by the way this code is is like actually on on GitHub and I'll have a link to it at the end if you want to check it out but this is not rocket science this is also not using necessarily like you know cuttingedge open source libraries and and um and Frameworks that you would see in production but like I want I want you to think about like conceptually what's going on like if you're using like airf flow or something conceptually like oftentimes this is what's happening whether that is you know being orchestrated by by some sort of advanced schedule or running in in kubernetes or whatever so there's a lot of issues with what I just popped up there the first is verbosity uh so not only is it just a lot of code but it's a lot of code you have to keep in sync so there is uh S3 magical S3 URLs that are copy pasted throughout the codebase and and you know the previous step depends on the next or the next step depends on the previous step and they have to agree on the S3 URL uh testing is really hard it depends on a production S3 uh and you have to go back and forth to that which is going to be slow it's going to be be unreliable and if multiple people are running tests at the same time they could potentially overwrite each other's data which would be a huge bummer um the third is efficiency so if uh if if multiple Downstream data assets depend on the same Upstream with the way that we've currently architected it we're going to rematerialize that Upstream data over and over again that can be really expensive and cost management by the way is like a thing that in 2023 especially like everybody is like cares a lot about uh failure recovery you know if part of our pipeline fails we have to rerun the whole thing again that can be expensive that can result in data delays incremental refreshes uh same thing um and then finally uh latency this the way this is architected everything runs serially and that can be really slow so uh when I was um kind of doing some some research for this talk I came across this awesome blog post by the author of The the mill build tool and um he draw drew this really cool comparison between build tools and pure functional programs as well and build tools share a lot of things in common with data pipelines right they are fundamentally taking in artifacts in one side you know combining them together with some some Transformations and they're spinning out artifacts on the other side and they have this reputation for being really really stateful and having tons of side effects and having implicit dependencies on the environment that make them really hard to um to containerize and kind of wrap up in a in a repeatable way and so um you know this is another great blog post you you should check out but um just like build tools have a dag of dependencies uh like the libraries going into their output binaries you know data pipelines have a dag as well um and let's zoom into this subset of the dag where we have you know two dependencies and one output here so if we were to to kind of model this as a make file uh you know we would basically have our Target be orders and then the the dependencies would be staging orders and staging payments um similarly functions take arguments and return values and so we could create uh an orders function that calls the staging orders uh dependency and the staging payments dependency runs some sort of arbitrary transformation on them and then Returns the output right this is this is basically just a visual representation of a function call and so let's go back to this subset of our uh jaffles shop project and try to express it as a functional program this is our code before that would ingest the raw payments data and we can make a really simple transformation here to turn it into um into a a pure function uh which is to just get rid of all the code related to S3 now I know there are people out there that are like he just called this thing a pure function it's not a pure function and I know it's this is actually going over the network and that the data at that URL could change um I I get that but we're taking baby steps here we're we're going to we're going to make it more pure over time uh but let's just imagine for a second that that data does not change um I also want to maybe get on my um my Soap Box a little bit about what Purity and item potency really means it's really about the Coller being able to observe the side effect um and so for the purposes of this we'll just assume that that doesn't change but we'll get to get to what happens if it does later so similarly with this staging payments uh derived data it has raw payments Upstream of it we can basically get rid of a lot of the S3 code and we can say all right we'll just call my Upstream function it returns a data frame with all the data in it and then I'll just compute my derived data from it and return that as a data frame and so on and so forth with this orders um this this kind of derived data at the end the the end data product we call our two upstreams we derive We join them together with duck DV and we return the data frame right like this this is not rocket science right this is like functions calling each other it's like CS 101 right um but by simplifying this stuff even though um you know we've we've removed some functionality the code got a lot cleaner so there's there's no more mention of S3 there's there's very little verbosity the vast majority of lines of code that you just saw are like kind of directly tied to the business logic um and it's actually really easy to test now I can just mock out a couple of functions and um I can run that whole pipeline in process like basically instantly on test data and I don't have to rely on any sort of external systems however there's there's a bunch of issues with it still it's still not efficient it still lacks failure recovery incremental refreshes uh still has problems with latency and most importantly it doesn't actually do anything like the whole point of this data pipeline was to like move data from one place into our data Lake and then compute some derived data and like make it a available to other people and this thing doesn't do that it just returns in memory data frames so we can actually solve this through a tried and true uh functional programming technique called memoization anybody ever hear of me memoization awesome okay cool my people so if we take this subset of the graph right with a naive implementation that I had kind of laid out so far whenever we run this pipeline the staging orders data will get computed twice right because we would compute customers and we would compute orders and both of those would call the staging orders function and compute it twice and that could be expensive um we can just treat this like any other programming problem or any other functional programming problem and we can apply uh memorization to the problem so this is built right into python you can import the the cache decorator apply it to this uh this function and now we've solved that problem right so now we can run this data pipeline over and over again and only the data that hasn't been computed yet will be um will be recomputed so we just with like one line of code I guess two uh We've saved a ton of of money uh but this pipeline still doesn't make the data available to S3 but with a small tweak by basically saying hey you know rather than storing this memoized dat data in an inmemory cache what if we like you know had the same contract between the memorization code and the business logic uh but we just had a different memoization backend and we made it a little bit pluggable so um what this basically does is the exact same thing as that funk tools uh cache decorator but stores it serializes the data as a CSV and like puts it in a known location into S3 and so your code ends up becoming um you you basically don't have to change that like beautiful uh very expressive uh focused business logic and just by adding decorators to a couple of functions you can get the capability to like store stuff in S3 and so um again adding a decorator to a function is not like mind-blowing CES stuff but when you think about like building a data pipeline up from the ground and and like what the best practices are in Industry today where like IO is like interleaved in like all of your data processing code um this is we've got all the benefits of kind of like the the low verbosity the testability um of of kind of like a purely functional uh approach but at the same time you know we've got this much more efficient data Pipeline and we can now recover from failure too so um we don't have to reason about too many different failure States at this point like a function can just fail and we can just rerun the whole pipeline from scratch and then because you know we've memorized the success F steps it will just basically pick up where where we left off um so it's a much less surface area to to reason about in terms of failure we've introduced the cash um caches uh create arguably more problems than they solve in many applications I don't know if anybody's ever had to deal with this but it's like I don't know they could be frustrating bugs when you like think you fixed the thing it doesn't get fixed then you realize that there's cashing secretly in there that that um ruined your day so anyway uh We've introduced a cache and worse than that we've introduced like a globally stateful cache that stores this data into S3 and it presents a problem when the data changes over time right we're we're now if the data changes in any of our upstreams now we have to reason about invalidating this cash and that really sucks it's really hard um but it's unfortunately a thing that we have to deal with oftentimes when we are building data pipelines we are running them every hour because the business stakeholder like wants their dashboard to be up to date and and figure out what's going on in their in their business so there's this problem of modeling modeling like mutable data or data changing over time with pure functions like how can we get the benefits of this purely or or uh or mostly Pur uh functional approach with mutable data and the the answer is time partitioning um anybody familiar with like what time partitioning is all right less people that's cool um so I'm going to show you how it works uh it again continuing with the theme here there's not a lot of rocket science here it's it's like taking this thing which reads the source data from the external source and then returns it as a data frame and then the the memorization layer cop cies that into our data warehouse we're going to make a really small refactor here and we're going to add like a parameter that is a date and we've now solved the problem of data changing over time uh it's it's I think it's really cool how these Concepts build on each other so if you're not familiar with with uh how date partitioning Works um in data engineering oftentimes you know you will take uh you'll have this giant Source table and as you ingest new data you it's much more efficient to have daily slices of that table or hourly slices of that table and it's a great way to also take advantage of functional programming techniques um and and being able to handle data processing uh or or processing mutable data over time so like what this basically assumes is that you know at the end of an hour we're going to treat that data as immutable so every hour We're creating new data rather than updating the existing source of Truth and we can just pass this parameter along throughout our whole data pipeline and reuse all the existing infrastructure and take advantage of of data that is changing over time so you know again we can update our derive data the staging payments we just we pass in a date uh string into this function it passes the date string to its upstreams uh so on and so forth and this is our final data product which is which is what the business uses and so um what is cool about uh H oh what what's cool about this is um we actually don't need like a scheduler either we can just call the functions over and over again uh you you wouldn't actually necessarily do this in a real application but like conceptually what you can do is you can just rerun my data pipeline all the time and because the the data pipeline itself is item potent uh we don't need to reason about any sort of like KRON schedule or anything like that we just rely on the memoization and we rely on the the date or time partitioning scheme to uh provide the you know make sure that our data is up to date at the right time so I think this is really cool uh and it solves a lot of problems so uh again we've we've solved the verbosity problem the testability problem the efficiency problem the failure recovery problem and the incremental refreshes problem um this also gives us the ability to do time travel too like if all of your functions understand what the date string means you can say hey actually you know what I'm going to delete all the data in S3 for this particular date range and then I'm just going to rerun the data pipeline um and it can recompute all of that for you without you having to think about writing a custom backfill script like it actually can be pretty uh pretty straightforward to to build time travel or otherwise uh back fills uh using this technique however uh there's one more problem which is that this thing is really slow so our current implementation doesn't parallelize this even though it could right so orders depends on staging payments and staging orders and we should be able to materialize those in parallel um what's cool about Pure or item potent functions is that they can be uh parallelized uh transparently as well so in Python uh to demonstrate this anyway I'm going to show you this um async io. gather uh function uh and two thread this basically if you're not familiar with Python's uh async iio package it's really cool it lets you treat um you know it lets you basically decorate uh or Mark certain functions as asynchronous and then you can uh put those onto a thread pool really easily so what we can do is we can basically take this and like add the word async to it uh this is uh python syntax uh you can see we've added the word async to the function definition we've added the word await to Raw orders what a it does is it basically takes what's called a um a co- routine in other platforms you call a future or promise and it waits for it to result or or to return and then it Returns the result and then we kind of implement our business logic uh as normal and so on and so forth we can actually run these in parallel by uh calling async io. gather here and and running uh these two in in parallel so what gather does is it basically runs these in parallel waits for them to come back unpack the results and then we can kind of you know manipulate the data frame uh as we do so we're done uh again like we've just basically started with this very imperative side effect Laden uh python code and we've done really small refactors with an eye towards functional programming techniques again not totally Orthodox like I've been saying pure when I meant item potent Etc but like we've minimized side effects or at least observable side effects to the user um and we've also minimized mutable State and we've gotten a bunch of great um great great side effects from that side effects from that uh one more thing is that you can use this to to parallelize compute as well and and and really speed things up so if you imagine um I guess I said I wouldn't talk about AIS but but here I am um if you imagine that you have like a corpus of documents and you want to compute a bunch of embeddings on them and parallel and store them somewhere uh you can use um what's called Static partitioning so just like we use timebase partitioning to handle data changing over time we can use a different partitioning scheme to bucket different data and process that in parallel and so I'll show you what I mean so imagine we have this like staging documents data asset it returns a bunch of document IDs um and then uh this is just kind of pseudo code for us to to kind of process um these embeddings uh for uh for a given bucket so the idea here is that we take a big giant pile of documents we bucket it into 10 different buckets and then for only the bucket that that we're processing uh we compute the embeddings for that for that document and so by basically just adding an integer parameter we've made it very easy for us to scale out compute for this data Pipeline and so you know we're starting to run out of time here a little bit so I'm not going to talk through all this code here but it's all in GitHub but the the point here here is that we're just reusing memo we're just using memoization we're just using basic parallelization techniques and we're taking advantage of the fact that these functions are item potent and we're able to get a bunch of great properties out of it so in summary you know functional programming enables easy memorization using memorization plus the right partitioning scheme can elegantly solve like a ton of complex data engineering problems and there's a bunch of great uh resources here um and I I've got also these links on on the GitHub hub for this uh for this project um but you know I would just encourage you you know if you're building something that's Laden with lots of side effects think about like hey you know where can I apply memoization here where can I apply uh this think this this mindset of pure functional programming to the problem and with that I will leave you to it thanks everybody