SBTB23:Stefan Krawczyk, Hamilton:Natively bringing SWE best practices to python data transformations
[Music] hello um yeah my name is Stefan crachek um I'm excited to talk to you today about an open source project I created at stitchfix um and effectively one of the things that I want to leave you with is that you know if you use it you're going to get uh software uh you're going to uh software engineering best practices are going to come uh for free um in terms of you know how I found myself here so I've been in Silicon Valley since 2007 um been at a bunch of places most famously I was at Stitch fix for um six years where I was on the platform team helping over 100 data scientists prototype productionize and then be on call F they plug back into the business uh that's where kind of Hamilton uh created it open sourced it and then I decided to start a company around it uh called dagw Works um so onto the talk um so in terms of you know slightly rhetorical question for you all since you're probably here uh and you probably you know uh value this already but you know why should we care about software engineering principles right uh what would you what would you uh think of you know what would you say to your CEO if they came up to you and and asked you know why should would you care um to me is like the reason to care is that it helps amplify you know scale your kind of human efforts right since generally in a company we're one of the most expensive uh people there and so in which case if we can you know write code in a way that we can move faster right then the business uh will you know uh will get more value out of us right and so this is you know where I see software engineering principles um you know playing into uh business value um so in terms of this talk right I'm just going to give you a story of motivating pain why we created Hamilton going to explain what Hamilton is uh give you a brief mental picture of some general usage uh and then you know talk about you know the native software engineering practices that I think Hamilton kind of gives you for free if you use it uh and then you know summarize things um so context one motivating pain right uh as districs there was a data science team one of the oldest in there that had to provide operational forecasts for the business um uh the business basically had to make uh uh you know decisions based on their numbers so how many warehouse workers should there be how many uh stylist should be staffed you know um they basically were always under the gun to produce numbers and so this meant you know uh you know every week every month they had to adjust to a constantly changing world and ask and answer the questions that exec team might have as to you know what's going to happen with the business so we can know what to do um their flow looked like something like the following right um uh but you know in terms of iteration and getting things out the biggest problem was you know creating this uh featued data frame um uh and so this is what Hamilton helped uh was set out to solve to give you a bit of a grounding in a sense uh for what this the problem was right so I'm just going to walk you through a script so uh this script basically looked something like the following right it it loaded some data from some databases say the the amount of marketing span the number of users signed up uh they then would um you know add add uh columns uh to this table you know features is another way to look at it right maybe this was conditional based on some logic so there would be some branching um then maybe they would have some on line incantations uh and then you know maybe they would also delegate to a bunch of functions to you know add features and uh to this data frame that then they would save and then go for the model um you might be asking to me now what's what's the what's the problem here this code looks pretty normal this is how I write things right uh but picture the passes of time uh different people years going by and invariably with you know data work you're always adding not removing so inherently you're trying to make things more sophisticated and so things get more complex and so if you write code in this way like the the maintenance in update and iteration cycle right for them uh was tough right this code doesn't have a great unit testing or integration testing story where do you put the unit tests you have to think for uh right code in the from the beginning to make it work well integration testing is tough you need to run everything to get something uh and then this impacts your data quality since there's a problem like where did it come from right um and then if you're onboarding someone or bringing someone you right you know this wall of text is kind of intimidating uh maybe there's some special constants or numbers how do you document it uh there isn't kind of a you know a standard way to kind of do it um and then in terms of as the script grew and things start to depend on each other right you wouldn't know that dependency unless you inherently knew and read the script you know top to bottom right um and so uh tracing you know being able to be confident and making a change and not knowing you didn't break anything basically meant you had to understand the you know a lot uh and then invariably with with data work uh you're iterating and you're generally doing small perturbations of what existed before uh when you write code in this kind of a style right uh if you want to reuse some definitions you know the easiest thing for a data scientist to go and cut and paste rather than go Factor so then you have this problem of propagation of definitions and other kind of problems that come about of having you know a lot of uh the same code in different places and so then this impacted the team how quickly it was able to onboard someone uh and then also subsequently how fast they were able to debug something uh if there was an issue um so suffice to say you know I don't think this is this this problem is unique just to stitch fix or to anyone uh or to even to their particular time series forecasting problem uh so make sure I haven't you know put you all to sleep um does anyone relate to these problems or uh you know yeah got a few hands uh would anyone want to inherit code that was written like that no hands good yeah uh independently would you uh in anguish being an anguish if you suddenly had to inherit your colleague's code anyone no yeah yeah um so this is kind of what I think you know some of the requirements we got as to like what we wanted to uh build Hamilton for to help solve right um so what is Hamilton sorry I'm skipped head yeah so what is Hamilton um in one sentence it's a micro orchestration framework for defining data flows we write declarative functions uh I'll get into this but you get a lot of selfing best practices for free uh the point here is that it's also a library so it's very lightweight you can put inall get on uh get on your way uh very quickly uh and if you're bored of me you can go to try hamilton. deev and go and play with Hamilton in your browser right now so let me unpack that sentence so what do I mean by micro orchestration versus uh well the Contra is macro right uh macro orchestration is about you know something like what you use airflow for for example uh is about orchestrating an entire flow end to end where you're you know sequencing different computational tasks one of the other micro orchestration is you know I think handling uh the code within a particular step right so really focusing on one step and so that's kind of where where Hamilton's Roots roots are right focusing on this in Step uh and so then what's the data flow right uh it's just a fancy way of saying how data and computation flow right uh and effectively in computer science terms this means you can express it as a disected aycl graph or dag so this code I showed you before this procedural code is in fact a data flow right and that's because uh you can express it as a direct to day cycl graph right and you can see here that spend is required to compute average threee spend right and then it's also required to compute spend uh standard deviation right and so you can kind of um map this procedural code into a data flow and so that's really what Hamilton is really trying to help orchestrate right is is this kind of code um so what do I mean by declarative functions well effectively uh when you read the function it declares what what you can create what you can get out uh and then it also declares the dependencies that are required for computation uh now you don't run these functions directly uh but there you know the key is that if you read the function you'll understand what it does what it needs um the reason I kind of you know this Paradigm kind of came about was uh you know the problem was that given I mean I'm sure all of you had to debug some you know data frame or table of an output part of the task of that is to understand hey for this column what is the code that you uh you know uh transformed or created it right that's generally pretty difficult so the idea was can we make this as easy as possible so given an output say in a data frame or uh given a column can we make that column what created that column M to exactly one python function and so then the idea was okay given that we can we map it to one function can we also determine you know the dependencies from uh from How It Was Written so I'll show you this on the next slide but Hamilton right the output something you can get is determined by the name of the function uh and the dependencies are determined by the input parameters to that function uh so in the Hamilton Paradigm instead of um writing this procedural code here so this is Panda's code which I assume you're familiar with um instead of you know assigning two columns C and D kind of this in this pres procedural style you instead would declare them as as two functions c and d right so C and D are two things we can output so they map to function names and then the input function input arguments uh uh you know map to the uh input parameters to these functions so for C we only require A and B so that's why it's declared and then for D only C is C is required and then you use the the python body for whatever code python code you kind of want uh and so uh but you don't run those functions directly so a full hello world kind of looks like the following right you describe uh you put your Hamilton uh functions into modules um and then what you need is some driver code to kind of execute it the point of the driver code is to import you know the Hamilton module um but also to import uh this feature logic module here the real magic happens when you pass it to the Hamilton driver and so what it's doing under the hood here is given some configuration uh and whatever modules you pass in it is crawling uh the modules passed and effectively doing graph Theory 101 uh so if we can find a function it's going to create a node uh and then the and then it's going to use the function input arguments to then Define edges so you can see here on the right uh from these two uh functions we have a pretty small graph here where A and B are inputs uh and then C and D are two functions and the reason why a and b are inputs is because Hamilton didn't find another function definition that mapped to them so then it assumes that they must come from input uh and so then once once you've kind of given this graph right uh you can then tell Hamilton or ask Hamilton to compute something and then Hamilton will only walk that part of the graph to comput a result so you can end up building a very large superet of possible uh things you can compute uh but Hamilton will smartly only compute the things that are required for the output you wanted uh the examples I'm going to show here are going to be you know uh Panda Centric but just to point that Hamilton doesn't care what python object type you're operating over could be polers could be numpy arrays could be your custom object type uh and so it's pretty um flexible and extensible some other things I'm going to uh to mention uh but I won't cover is that uh you write functions but you can also decorate them and the Hamilton has a rich kind of set of kind of things you could decorate things with uh uh so add metadata you can parameterize functions you can have syntactic sugar to extract things you can add runtime data quality checks you know conditionally um uh you know include functions or not and then even parameterize different parts of your subag uh the and then more so I won't you know talk more about this but Hamilton's portable right runs anywhere that python runs uh and so if you and it also has hooks to scale so has plugins for pypar uh you Ray and dask um and you can use it you know DB BT to fast API to your Jupiter notebook uh and so uh yeah um I refer you to documentation to to know more but just to um uh plant Deb so just to complete the circle at Stitch fix right so uh stit Hamilton has been running in production since 2019 uh so the team uh before Hamilton was only managing about a thousand kind of feature transforms but afterwards they kind of grew to easily managing over 4,000 right and adding more of these feature transform definitions did not slow the team down uh and so the teams that stitchfix used it loved it um and so the anecdote is at stitchs one of the teams uh used to take them a day to uh update features uh fit a new model and be confident uh that they didn't break anything they used to take on the Euro you know a day and then it took less than two hours with Hamilton all we did was just change you know what code they need to change update um and be confident to get a new forecast out um sub subsequently those meant we found out you know it was faster onboard team members because they had linear engine documentation code reviews were faster and then they finally had you know unit tests uh and then also the ability to uh expose um documentation via Sphinx uh and so if you want to adopt Hamilton you wouldn't be the first it's been you know um since we open sourced the project uh there's a bunch of people who've kind of been using it in production so um and but otherwise since we since I started the company dagw works we had TOA the repo so there's that's why there's two but there isn't any Schism or anything where um where the main caretakers cool um so just to give you a little bit of a mental picture of of you know what a hand project kind of looks like right um uh is that you know it's very general purpose uh and so the only real requirement is that if you can draw a flowchart or a direct day cyc graph uh you can put it into Hamilton uh so feature engineering classically is a great use case right um uh maybe if you're doing just regular python data processing right you can kind of uh transform your scripts and rewrite them to Hamilton uh you can express end to end machine learing uh pipelines mlops Integrations uh you can even use it in a web request so people using it in fast API to like manage the web request stuff and then you can even you know use it in place of Lang chain and if you're a software uh uh Enthusiast right you probably like Hamilton because it helps your code look more structured and reduces your code complexity um in terms of you know the structure of a project right um with Hamilton so a classic example feature engineering as I mentioned uh the thing to kind of not is that with Hamilton you end up writing modules right and so uh if you want to do feature engineering you generally end up you know thematically kind of grouping functions that you know load data uh and then you know uh transforming uh functions that use the outputs of uh the data that's loaded to kind of uh you know transform it and then uh you're then going to be writing what we call drivers that then kind of help you know uh materialize a result uh the point being that for each driver right that couples context it will only kind of walk the graph for what's required from from that particular output um but you know from a code-based perspective high level a natural structure kind of emerges which I'll kind of touch on uh in in a few more slides uh but effectively what hamon helps you SE separate is logic um from execution context which is you know a very useful property for the longevity of uh good projects cool so on to Native software engineering best practices um so uh here are five common ideals right I think uh trying to assess software engineering best practices is a little kind of um subjective right um uh and so here I've kind of picked five common things so who here knows all the acronyms um that are on the slides few people no kind of right um uh don't worry I'm not going to go through all of them because I think there's a bit of overlap between them but you know kiss is for keep it simple stupid yagy is you aren't going to need it uh so don't over engineer things dry means you know don't repeat yourself and then solid which is comes from uh for those of you who've done object AR Java might know it but I'm going to take my uh my interpretation of it and and and and compare how Hamilton um fulfills it um cool all right uh so uh I think you know testing and documentation are pretty much you know uh I think when people think good software enging best practice these are the first two things that people kind of reach for to say this is what you need to have right um and so I think you know with Hamilton this is this is pretty easy right you can give this to a junior data scientist they can write functions and the magically the the code is always unit testable right why it's because you force people to write these functions um and because they're declarative right you're forced to not actually include too much uh in the function itself uh and so this means that it's very easy to kind of write a unit test that then passes just the things into test the logic uh that you want uh yeah in terms of you know if you want uh if you're doing data work it's also very important to try to add you know expectations on data right and so Hamilton if you're familiar with the term shift left uh has shift left to the extreme uh allows you to not only have the function definition right but it also actually set an expectation that will run at runtime uh that you can you know log a warning or pass or fail uh based uh introspecting the output of uh a function if you're familiar with a library called Pandera if not I highly recommend you check it out Hamilton also has uh integration with that uh and then you know you read code more often than you write it right and so uh you know documentation is usually a bit of a chore something that tchs on right but I think with Hamilton um you know it's it's kind of part of the way that you end up writing code um and so because of the way that you're forced to name things as an a function name are supposed to mean something that kind of propagates through your codebase so you might have things that are a little more verbose but in some sense they're actually then much easier to read and understand then of course you have the doc string of a function to add more documentation uh Hamilton does allow you to add more annotations um and so uh all this kind of information is all accessible once you build a graph right you can actually access uh these tags on things to to do some pretty cool stuff uh and then as I'll show you on on another slide you can actually visualize everything uh pretty easily with Hamilton um and so suffice to say with just these two things you have a pretty strong base to kind of you know scale your team and code right um but it's not going to uh as I show in a second like to a code that's easily testable and documentation friendly doesn't generally also mean that you can move very quickly um um more on that in a bit um but as I mentioned um Hamilton has this visualization capability right um every function you write effectively can map uh pretty easily to a node in this kind of graph uh and so this means if you ever want to um if you've ever drawn something like this by hand of going through a codebase to understand how everything connects while with Hamilton uh if it's in Hamilton uh you can you get that for free right and this means that you can always have up-to-date documentation so when some when someone uh changes a particular graph uh it's very easy than to go visualize it uh the flip side of this is you also get um lineage's code um so if you connect Hamilton up with your get repository system uh you can encode this graph right um and so you always then as long as you know what get commit sh you kind of produce an artifact or a data frame with you can always go back in time and understand you know uh the lineage so to speak as to how things were computed um and so just to compare uh to the code from earlier right uh this code I think you know struggles to embody easy testing documentation and visualization and so which case you know I think you know Hamilton is a is a great Step Above um so let me show you you know how Hamilton even goes further right and how it can actually help you with more of uh you your software development life cycle by imbuing some pretty good you know software engineering principles um uh to kind of you know uh the thing that I kind of ask you to think about is you know think about your code base um and the things that I'm going to go through think you think about how does your code compare or how easy and fast would it be for you to you know change or modify things um so solid uh for those you know is you know comes from object-oriented days um it's an acronym I'll walk through it in a sec but essentially the ethos or thinking behind it was that uh if you follow these principles you'll get more value out of the work that you're doing because it'll be easy to change manage and update right um and so uh I'm going to kind of you know take Liberty and interpret this um uh but applied to data flow since we're not doing object oriented kind of work here but I think the ethos or the thinking behind it is uh uh very applicable right since we have to change and update our data flows since it's not just getting to production once it's like the you know how do we evolve um you know the code right so s stands for U single responsibility principle uh the idea is that code should really only do one thing uh and the reason why is that if code only does one thing right it's just easier to test maintain manage uh and change right and so with Hamilton right you're forced to kind of do that with your function definitions is that the function does one thing right it's a one named thing um uh that you kind of create right um so then if I want to change you know for example what embedding means there's only a single place there and it's very clear um uh you know who's who's using it or consuming it um Etc um from from a a a business logic standpoint you know that's what these functions do but I also want to point out with the driver that you get with Hamilton um it's does not it doesn't isn't coupled to any of the business logic uh that it kind of runs all its purpose is to handle context of what and where um and so this that result means that you have you know uh uh execution is decoupled from logic uh and so this then means that it's easier to test maintain uh and change and uh this is I think a nice embodiment of the you know single responsibility principle o stands for open and close principle uh the high level is that things should be easy to extend so existing code you should be able to use it and extend it but then hard to break and so uh the idea is that you know you should be able to prevent issues uh from happening with code Evolution um so take uh I have defined here a data flow so LMS are hot so I have you know two functions one called embedding that computes an embedding from openi uh that then is fed uh so and have another function nearest neighbor IDs that takes an embedding uh Vector DB client and then you know computes uh tries to find the nearest uh uh results given uh this embedding uh if I want to extent right I don't have to go in and modify an object oriented hierarchy I don't need to go modify a script I just need somewhere uh in the world where I can pass this to the driver is a function uh that just depends on the things that I want so if I want to extend and use nearest neighbor IDs all I need to do is write a new function that takes nearest neighbor IDs as an input argument um and then I have now easily and quickly you know extended existing capabilities without you know too much uh software engineering work um in terms of hard to break right Hamilton gives you a lot of fix uh you know a lot of layers that you could add in to to make this uh true um and so when you write functions with Hamilton Hamilton does a check on kind of type annotation so that if someone is consuming height Zer me unit variance they should expect you know a panda series um uh and then if you are trying to run this function hamilt will complain hey if you have not satisfied and provided uh an input for for an output that you want and then as I mentioned ear earlier you can also add runtime data quality checks so you can even you know further uh ensure that you know logic and the expectations are in sync uh and then as mentioned previously it's easy to unit test uh and then uh integration testing is also uh pretty straightforward uh a lot of things that people do one way that people make use of this is actually in CI systems it's very easy in Hamilton to just uh compute a single path of your graph and so if that's something very important to you very easy to kind of you know uh add that into your C CI system and just compute the that's important to you so this then means that if someone else is going to change something you know it's hard to kind of break uh things without knowing it uh L stands for lisof substitution principle named after I think a lady called Barbara lisof um uh the high level is that it should be easy to swap parts of the flow without altering its correctness um uh and so here I have an end to-end machine learning uh kind of pipeline right um uh the the boxes here correspond to three modules that kind of Define you know data loading uh feature transformation uh and then kind of model fitting uh in terms of you know Hamilton gives you a few interpretations of swap right uh this code here uh it's very easy we haven't uh the logic here if we wanted to run this in a web service or in Jupiter right uh there isn't anything inherent that we have encoded here to stop us from doing it so it's very easy to swap uh where code runs in terms of um uh modules right Hamilton makes it uh in terms of you know if I had uh you know uh in development I load data differently uh from production uh I can uh very easily you know uh swap out this data loading module uh for you know how I load it in Dev versus Broad and the correctness of the program will continue to flow as long as I have you know um uh created the right kind of function definitions that then the rest of the pipeline depends on uh and then in terms of um uh if you don't want to swap out modules you can also swap out functions uh very easily right and so say I wanted to extend the example before but add in uh you know support for anthropic um we can actually very easily you know uh with some simple orentation and this kind of at config when annotation you know conditionally uh swap out a function implementation very surgically uh and then uh anyone Downstream of that is none the wiser uh since they only you know depend on you know the name of the function uh so the input parameter and as long as that's satisfied uh you know it's very easy to kind of swap in different implementations uh so since I only have a few minutes left I'm going to uh speed through this but um so I stands for interface segregation principle right uh effectively you should only depend on what you need uh and the net result is that you should have then claric code for for uh ex uh C clear code to understand and and for execution uh again you know with Hamilton you write a python function it's the declarative it only depends on what it needs doesn't need on anything extra right so it's very easy to uh to maintain uh if you add other functions and you don't request them to be computed you know ham is not going to run it um so in the case here if you have a really large uh pipeline right uh I don't have to uh you know uh if I only want some only one thing of this computed so in this case say a data set here Hamilton's smart enough to like skip Computing this right and so very easy and cleared then from a principle to understand what's run d stands for dependency inversion right uh you should effectively avoid T coupling between different parts of your code via abstractions this is important uh if you want to Ure flexibility for the ability to change and refactor your code um again you know because Hamilton all you get you gets you to write functions the abstraction here really is the name of the function and the input parameter types um and so everything in terms of you know functions therefore are pretty independent of each other so they're you know pretty if you can think of them as an independent iserable unit it's pretty uh as long as the name and the input and the type match you can pretty change the implementation very easily and then from a more macro level right uh you can group these functions into modules and so if I wanted to refactor uh how I did you know feature engineering um I wouldn't have to touch data loading or model pipelines as long as I uh adhered to the abstraction of I need to provide you know the right um functions that declare the right thing that can be output from this module with the right type so uh to summarize the talk um hopefully I've imbued on you that Hamilton is a micr framework for expressing data flows in Python uh and then if you write code with it you know you hopefully people won't be terrified of of of inheriting it um Hamilton Paradigm itself I think you know helps uh uh you know increase software engineering best practices and therefore the value of your work uh by ensuring your code is naturally testable and documentation friendly uh it for it helps you write more reasonable and modular code so you can actually move faster with your refactoring and as you're changing things um it's hard to do bad things uh you since Hamilton can provide you the safeguards to uh to safeguard it and that's because it helps standardize the way that you iterate and kind of add to a code base um and then one meta point from the talk is that you know of the tools that we put out you know how we design actually can guide users towards you know good software engineering practices uh and the few the talk uh before mine I think from uh the guy from Dexter right I think you know a functional Paradigm is hard to beat and in which case like I think you know Hamilton is another example of a framework that uses just the python function uh as the kind of the interface that people kind of use and so if you aren't going to use Hamilton at least think about you know giving people uh functions otherwise I'm excited to mention uh I'm soft launching something with Hamilton where we're trying to enable people to more easily and quickly have things off the shelf uh and so hub. dagw works. is going to be a place where we're trying to curate these kind of flows that you can express with Hamilton to make it easy in three lines of code to download and have something up and running uh and then otherwise if you're interested in what I'm uh building on top of Hamilton with a one like code change uh we can track versioning lineage catalog and observability uh so if you have these concerns or if any of these uh you capabilities are interesting to you would love to chat with you it's self-service you can sign up for free and then otherwise thanks for listening uh Hamilton is just a library you can p inst stall get started uh pretty easily uh but otherwise you know we love us star and get Hub and um yeah thanks uh any questions uh at table one upstairs on the fourth floor thank you yeah feel like it was a pretty dense talk so and I speak quickly so thank you uh if you managed to follow along but yeah um have you take questions or I'll be upstairs um at the [Music] booth