Scale By The Bay 2018: Nimbus Goehausen, Distinguishing features of production quality pipelines
you hi everyone I'm Nimbus I work at demand base where we do a kind of account account based marketing platform so kind of b2b marketing and kind of helping helping people achieve that through the use of a lot of data so you know some you know kind of working on these things over time you you want to have you know kind of productive production quality in them and I'm gonna be talking about what does that actually mean so you know these days these days are very exciting times there's all sorts of tools and wonderful open source ecosystems available to us that we can start you know putting things together you know there's the you know people follow the creative process starting with some some piece of data working within a system adding some more data adding some more systems sometimes it gets a little crazy and you know you've got you've got some big ask I was like let's we need this and this and this can't you just throw it together it's just gonna be a quick little proof-of-concept we're just gonna see how it goes and and you know you know you wind up with some kind of some kind of product out of this and it's you know it's beautiful it's lovely it's the product of your creative effort and then then something happens where you know someone like this guy is gonna tell you hey let's put this in the app it's gonna be great people are gonna love it so now there are one of the things when that comes with is the the ability to do this again in the future so the first time you did it you know it it worked out fine but then when you go and repeat the process you've got new data coming in you've kind of fixed things things that we're kind of not quite right before and you have things changing over time and some this is often good we want things to change over time we want to take in new data we want to fix things that were wrong before however sometimes just sometimes just changing things is going to cause problems for for for your consumers because they it's like whoa why how could this change I don't want this to change except you know and now this guy comes back to you again it says it well we got to work on this we got to productionize this system it's a kind of time we got to get it out of death and make it harden it and so I'm gonna be talking about what what that mean and there's all sorts of happy adjectives that people throw around that end in il T Y and these are all good things but there's you know if there's a plethora of these things and how do we know what to what they mean or what to focus on so I'm gonna be trying to focus it down a little bit to try to get to the things that at least for me these are kind of things I worry about things that I get headaches about thinking uh is you know am I gonna have to worry about this or is this going to be fine so I'm breaking it down into three areas one is kind of consistency in your results over time the actual product the results you're developing the data set itself what you're delivering to people the other is the speed and convenience of your own development how how is your tooling working how are you writing this and and how are you developing it kind of before it gets to anyone else and then the way you're delivering things so the the safety and frequency in your delivery itself so you know how you like how are you deploying this how are people consuming it how are you switching from one version to the next so these are the three broad categories that I'm going to be talking about and let's just jump into it so now what are some of the things that so mainly I'm talking identifying is like here are common issues that you might run into into these categories and obviously I'm not covering everything there's only a 20-minute talk but I'm gonna cover a few of the the kind of top items that that I consider when I'm when I'm thinking about these so one of the one of the main things that you can consider in your kind of consistency of your results over time is are you missing records from one thing to the next and this depending on what kind of system you're working with this may be a bigger or smaller problem or not a problem at all the other is kind of like oh no we've we've kind of made this thing and sometimes if you have like an accounting system or something it needs to be a hundred percent rather of every time where else it's wrong but if you have something that's more fuzzy it's like oh we want this to be accurate but we're not expect you know 99% accurate that's still some things wrong and maybe from one to the next you have regression in quality or we have things that just literally cause things to break between your consumers because you have you know kind of different schema than you had before now there's also issues that occur when you're developing things so kind of broadly there's you know kind if you have bug reports and kind of issues that are brought up how are you going to track that down sometimes that is very difficult and sometimes when you go and want to make changes these can be it can be very difficult to see these changes out depending on you know what you've got going on in your system and often these systems can be very slow to see new results so and when it comes to delivery kind of main main two topics in that is it can be these batch processing systems they're not super fast and sometimes sometimes you want them to get to results on a more on a quicker basis so that that's one topic the other topic within that is sometimes there's inconsistencies and the way you're rolling this out and there are any number of things that can go wrong and I'll be talking about a couple of them so now now let's talk about actually addressing some of these problems so first when it comes to missing records so imagine you have a system like say a company database companies kind of come and go and you might have you know kind of depending on the way you're producing it you might have kind of some some of them in there from one thing and then you use perhaps different data or perhaps different logic and somehow not everything in there that was there before is still there and that might cause problems for for some of your consumers depending on how they're using them in say an application so one of the reasons why you might kind of lose records from from one from you know kind of one version to the next is that you might be depending on data that's changing over time you might be working with sign of like kind of a another data set does well like one of your sources which connect it itself comes in versions and as it changes over time and it might not have what it had before you're not gonna have it either so one way that you might address this is to try to stick to append-only data sources that is keep everything in your full history and always in overtime only add data over the into the pile don't remove it of course if you're if you're in a situation where this is you know you don't have this you're depending on a database which is getting updated and deleted you know and it depends on how much you care about losing records over time if you do care you might have to actually start storing each version of the database as you take it in over over your your own versions other strategies include kind of introducing a different sort of contract with your consumers so for for good reasons you might have things that falling out of your data set that you want to fall out at their data set because it's like oh this is now inactive and we want to treat it as such but for so that things might break you might include them but mark mark them as act inactive and other things you know sometimes things you know kind of the identity of them merges together so you might define like an alias saying system you see we're defined like this ID now points to this other ID and it's okay so that instead of like throwing things out that you had before you make sure you kind of either declare them inactive were you alias them to another identifier so another another topic is kind of validating your results so if if you're kind of trying to make an effort to kind of verify your results and make them you know kind of guarantee them to your consumers in a better way you might your validation itself my rival the complexity of your actual build process and this is this is a good thing there's all kinds of things that you can check and like whatever kind of assumptions you make you want to carry them out and it I think what an especially important area is to just dip your versions and and see what comes out of that and that's that can be challenging because like obviously you want you want it to change and sometimes the way but you know so you don't you're not trying to say like this should be exactly the same over time you just want to kind of find ways as many automated ways as you can in order to see if there's red flags that get raised so that you can dive in further for this so as an example like you know if you just like go off the columns of your data set you see one one kind of easy way to thing to do is count the number of distinct values and then see if that how that changes from one version to the next and if like for some reason there's a drastic change in them you might you might say like oh that oh no we should we should look into that so you know and often you have to kind of manually observe these things but the best best way to do it is to kind of narrow it down first before you start start you know kind of looking at millions of things which you can't really do very effectively so various various validations not going to go too deeply into any of them but those are great so now when it comes to development and kind of making that improving that you want sometimes these things can be black boxes these systems are don't lend themselves very well all the time to kind of tracing the lineage so you might have people kind of asking you hey this record in your your data set that you produced it came out like this I don't understand why can you tell me why and often you can't or it you it takes great effort for you to go and explain how something came to be and it doesn't come kind of understanding the lineage of the data through your system it doesn't come for free you have to you know often you know deal with this so one of the properties that is very helpful for for making sure your system is is you know you're able to track these things down is reproducibility so reproducibility is the idea that you know with the same inputs the same parameters you're able to produce the same output and that for a lot of these systems you don't have that by default and if you're not checking that you have this there's a good chance you don't so you know kind of there's there's different ways that you know you can have systems that aren't reproducible one if you know obviously if your if your logic is non-deterministic which in a lot of these big data systems can easily creep in because of the how things get ordered differently different from the shuffle and then what you do after that point and if you know so obviously the best way to do this check you know do multiple runs on these things and see that when they should come out the same they actually do and you might sometimes be surprised that they don't so the other the other thing is like your input data can be changing over time right and this is you know good thing this is how it works but so something you might consider is introducing in your config you would say like I want a high watermark so that I only want to use the input data up to this point as opposed to say use all date use at all data including the current data and if you do if you specify a high watermark that means if someone asks you about like last versions like what happened there you could actually go and reproduce that with the parameters and have it come out the same so you can isolate these issues so isolation is a very important important topic because like if if you don't have the the reproducibility of your you won't be able to you won't be able to isolate things very easily now we generally want to fail fast in these systems when we make changes when we have new data we want to know as soon as possible whether it's going to break or not and this there's a whole sort of range here from you know kind of the worst situation to the best so the worst is you don't know if it's wrong or not you just it just you know look looked fine and you don't find out that it's something's wrong until someone that depends on it complains to you about it and that's that's kind of it you know kind of very slow sort of testing framework a better better approach is to get as much kind of you know if the if it kind of took you get a compiled failure because you have you're accessing the field that doesn't exist or anymore or you have you have a schema mismatch often you can detect that at compile time and that's really what you want you want to arrange your system so that it's as front-loaded as possible where it'll it'll fail if some one of your assumptions is wrong and there's all sorts of things in between you know generally you know recommended test on your full data set see what comes out before you kind of merge new logic or new data into production now something development can be very slow and one of the common strategies to speeding up that process is to actually cache things and use intermediate data you might split up your logic instead of so instead of having one big box wouldn't win big black box you have stages along the way that you can save and kind of run independently and then that allows you to iterate more quickly because it's just like maybe the first stage is pretty slow and you're not changing it but you're working on a later stage and you want it you want to work off that however there are some dangers that can come along with this one is that if you have these different systems that are kind of riding out and reading back in you might lose compile time checks depending on how you're defining your schema so if you're using like SPARC with data frames you write out data frames and you read it in back later and cast as a data set you might you could have schema mismatches between your two runtimes and you then lose that property that we want of failing fast to kind of catch some of these things at compile time so so generally I recommend approaches where you maybe don't split that you still kind of maintain as few systems as possible but you use perhaps get or create patterns so that you can get the benefits of kind of speedy speeding up with intermediate data you have cache data but but you're not but you're still keeping things within one system you can have like a type checked system all the way through and that that's usually a kind of a plus so now talking about issues you might encounter and delivery and how to deal with them deployments deployments are very difficult and I would say generally when you when you start with kind of you know I've made this data set now I'm deploying it to consumers is it you it's not it's not a simple process sometimes especially if you have multiple consumers or if you're delivering to multiple systems and especially if there's multiple people involved that with multiple buttons that have to be pushed in a synchronous order and the the kind of more people involve the more buttons involved the more systems you're deploying to the greater number of things that can go wrong or you can have things where you partially deployed apply it to one but not the other and then there's an inconsistent see and you know people are upset so the the ideal is to get to us get to an arrangement where your deployments are atomic that is there's one button that one person can press that it switches from one version to the new version and this is easier said than done and I can't really go into how you achieve the how you achieve this in practice but an example of this would be like if you have a new version of a table in a database doing just a rename and assuming everyone's depending on that just that one database that's an example of like an atomic to a point another example is if you have like say you have a route to a database instance that is defined and everyone's pointing out that route and then you bring up a new instance and you switch you know kind of in it and it becomes ready and then you hot-swap you switch switch the route to the other thing and that's generally good practice but easier said that done some of these and you know generally the if you can work to get fewer buttons and fewer people involved so that year you know there's less things that can go happy less less chance of a partial success in your deployment that would be best so now about item potency so when we're deploying things we're delivering things together we're carrying out these operations there's failures that happen and we want to deal with them gracefully so say you're you're writing to a database system and you you know have some process that's doing inserts and it fails halfway through now you have a bunch of record now if you have half your records there and how are you going to repeat that if you go and repeat that process and it succeeds the next time you still have a problem because now you have you have the all the data that you want but then you have another you have duplicates of about half of it which is not what we want so it's best if you if you kind of use operations that are item potent that is if we have failure or partial success we can simply retry the operation and reach you know kind of we we reach a desired the desired state at the end and so this needs to be resilient so that it can be run multiple time so if you had a failure and you we tried it you can reach a success and also if you had a success and you try it again you still have a success that's that's the that's the kind of Ida that's the that's that's the principle of idempotency and it's something you probably want so you know kind of any examples of you know say if you go kind of like database operations inserts and appends are not idempotent if that's all you're doing overwrites are however overwrites can possibly lead to other problems such as hey why did you delete all the data and then leave this period of time where I didn't have the things I had before before you inserted the new version that's unhappy as well upstarts for databases tend to be kind of the you know for like if you're kind of deployed you know kind of deploying data to an active system and you're trying not to screw it up that's generally the thing you would want to do but you know it's you know it's obviously it's kind of a little more complicated to deploy sometimes now if you're if you're worrying about kind of getting results to people faster if one of your problems is your updates or your batch is too slow we need more frequent updates something you might consider is deploying what people kind of turn as have term to lambda architecture and the thing is is you probably don't want to do this unless you it's really important that you get those updates out quickly because it's you know very expensive and complicated in order to deploy a system like this and what it is is you're duplicating your your input data when it comes in it's going to two places one it's going to your warehouse where your batch layer will be able to to pick it up and it's also going to your speed layer which kind of makes inserts data kind of makes updates live in the circuit in the serving layer but then periodically your batch layer is going to be basically kind of defining the the kind of solid truth over time and it needs to you kind of play nicely with the speed layer but they they both need to be play nicely with each other so accomplishing this is really going to depend on your ability to be successful in the other topics that I mentioned and you know so now what I know couple to couple takeaways I generally think that a suffering based approach when it comes to these things is best you don't want to do we don't really want to bust your ass to do any of these things with unless you have really good reason to so addressing the underlying issues that are causing pain as is generally what I think you should do and it's best to you know kind of a an ounce of prevention is worth a pound of cure that is if you kind of think about these things ahead of time before you've gone and deployed these things you understand the risk you're gonna have that's usually best rather than trying to go and realizing it after the fact and change changing everything that can be much more painful and and and that that's it does anyone have any questions [Music] Oh as far as like continuous atomic deployments I mean you know there's there's no reason you can't do you can't use tradition I don't have any special recommendations aside as from existing things there's no reason you can't do continuous integration with atomic deployments with something like Jenkins or you know teamcity it's just a matter of like what what are the operations that that's actually doing that determines whether it's atomic or not so you know no special answers there other than existing [Music] standing up an entire yeah I yes [Music] four hours yeah so this is this last question because I'm at a time but so the question was like what do you you know these systems if you try to do a hot swap or you bring up an entire new database and switch it over some of these things it's gone that's gonna be very expensive both in time and money and if that's yeah that you know when that's the case I'd say you know this is a this is a balance it's like you want to go with something that's going to be reasonably priced and like effective and simple and work for you and I'd say in that case you know stick to up certs if you can't if it's kind of prohibitive to be standing up new things then don't you you kind of have to bite the bullet and do the do the hard stuff with ups appropriate up certs all right thanks everyone [Applause]