Lambda architecture - a blueprint for bu...

BDSBTB 2015: Jieren Chen, Lambda architecture - a Blueprint for Building Big Data Systems

Recording: BDSBTB 2015: Jieren Chen, Lambda architecture - a Blueprint for Building Big Data Systems

so yeah I'm gonna be speaking about the lambda architecture maybe you've heard about it maybe you used it you might not have what I really want to do is to dispel a lot of the how you say around it because what it really is is that it's based on really old solid ideas so things like events sourcing server logs command query responsibility segregation but you know because all these things have kind of been repackaged and rebranded as lambda architecture you get you know a lot of this crap right so I want to move away from you know a lot of this marketing hype and a lot of these kind of like predefined recipes for how you would want to build a way into architecture and I want to move more towards more towards this because the land architecture is just a blueprint a template if you will for building big data systems and I would actually argue that it's a template for building any kind of data system you know if we go back in time none of us clearly remember this but in the 1960s people started building these particle accelerators and they were discovering all these like random particles that they were you know trying to put together into some kind of cohesive framework and then they discovered the standard model of particle physics which you know not only was able to describe all of these existing particles that they found but also predicted new particles that you know they didn't even have the technology at the time to find and so I'm kind of hoping that you can look at the lambda architecture as this kind of standard model of data systems maybe that's a bit ambitious we'll find out so a few my goals here you know this is a conference so obviously I want to sound really smart I want to provide you with this practical template for building data systems that fit fit your business requirement so I don't want to just give you this exact recipe I want you to you know take this template and then take your business requirements and hopefully be able to build some kind of bespoke system that really fits what you need and what you want and you know I also want to show you that a lot of these fancy buzzwords are just things that come from very simple concepts I want to break down the lamda architecture and build it back up from from real-world problems so let's let's start with this question so what is the data system so I went on Wikipedia itself a pretty interesting data system and I found this so a data system is a program that collects stores and manipulates items of data to provide meaningful information and so I kind of want to break down this definition a bit first starting with the verbs so what a data system does is it collects it stores it manipulates and then it provides or displays right so this is this is all pretty good this is important but I think it's actually the nouns here that are more important because what this definition does is it breaks down the data system into two parts there's items of data and then meaningful information right these two things are actually quite distinct like you put data in that's the input and then you get meaningful information out and that itself is a data system so to just go through a few examples here you know a high score table for a game is a data system right you collect points from users you organize them and then you display those I'm sure all of us can think of some really nice little sequel queries we couldn't run to to create this data system it's pretty simple but then we can get more complicated right we have a social media newsfeed where you're collecting users friendships posts you're joining these all together and then you're ordering them by some kind of function this is getting pretty complicated this is pretty big and then you can get even more complicated right you can have product recommendations for an e-commerce site you know you're collecting product purchases user ratings user interests you might be running some kind of machine learning algorithm on this it can get can get pretty real but what I want to do is to you know tie all these systems together and if you remember back to one minute ago where I described the definition of a data system it has its input which is you know pieces of data and then its output which is meaningful information so maybe we can think of something that takes in inputs and then gives out outputs right maybe I don't know a function right and so you know I think this this equation kind of really captures what the ideal data system should be you know maybe this you know you can't always build the ideal thing but I think this is what we really want to aspire to right and so you know suppose we have an infinitely fast computer let's just imagine that we do this is sort of what we want to build towards right something where you know on every request for V which is our view we're gonna you know compute this function f on our on our data which should be immutable so you know this is a this is a pretty good system right cuz you know purity is very good there's a lot of nice things about it so one is you have immutable data so you never lose information when you delete things when you change things you're actually inserting more data in instead of actually removing it and what this kind of gives you is this this like time machine where you can go back to any particular state of your system and then generate what the system would look like at that particular point in time so that's pretty nice what's also nice is that this makes the system very resilient to errors because most of the business logic in this you know very pure model of the world lives in the function lives in this pure function that you're computing you're applying to your data if you screw up that function all you have to do is just fix the function right and then deploy that and then everything's good right because the view itself you're not storing it's ephemeral it's just this very pure function and because there's just this pure function where most of your business logic lives it's very flexible to business requirements you want more you want to do more for the business which means you want more views then you just write more functions right it's pretty straightforward unfortunately we don't have infinitely fast computers yet that's very aspirational so we have a lot of problems with this system it's gonna be very slow it's gonna be very expensive so for example you know you have your view getting queried a lot and we can draw from our example of a high score table you know suppose we don't even have a lot of data in there suppose this function we have is very simple and can calculate it very quickly well suppose we just get a lot of requests for it right suppose we're talking about I don't know Starcraft right and you have your professional Starcraft players who are actually playing the game but then we're like in South Korea where you know the entire population it cares about Starcraft so they're always querying it I mean that's still gonna slow your system to a crawl even if you don't have a lot of data in the system and even if you don't have a very expensive function so that's a big problem but another problem is if you do have a lot of data right and we can draw on our social media example we have tons of users we have even more friendship relationships between the users and we have a ton of posts that all of these users are doing and what makes it even worse is that we actually have to calculate the Cartesian product of these to get our news feed and we then have to sort it right so we're massage were running through a lot of data and of course we can have a very expensive function right our business logic can be complicated we can take our e-commerce example you know that's you could do a lot of things with that right and sometimes you actually have to pull in in a lot of machine learning algorithms you actually have to pull in like the global set of data you can't run things piecemeal so this is this is pretty bad but there's there's actually a pretty straightforward way of solving this problem right I mean how do you solve any problem that involves repeated and expensive computations you use caching right caching or memorization and I'm sure I don't need to remind anyone here what caching is I'm just going to go through and talk about the different types of caching that are possible so the first is right side caching right so this is when as you make changes to the data you're also mutating the cache in some way so for our high score table example if we you know player scores a point we write out points to our data and then we also mutate some kind of counter that represents our high score table right so that's that's right side caching and I would argue that any kind of mutation that you do to a data system like any kind of stuff you would do right now that you wouldn't really label as immutable it's kind of just a type of right side caching so you know I just gave the counter example where as you insert you also mutate a counter something a bit more counterintuitive is suppose you have you know a post table let's suppose your Facebook right and a user goes back is like oh this post was really stupid and embarrassing so I'm going to edit it updating that post is actually not really changing the data you're actually changing the view right in this kind of system where you just have you know a sequel table with all your posts in it you don't actually have data you just have the view into the data so whenever you change the post you're actually throwing away data you're just mutating this view on to that data this cache and so you know the way I put it you can kind of follow that there's going to be a lot of flaws in this in this kind of caching right it's mutable which means mistakes can be made and when you do make a mistake the truth can be hard to recover right the computation that produced the truth might be really expensive and in fact in some cases you just lose the data you might not be able to recover it right like in the case of you know if you just do an update on a row well you know the old the old version of that is probably just gone the other type of caching is read side caching I'm sure you guys have used this before so basically what happens is you know you want to get something from the database so the first time you want to do that you you actually do that you run the query and when you get back the results then you cache it so that the next time you want you want this data you don't have to call the database something like that right so this doesn't have any direct locations for the lamda architecture for a number of reasons number one the computations you're doing might still be really expensive right like suppose you know I want to calculate a news feed for this user so I do this insanely long expensive computation and then you know well the user might not actually want to get that again right the only benefit I get from this is if the user comes back and tries to get that data again and moreover a lot of the raw data that you have might not be in a position to be accessed could be in a situation like s3 or HDFS or something like that where you can't just you know it's very difficult to just read that on the fly so instead of doing read side caching we want to accomplish the same kind of things but in a more reasonable way and so we have this concept of batch caching so with batch caching what you're doing is you're pre computing all of the views right so you know at time T God comes down and he's like let there be views and we get all the views right and so it's pretty obvious that this is insanely slow right this you're computing the entire universe but it's good because it's it's you know horizontally scalable right so you can break down this computation into your individual pieces of data and you can basically scale arbitrarily in an arbitrarily large way horizontally and it's also safer because every time you do this you are recomputing the world right so this is actually really nice we want to we want to do this as probably as close as we're gonna get to that pure function we were talking about before so batch is really great what the lambda architecture is and you know we're finally getting to it is just a combination of these caching methods you know it's it's nothing special it specifically combines the batch side caching and write caching maybe you could incorporate read caching into I don't know maybe that's possible but it does it in a way that emphasizes the strengths of these particular caching methods and then minimizes the weaknesses and we want to get as close as we can to that ideal universe where we have you know infinitely fast computers and we can really just do purely the ft so let's go let's go into the box's world here this is the batch side so you have this this box this is our source of data and it's feeding data into our master data set so this would be something like HDFS or s3 whatever you really want you feed it in and then you have you run a batch computation on that every once in a while right and then that feeds data into your serving layer which is where you can serve the data I'm pretty sure you guys have seen this and have used this in your pipelines this is like a very classic big data pipeline what's unusual about the lambda architecture is that we're also gonna layer on this extra speed layer which is actually very ill-defined right because the speed layer deals with a lot less data you can do some weird stuff with it right and it's only dealing with new data generally you have a lot of flexibility here because you're operating on a much smaller data set you can use different technologies you can do streaming if you want to but you don't really even have to you can just put data into like I mean honestly you might even be able to put data into like a sequel database here right that could be your speed later and another thing that sometimes gets brought up with lambda architecture is that the speed layer actually can be really inaccurate it can be inaccurate and you still kind of get a reasonable estimate of what you want to see and that's fine you don't have to do it that way but you can so at the end we have these two different size the the batch and serving layer and the speed layer well we want to take take data from our serving and speed layer in an API request and we want to combine it together at some time so this is really nice because it not only allows you to deal with two different types of caching it also allows you to partition your technologies right so you no longer have to search for this magic bullet database that does everything well you can just have you know some crappy old database for your speed layers I mean it shouldn't be crappy but it should do what you wanted to do really well it doesn't have to do anything else really well so another good way of visualizing the lambda architecture this is the way I prefer is using a time line right so you know suppose we start with nothing and our first batch job finishes right we fast forward time to this point where first batch job finishes we have a view finally and this is all good but time has you know a tendency to continue going so we you know we keep going and all of a sudden our data is stale by maybe a couple of hours this sucks so what we do is we use our speed layer to keep us up-to-date right it's just it's just filling up here and as time continues you know our speed layer gets larger and larger we actually don't want it to get too large right because that's not what the speed layers for we wanted to keep it at a reasonable size and we do that we do that by defining a partition or a boundary between the speed and the batch layer so that anything older than that boundary on the speed layer we can just toss out and this is nice because then BAM you know the the second batch job finishes wipes out all the old speed views and that's totally fine and I also want to note that this is not like a supplemental thing to batch job one this the batch job to provides views for you know for all of that time so it's actually wiping out everything and this is really nice because you know any kind of human error that we encountered before is fixed right so if you made a mistake in batch job one fixed it by the time batch top job two kicks off well then you know you're good to go if your speed layer had some kind of problem well you could you know you can turn off your speed layer and you know just deal with your stale data make that fixed and by the time batch job two finishes it's gonna it's gonna have good data up to that point right so this is kind of a very self repairing system so so now that you have a good grasp of what the laminate architecture is like from a theoretical standpoint let's kind of take a look at a few examples of it so that you can kind of see how how the different kinds of ways to do it and I'm actually going to dive into some how you say degenerate lambda architectures which you know incorporate either only just the speed layer or just the Batchelor and see like why you would make these kinds of decisions so the first is you know the nathan mars lam - architecture it's pretty straightforward right it you dump data into HDFS that's your master data set you process it using Casca log and then dump that into elephant DB I've never used elephant DB but you know this nathan mars a smart guy he stands by it and so that's your your batch layer on the speed layer you dump data into queues which are processed by storm topologies and these topologies dump data into HBase which serves as the speed layer database then when you're ready you take you query both the elephant DB side and the HBase side combine those results together and you can get exactly what you want so before will I worked at clout and one of my last projects is at cloud was to rebuild our data pipelines so that they could support real-time views so you know I read all the things that Nathan Mars broke I really liked it but I didn't want to do this because that involved a lot of changes we we didn't have Casca log setup we didn't have elephant DB and probably didn't really want to use it because it's not fully mature we had HBase setup but we didn't really want to deal with a lot of storm and stuff like that so instead I built something like this right so we already have HDFS for a master data set we already have hive to process that data and we already had HBase so you know that batch side is was already built for us on the speed layer we we had elasticsearch setup and what I was thinking my thinking on that side was instead of dealing with all these weird storm topologies and so that every time you had to create a new view you had to do a lot of work let's let's reduce that because the data we're dealing with here on the speed layer is actually pretty small it's not only constrained by time much like this presentation it's also constrained by you know we don't even care about all of our users we only care about active users the only active users need a see real-time data so we can try to keep that data really really small and just push you know that bra almost raw data into elasticsearch and then do more computation on the read side right so it's every time you want to develop a new feature on it well you can just write an elastic search query and you're good to go instead of needing so like set up all this all these you know storm spouts and drains and other water things and that kind of goes into the old pipeline so before we added the real-time components all we had just you know HDFS batch jobs in hive going into HBase this might look kind of janky but like why would you do anything else with this right if you think about the cloud score it's not something that changes very often right it might go up one point at a time per day so it's really not a big deal it's not very useful to have real-time data for just the score and that's what the old pipeline was built for so you know even so in this is a lambda architecture but it's just one side of it right and this is the stuff you will want to consider when building out whatever data system you choose to build and likewise you know suppose at some new startup maybe named will we don't even we don't even care about the batch side we don't have to deal with that much data right we can kind of get away with very quickly building out these you know these speed views on something say like Postgres and just run with that and actually even in larger in larger data situations you can do this too in fact I think this is kind of what the kappa architecture is built on so the cap architecture is something is similar to the lambda architecture but it's just consuming things from a stream and the idea is that you don't really need the batch side instead you can just reprocess old stream data and just spin up more workers and so this is actually a you know just a very much a speed view a speed view architecture so just to summarize lambda architecture it combines just two types of caching right that's all it really is it's an elegant use of cache combines batch side caching where your recomputing the whole universe and then pushing that out somewhere as well as right side caching where every time you make a change to your data you're changing your cache so it combines these two techniques this allows you to partition your technologies and specifically on the speed the speed side you can use interesting technologies to get whatever you want the batch side is a bit more prescribed I really want however yeah exactly you still need to design your data systems like you can't just take these things as recipes and then just say okay I'm done right you have to think about your business requirements think about what kind of trade-offs you want to make but this is a pretty nice template for you to think on so that's that you can contact me in various ways but don't be a recruiter Thank You Jared I think we're just about let me go to one really quick question so how much more complicated was your codebase compared to having a more typical architecture in terms of like code replication and the addition of the the CIM the combining layer they're combining layer the combining layer I would say is is definitely the moat it can get the most complicated there's a the combination of these two things will depend a lot on business logic the nice thing about our thing was that it was easy to duplicate things because what we wanted to combine were you know views like all the users posts aggregate data so there's actually because these are a very time-based there's like a delineation between them and you can take advantage of that there's a lot of things I don't know recommendations would probably be pretty hard to do it would be hard to build this kind of partitioning thing for all right thank you