BDSBTB 2015: Lee Mighdoll, The Live Layer and The Lambda Architecture
Recording: BDSBTB 2015: Lee Mighdoll, The Live Layer and The Lambda Architecture
alright I'm going to talk to you today about a architectural concept called the live layer i'm lee McDowell I have been associated with a number of companies that have previously employed me a few that still currently do the opinions you're about to hear though or my own please don't blame the the people who've paid me if the wheels fall off your data pipeline based on what I tell you today then they're not to blame so in this talk there's a few few few things to think about that are coming first you know I want to encourage more use of big table class data stores Cassandra file ODB like we were just hearing about in our big data pipelines second I'm going to talk about this live layer concept this architectural concept for for our data pipelines basically the idea of doing more data transformations on demand and to drive the talk i'm going to talk about the lambda architecture I'll set up a straw man lambda architecture and of course what as one does with straw man architectures then I will try to criticize it so what about the lambda architecture one of my colleagues a little while back asked asked me what I thought of the lambda architecture and so I read about it and of course part of it is very sympathetic to the lambda part of it is very sympathetic to us to those of us who are in the functional programming interest groups right this idea that queries are pure functions that operate on all the data what a wonderful way to organize our data systems will get all the advantages of pure pure functional programming composition testing etc and and that'll be great but of course the lambda architecture isn't that much about lambda right it's really a reaction to the batch systems that we have been suffering for the last 10 years or so you know the problem was that these batch systems were too slow and so if we wanted to have data that was fresher or results that came sooner you know we're gonna have to do something that was something else and so will combine our batch systems with a streaming layer that will be fast but approximate will be less reliable perhaps and the results may be calculated with less accuracy right so this gets us to the canonical lambda architecture and you know it works the lambda architecture is effective it gets us some of the speed we need but being you know there's a number of problems with it that have been often discussed the dual data pipelines and and the you know the sort of lack of reliability that you may have in a traditional streaming pipeline you know or the lack of accuracy that you know you're going to have the the back system is going to be the canonical correct one and if the streaming system isn't it's accurate well you you may be walking yourself out into a place where your current business case works really well writing it you kind of convince yourself that well I don't need that much accuracy or or it's okay if it's not quite the same as the other thing for this case but it puts you in somewhat of a shaky spot when the next business case comes along and all of those rationalizations you know all of those arguments you made you know they may not be as well remembered by the rest of your organization so and I think another problem with this strawman architecture and you know it's really it was really proposed as kind of HDFS MapReduce plus storm the advent of spark changes the picture only a little bit although I hope that will change more but it doesn't really leave a lot of room for these no sequel stores or does certainly doesn't emphasize some of the wonderful things one can get out of them and I want to encourage us to to use those so so in the lamb to Architecture you know where do you get to do the data transformations where do you run your your code that that operates on the data well you know essentially the streaming or the batch layer and basically with the live layer concept what I'm trying to get people to think about is there's this third spot that's even obvious on this initial diagram all right let's do some more data transformations you know they're right and and what do I mean by a live layer well I mean you know let's have a a storage and computational layer right designed for doing low latency data transformations because we're at the end of the pipeline so we're going to need to be fast and let's have this let's be let's be able to do transformations that are dynamic right that depend that we can change based on based on runtime because here we are at the serving layer we might know what the what the users want right and we needed a storage and ideally to get the most out of this I'll argue that we would really like to be able to trigger these fast transformations on data arrival not just you know doing them on a schedule or something like that so and why do we want to do that why should we create this third place why should we mess around with with doing data transformations in this spot on the pipeline well the primary reason I'll talk about some others but the primary reason the sort of reason that got me thinking in these is is we want to make data transformations dynamic we want to make them things that you can change flexibly and we want to expose more of the flexibility in our data pipeline to two people downstream that are using it not just the internal developers if you think about it in the the traditional lambda architecture you know a lot of the the data transformations we write we deploy them in a topology we we have a batch job that we ran yesterday right and then the results of those those things are static for our users right they can get what we have created all right and this you know I've been working in on data visualization recently and that was kind of the the inspiring idea for this all right so let me show you a quick demo this is an open source a project called Sparkle you can find it in my github repository it's a little out of data I'll push some new stuff relatively soon follow follow me on the twitter feed and you'll see when that comes but here let's just upload a little data and i would say this is so what this is is a cassandra back end with a columnar datastore built internally using reactive streams for processing he's currently using the the observable library the RX stuff and then feeding the data to do dynamic transformations on the server and delivered over a web sockets Jason protocol to a d3 based browser client okay so here we're just looking at some some of this data that I've quickly uploaded there and the thing when you're doing interactive you know visualization we're trying to explore the data you don't you don't really know in advance you know what are the interesting parts right here we get the data loaded intently see a picture and you know there's this interesting spike pattern this is some latency data from a few years ago I just had lying around but you know look there's this kind of 15 minute cycle here right well we see stuff everything you know so maybe we want to zoom in there right and see the see the data and the the you know we're looking at the you know the 99th percentile latency here and you know like there's this interesting pattern we know that it's here we we didn't decide in advance which section of the data which which area to look at or how to aggregate the data you know we decided as we explored right and you know maybe we want to see the you know the the 90th percentile data mixed in right we're looking at why is there this this interesting spike pattern you know and and here we see if we look at the you know the 90th percentile data mixed in it seems like our problem is occurring every 15 minutes but you know it's actually getting worse over time and getting deeper into the percentile so maybe that's going to teach us something about you know maybe we've got some queuing problem seems like we got some resource here that's that's filling up right it's not just a 15 well anyway the point isn't about exploring latency issues but the fact that we wanted to do these data transformations only based on what we learned as we were exploring right we didn't we wanted to parameterize the data transformation and use the power of the all of these computational elements and storage elements on the server to do data transformation but we don't know which data transformation to do and we can't possibly pre aggregate precompute all the possible transformations so this is an example from from data visualization but i think the principle applies to many kinds of data analyses we might do on our data okay and so that flexibility that I talked about adding for the users well it's not just for the users right we might want to add that flexibility expose that flexibility to our downstream application developers as well right think about the the front end you know the people doing web applications and mobile applications right rather than we you remember when you first tried using the spark shell and you know you got this wonderful interactivity was flexible you could it was fast you could you could type in the stuff and see see the results right but but after we create our pipelines and then we give that to the applications they don't they don't get that flexibility right now now I'm not going to argue that we should you know expose a spark shell API as part of our you know our rest interface to to a mobile app right but but maybe there is something that we should parameterize so that they can you know have some constrained flexibility and take advantage of the flexibility in our data pipelines and then they can iterate on their own right when they need a new feature you know a lot of things they may be able to do on their own so so again this the the inspiring argument here or the inspiring idea is I wanted to see how we could do more flexible data pipelines and that's what's what pushes this this live idea and later in the talk I'll I'll talk about some other nice things you get out of having this live layer all right so here's the the agenda I guess I'm going to have to go pretty quickly I'll talk about storage for a live layer I'll talk about how a live layer would integrate with the streaming and batch systems you talk about in traditional lambda I'll blaze through some comments on code and talk about the the limits of doing live transformation okay so storage first go so you know so how are we going to structure our storage you know we got the data coming in we got the day going out right we're going to store it somehow in the middle you know it's a block oriented world we're going to be living in here and here in a big data context at a big data conference so we want to figure out how we can you know jam in just the right stuff into our data blocks that's what's going to make a sufficient if our data blocks are you know optimized for our write or read pattern life will be good right for writing boy you know if we could just write the data you know right as we get it right it right it into a queue into Kafka and we'll just fill up our storage blocks with the data as it arrives I think that's a great idea we should do that but you know just a queue just you know putting Kafka at the front of our our pipeline isn't gonna be enough because we're only going to read it out in the way we wrote it in and we probably don't want to read it in you know it's it's probably not you know in this diagram the blue is supposed to represent the the recorded time right but we the stuff comes in in you know in arrival time right and especially when you think about you know Internet of Things use cases but really in any use case the the data arrival time and the time that it's recorded are not going to be the same as asked you right and potentially a very significant skew between the arrival time and the recorded time so so you know overall writing in this right pattern is is efficient but not going to be you know not going to be the way we want to read it right so how do we want to read it well a lot of ways are interesting you know column stores are particularly appropriate for a lot of kinds of applications we may want to write our data out in more than one pattern right maybe it's not single columns of data maybe it's it pairs or triples that are interesting for our application or maybe we need to store stuff not timewise right here on the left side with a the time there you know you can see you can do a lot of you get a lot of time you can do a time slice query on one you know one kind of data for one entity one server device right but maybe we want to do a breath wedding when I get a lot of entities in there you know a lot of different servers but a little bit of time right there's there's different ways and the way we choose to pack our data blocks is going to drive what is efficient for our reads right that's the the storage grain is going to be important especially when we want to do things live but even in talking about these these writing and multiple reading things you know there's a few different ways to read it but the way we write it and the way we read it are almost certainly not going to be the same right so how do you deal with that well you know in in kind of your HDFS world well you're going to have to reshuffle the data and the this is where I want to encourage people to think about using these modern databases you know like like you know Cassandra big table etc you know they'll do this wonderful coalescing of rights for us right that you can write in one grain and the data will end up on disk eventually after a compaction pass or two in in a way that's much more read optimized and the nice thing about that too is that the you know that this late arriving data this skew between the arriving data and the recorded time you know can be handled transparently by the system for us so we kind of get one free shuffle write an adding notification I mention notification would be nice for a system like this is pretty straightforward certainly some of the databases support triggers directly and even if they don't you know it's a closed system we can we can add some notification on the outside and then we can trigger some of the the queries and the fast computation I'll talk about in a moment via notification mechanism so net-net it's not too hard to build a store these days that contains very fresh data transparently shows you the data as it arrives and is really fast for doing readings all right so now let's talk about how this will integrate in with the rest of the system streaming first so the lambda architects are really called for this streaming layer because we wanted stuff that was fresher and quicker right but but the livelier kind of means you don't need that as much right you I mean just going back to this this diagram you can kind of see it's it's a pipeline and you know whether you want to process the data before it hits the disk or after right it's kind of the you can kind of do it either way right and there's some reasons why you might prefer to do it later you know one obvious one is you know you've got a history available right you've got this this potentially very large data store that you can query quickly so you don't have to worry as much about the windowing problems you know if you got late arriving you know how big a window should you make in RAM for late arriving data well it's it's sort of almost impossible to decide but if you have a you know essentially infinite you no disk flash storage then it's not as much of an issue also in the streaming later we tend to process everything right but certainly if you think about these visualization cases well a lot of the aggregations you might care about a lot of the views you might care about are never never looked at by anybody in real life right it you know if a tree falls in the forest and there's nobody there to listen to you know why are we cutting down the tree in the first place right so so we might gain some efficiency that way so in a sense the live layer is just an extension of the streaming layer right just with this reliable you know reliable storage layer added to it if you like but I think maybe it's perhaps better to think of it as a different thing of course there still is a role for streaming right particularly as we're doing format conversions and you know well stuff is is kind of flowing in and you know it's sitting right there in RAM and we want to we want to do something that's going to apply to absolutely everything and be useful to absolutely everything downstream well it's you know streaming still makes a lot of sense all right so then let's talk about the batch layer a little bit so the first thing we might we might do with a live layer right if we might use this live store as an ingest stage for our batch pipeline right we've got this buffering this transparent buffering that kind of gives us you know there's coalescing for us right so it'll kind of do a shuffle for us and gives us a way to address some of this time skew so the first idea is well you know let's just feed the data you know into kind of scalable no sequel store and then flow it into the batch store of course it once we put stuff into the to the live layer you know we have this same same question of well you know should we do operations live you know late or should we do them batch in advance and there's this there's some of some similar trade-offs right if we do it live we get to defer some of the decisions we can be lazy and defer some of the decisions about how to parameterize our transforms that gives us some new capabilities and new new flexibility operationally to it can be nice because the if you change your data transformation you know tomorrow you fix a bug or add a new feature you don't have to reprocess all of history right and and do this major chunking a ball of your storage and I think your operations folks will will help will like you for that if you can avoid those operations as much as possible so you know batch isn't going to go away any more than streaming is going to go away some operations are simply going to be too large to do live right and the batch systems are certainly more mature at this stage but I'd like to think what we can start to do now is you know feed the results from the live layer into the batch layer and then if we do batch compute computations feed those back into the live layer so that they can be further parameterised and handled by transformations that are are more directly controlled by by the users and the application developers and ideally of course I talked about feeding back and forth but really if this can just be in one store that is fast enough and capable enough to do both then it's just you know we're just changing some pointers we don't actually have to move any data around so you know as with talking about extending the streaming to extend that you can think of this live concept as just extending the batch system right we're just we're just making a batch system that's faster and fresher I think that's also also true so anyway we end up with an architecture that looks looks something like this alright let's talk about me I'm gonna go through this very quickly so we talked about how to do data data storage quickly right I think that's going to be the first order problem but if it's easy to make mistakes in programming in these higher level systems on the JVM so and not have not end up with high-performance computation or accidentally end up with very poor performance so the first thing to think about when building you know the high performance execution of your data transform is how you can do the storage right and really the key there is you want to end up with dense blocks of primitives right and when I say you want to end up with dense blocks you want to have these arrays you know whether they're on hyper off if you want to have you know an array of doubles you know but maybe don't maybe not millions of elements you know think think thousands of elements especially if you're on heat there's some you know you're going to be dealing with the new generation there's some interesting issues there you don't want to you don't know avoid that but you know you got these these you know so think like a flow of data blocks of thousands all right then we go to code this stuff you know let me just take a timeout here to encourage people to use scholar for that all right how many of folks here are our kind of just getting into getting interested in the sky you know coming from the data side and getting into the Scala side you guys are all Scala people or a few okay all right so eat so let me offer a you know people talk about the advantages of Scala for building systems like this in a lot of different directions let me let me try one on for you that you know maybe I haven't heard as much right it's my belief that in a software system the most important thing about the software system is how how easy it is to change that system over time over its lifetime it's not just the the first set of code you right it's any software system that's going to get interesting right the interesting thing is you know how can you bring new people on how can you change what it does you're never going to get the requirements right the first time so how how mutable is your is your software so when people talk about these these kind of nifty Scala features and when you're trying to sell your nifty Scala features to the rest of your your organization you know think about these features as as refactoring enablers right all this composition absolute fancy ways to do abstraction right that what that mean you know if you if you can write code that's a little smaller what that means is you can you can fit more code into your head right each each person can understand a little more and can change it a little more right and that that allows you to be able to change your code base and that's really really valuable you know we talked about the type system I heard Martin earlier this morning talking about the type system and and this thing you know where where you write a bunch of stuff and you get it compile and it just works right and you know that's kind of neat you guys have probably had this experience but really what's important about that is not that it works the first time right because you're going to refactor the code 10 more times and the fact that the type system scaffolding will keep you safe you know and keep the system working in through the lifetime of this code base that that in fact is much more valuable all right so so scale is going to be great for this unfortunately you know the state of the art in most high level languages Scala included is that the kind of basic things that you write in Scala you know let's say we're trying to write this you know here's here's a simple data transformation we're trying to write right it's going to be we want to write this cleanly but it's hard to actually have that run fast so but the good news is as martin also was talking about is if there are these lovely libraries that are coming together so that you can you can get some of your you know the goal is to be able to get have your cake and eat it too alright you want to write this clean code that's fast right and you can do that manually today but really what I think you'll be able to do increasingly is to rely on you know these some of these libraries that you know are maturing now you know and yeah I won't go into those now in the interest of time okay so building a lie live layer is fairly straightforward following these these these bits we've just talked about but you know we can't we can't do everything live right there's this tremendous benefit from being able to transform live it would be wonderful if we could do everything that way but you know unfortunately we you know the architectural position that we're putting this live layer and it sits right next to the user right and so we've got this this latency constraint our users are waiting for our results so it's going to we're going to be limited in the amount we can do live by our storage and computation and you know I say our storage and computation but if we do our computation halfway right it's really going to be our storage layer that should limit us right so where is that limit its it turns out it's not as bad as you think or you might think right so if you think about how money you know hundreds of megabytes we can get off of a you know a single storage node you know it flash and and you know 100 500 megabytes per second if we could only use drm can only afford to use deram life would be wonderful right but you know that still means that we can get you know 50 hundreds of megabytes in you know available to our transform engine well within the time we need to do a data transformation and that turns out to be a lot for you know you can have a petabyte size store with a zillion sensors but if you're only looking at one or two of them at a time you know you can pull an awful lotta you can pull years of data out you know and relatively broad swaths of your population out even off of a single node right and that's just looking at kind of the raw the raw speed if we could prefetch the data because you know at the start of the user session for a visualization cutting and then we get two orders of magnitude improvements in RAM or if we cash through you to get a ninety percent hit rate in our in some cash well then you know we get another order of magnitude and not have to do the work for nine out of ten times you know column stores you know give us we can easily see three four five X compression you know get it giving us another multiple any we could fetch up multiple anyway I can go on and on but there's there's it turns out that there's a lot of things you can do live a lot more than you might think right we don't have to just deliver static results to our users with the storage system you know with the storage system and computation system it'll depend on your data right you need to analyze this for yourself if the wheels fall off the data pipeline you know you don't have to blame me but at the end of the day if I talked about this the storage grain right if you can align your reads to a lot if the query you need is kind of with the grain of the storage and not against the grain you know there's a really good chance you'll be able to do that computation live so you know what do I think is coming next in in this direction but like I think we'll start to see more framework support for doing live computations I think this flexibility argument is going to be more and more interesting for our data pipelines and we'll start to start to see framework support you know I'd like to see those ap is like the spark API start to you know unify streaming batch and live right and I think at the live layer to there's a an interesting opportunity because that's the layer where we're we're out speaking to our applications you know so how do we how we can talk about and perhaps slowly work to standardize some protocols for access to the live layer and there's a little bit of work on that I've done already with this you know to start the conversation on that with with this web socket Jason thing so so to take away I trying to go as fast as I can because I know lunch is coming but here's here's three things to take away so first you know be lazy right you know flexibility through laziness that's what your physical therapist teaches you're right flex no wait that's not it but anyway flexibility it try to try to be lazy think about deferring your computation until later in the pipeline and see how that can help you out in in your architecture look harder at using these these you know big table derivative architectures not just HDFS and when people start talking about the the lambda architecture you know think don't just take it whole cloth you know think about how you how it can get better you know see about how you can improve the lambda architecture with ideas like this this live layer thank you very much let's do questions after let's say we get lunch come come grab me privately if you'd like