Stream All The Things!

scale.bythebay.io: Dean Wampler, Stream All The Things!

Recording: scale.bythebay.io: Dean Wampler, Stream All The Things!

all right it's it's good to be back at it was Scala by the bay the last time I was here I couldn't make it the last two days I had to be in London earlier this week and I got here yesterday so if I say anything brilliant then I'll take credit for but if I say anything stupid I'll just blame jet lag if I may I'd the photo if I like to stick photos for things like places I've been and airplanes and stuff I picked a bunch of photos of water this happens to be in Glacier National Park this lake here I wrote a book that kind of summarizes what this is really about this is one of those thinner Riley reports we have them on our booth out there that we're sharing with IBM and I'm actually gonna sign copies at 12:45 if anybody wants a signed copy if you'd rather just get a PDF then I'll just sign your Kindle or whatever so let's let's start with little context and I know that this is a group of people that you know you guys really have been in this chances are being in Silicon Valley you've been in the thick of this stuff for a while so mostly I'll be saying things that you know at least up front here probably all of you have encountered Hadoop is kind of this sort of baseline of what big data architectures are and we think about it and it's you know it rich innate is a batch architecture really where we have this problem of incredibly big datasets bigger than anything before and we needed a cheap efficient way to just kind of capture it and then go back and do analytics over it so you know the typical Hadoop architecture looks something like this where you have some big distributed file system for storage HDFS as it were tools to actually do analytics over it once you've got the data parked and used to be MapReduce jobs how many of you ever wrote MapReduce jobs yeah I can see the pain in your eyes I used to rail at MapReduce because it sucked just as a programming model but and then SPARC came along and it was much nicer so that's what we all tend to use today and when we you know write these jobs and submit them then there's resource managers to figure out how to break it up into tasks that's what the yet another resource negotiated or yarn is all about and then for completeness I'd added tools for you know getting data in and out of these clusters so nothing that probably most of you haven't dealt with before and on a very personal level and why so just again setting context why was it good well it you know it it solved the problem for the time you know it sort of we just seem to capture data and do batch analytics or even it's more you know less far back in time even interactive analysis with like hive sequel queries gave us massive relatively cheap storage and supported multi-user access for people to do a diverse set of workloads and when I was doing Hadoop Consulting half the projects I did were actually replacing data warehouses it was sort of like we dripped if you look at this diagram it's really we've ripped the database apart where's the database is this black box of storage and compute and whatnot here we've just torn it apart and said here you you manage it you figure out how it works so it's it's kind of a rough not as mature alternative to data warehouses but a lot cheaper and that's why it was popular so why streaming well even in the batch world it's actually is beneficial if you can get answers faster out of your data my favorite example of this actually if something that in principle works just fine with batch but actually is a competitive advantage to do streaming is searching in indices or you know tools like Google and so forth you know imagine that you heard about some breaking news story and what we all do these days we open a browser tab and start typing you know tell me about you know the train wreck or whatever and expect to get like a CNN page or something like that well you know if I was analyzing that data once an hour let's say to update my index then I would be out of date by an hour and hence I would be at a competitive disadvantage to people to do it you know on the fly as things happen so it's actually much better if we can do it quickly this diagram is taken out of that little book and the numbers correspond to notes that were in that booklet but mostly what I want to do is just sort of lay out kind of the architecture that we see emerging and then focus on the stuff on the right which is the streaming engine so obviously there you have to have data or this doesn't make any sense and they kind of broke it up into three groups so rest meeting like traditional quote unquote ways of interacting with systems where we do you know submit a request for information wait for a response or hopefully not blocking but often that's the way it's implemented and one of the things I want to argue especially towards the end is that I think this architecture is pushing us a lot more to integrate micro services into our systems and not just have like a separate Hadoop cluster and a separate like a cluster for everything else and so we'll still be doing restful kind of stuff but maybe maybe not for the data part so much as just you know interacting with our services but nevertheless it can be a source of data mostly though when you get to certain scale you're actually probably doing asynchronous analysis of stuff that's streaming in and so I use sockets and logs to indicate sockets being like the external stuff like since we're on Twitter and I still talked about the Twitter firehose so if you're one of those companies that ingest the Twitter firehose it's not so easy to go back and ask for it again that's some you know chunk of it if you lose it you know if you forget it or something so you'd like to this is data that you just want to capture and store or move down the pipeline quickly and not have to go back and try to get it in some way if you even can and then I use logs to kind of represent everything else that might be internal like clickstream logs server exhaust that I might be using for operations and so forth so and I mentioned clickstream logs because it may not necessarily be just operational logs it could actually be a way of you know sort of distributed logs or event stores that we use for actual business logic to so you know a biker services obviously my light Bend has its own ideas about how micro services should be done or so-called reactive platform but you know just you know to be Switzerland here it could be you know go based or heaven forbid nodejs or you know anything like whatever you like to use for completeness you need zookeeper because Kafka relies on it and other services use it for you know Federation and storing a shared information at some level really the data back though it's you need some sort of event store these days something that is very good at scaleable capturing of data and you're serving like multiple producers multiple consumers and Kafka has kind of emerged as the sort of de facto standard for this so most people today are deploying Kafka in one way or another how many you've actually running Kafka in your environments probably most of you yeah I figured okay so here again I'm preaching to the choir a little bit but there's there's some interesting advantages to Kafka one of which is this you know deliberately messy diagram where it's actually hard to reason about this architecture if I have all of these point-to-point connections and even worse you know let's say service one on the right here goes down then I might have data loss from all of these sources that are writing to nothing on the other end and how do we solve any problem in computer science you know another level of indirection so here we go so Kafka it not only makes it look prettier like that but it it gives us a unifying principle for connecting all these microservices and the resilience of you know everything over here could be writing data into Kafka everything over here could be crashing and restarting but it's okay because we've got to cut it you've got it reliably captured and we can go back and get it when we need it and so forth the there's really four streaming engines I want to talk about in a minute and actually a fifth one which is Apache beam I'll come back to those but I'll just mention them now because I kind of start talking about them before I really describe them and that is flank which you may have heard about if you got here at 9:00 o'clock this morning you know really interesting streaming engine at scale actually the diagram is organized a little weird in a way it really I'm going to talk about flank and spark streaming together and I'll explain why I think those are grouped nicely together in a little bit but then you might also have things that work more as application libraries that you embed in your micro services like akka streams kafka streams etc but actually before I even talk about them I want to talk a little bit about how you but maybe pick one for your own use is whether you like these which we just happen to like or you wanted to use something else like apex or something of that nature but I'll come back to those a minute but just to complete the diagram here you know these are all open source tools and we all have standard ways of talking to them now so you can kind of bring your own persistence to this architecture you can use distributed file system so that makes sense databases elasticsearch you know object stores like s3 and so forth and this is pretty agnostic about where it's running to you people are doing this now in Hadoop clusters although I personally don't think it's the right architecture for this kind of more dynamic system of things that have to run a long time with combined Micra services but you could do it in Hadoop if you want we actually like maysa a lot we've been working with mesosphere on you're running our stuff and meses DCOs you can run it in cloud environments meses on the cloud or whatever if you like and of course kubernetes that isn't on the picture here so you know I wrote that little booklet like 16 18 months ago or something and kubernetes was just you know kind of out there but not really as as hot as it is right now so things really move fast sometimes otherwise I'd probably have it in the list up here at the top as well okay so let's go back and talk about those streaming engines a little bit here's a friend of mine getting sucked into the river here so what are the some of the things you were gonna like you've got to build an app someone told you all right we need to do streaming so how am I going to decide what to pick so let's talk about some criteria you might consider maybe one of the most important is what is my latency budget you know how much time do I have between in a datum or event arriving and I and when I have to get something out the other side well usually we're not really talking about true real time we use the term real time kind of you know in a very loosey-goosey way in our business but you know I'm gray because I've been around awhile and in my day the kids didn't use the word real time when unless they meant it which shows you know like picoseconds to microseconds you know like rent landing stuff on barges in the middle of the ocean but you know we typically don't really deal with that too much because usually that means very customized hardware writing in C++ and crazy stuff like kernel bypass working and stuff like this really it's the game here is getting everything out of the way of you know that the signaling that I can possibly get out of the way but mostly we tend to move into the realm of hundreds of microseconds where you know we need to we're doing like ultrasound stuff I used to work on ultrasound machines a long long time ago that's why I picked this picture or even trading that isn't necessarily high frequency trading but needs to be moving relatively quickly and when you're getting into this kind of realm of latency budget then you know there's a lot of actual tools in the JVM that are fast enough and they're designed not to have garbage collection pauses long enough that you can actually use them like acha actors could be used for this how many of you've heard the Lmax disruptor okay yeah that was a message a very high-performance message queue that came out of the London finance community a guy named Martin Thompson was involved in this if you've heard of Martin when you're getting into like tens of milliseconds then you you have more options now the reason there's a credit card pictures here is I was once told by a guy at Deutsche Bank that if you think about someone but clicking by on a website for Amazon or whatever there's obviously some plot of stuff that's gonna happen right and you want to get an answer back hopefully within a few hundred milliseconds because that's like the usability criterion you know after a few hundred milliseconds people get antsy and they start you know doing what they do but he said that they only get ten milliseconds of that entire you know like 200 or whatever milliseconds to actually authorize the credit card all the other overhead and transactional stuff going on just leaves them ten milliseconds to make a decision so obviously that's going to constrain your choices and now you're getting into the realm lower tools that are designed for scaling like flink can actually support this as well as tools like acha acha streams Kafka streams you do have to be a little careful when you're putting data in a queue like Kafka because obviously it takes time to traverse the queue so that can be a source of latency if you're not careful and then when you start getting into hundreds of milliseconds now you're in the realm of maybe I'm going to do more like interactive stuff like this is like a picture of a Jupiter notebook I think or I might be doing machine learning scoring where I don't necessarily need it to be you know very very fast but reasonably fast you like looking for trends and fraud or whatever and one of the optimizations that's been used here as micro batching so for example this is a technique that's used a lot in incremental machine learning trading where rather than like update my model which is fairly expensive to do with a per event based update I'll just do little batches of stuff so I get kind of economies of scale at a small scale and as we all probably know those who have been playing with these tools a lot SPARC actually started out as kind of a mini batch model as opposed to a tool for handling individual events and as you get into larger timeframes of seconds to minutes then you have a whole lot more time to do expensive things like you know training neural networks or incrementally training them doing extract transform and load is here because usually you don't actually need low latency when you're just moving data from A to B you have more time even if it's not that expensive to do and you can again use even bigger batches of stuff to get economies of scale and I personally think if you're going above a few minutes you should not be doing streaming at all you should just kick off batch jobs very very frequently and the reason being this actually gets to why I think micro services have become an important challenge for us data engineers is that if you run something long enough like you kick off a Streamy job and it's gonna run for let's say a few months you're eventually you're gonna see every possible problem you're gonna see network partitions hardware failures you know spikes in traffic all kinds of things that can go wrong that you have to be able to design for and if you come from like a Hadoop engineering background that's those are the kind of problems you haven't really had to think about too much whereas if you come from a background of building like let's say Twitter the Twitter UI or our other web services those are the kind of problems you're used to dealing with so I think that's kind of forcing a merging of thinking about these architectures but if you can avoid that kind of long term problem by just kicking off a batch job every now and then that just is immediately scaled for the right size that needs to be and then it does its work and goes away you can actually circumvent a lot of this kind of challenge so over a few minutes maybe if you don't need processing more frequently than that then consider just doing batch ok another criterion might be how high is the volume I'm processing you know if you're under say 10 a few tens of thousands of events per second well we're kind of used to doing that with various restful tool kits not not a terribly big deal you know when we start getting into hundreds of thousands we can still use rest but we have to really start thinking about using non-blocking approaches so that we don't consume resources unnecessarily and don't have you know things that might slow down responsiveness but we can actually still do things on an individual basis like complex event processing if we really want to when you get up into millions of events per second then it's time to start maybe thinking about tools that actually do take your data in and immediately parallelize it over the cluster like spark and fling do so that you get you know pretty much near infinite scalability in theory it's you know even it within tight timeframes now the reason I put a picture of the nest thermostat here as turns out nest is written in or at least partly implemented in akka and I was talking one of their engineers once he told me that it turns out people tend to get up at about the same time every day and whatever time zone they're in and because these things are phoning home they don't they get these massive spikes of traffic as all of these nest thermostats phone home you know at about 6:00 or 7:00 in the morning every day to get to change the conditions to daylight whatever people prefer up during the day and so they did all these tricks about like putting jitter into kind of squash the peak and broaden it and so forth but they they have these you know the kind of scale problems ok what kind of analytics are you doing sequel his so here's something I never expected I never expected sequel would become a scripting engine of streaming but everybody is doing sequel now and it's not just a fad I don't think you could say well I've got to do it because the other guys are doing it but it actually is an extremely concise way to express a lot of the sort of data transformation we do and if you have a UDF mechanism user defined function mechanism then okay sure I can plug in a hook to my business logic you know something that maybe I can't just express as in sequel I think it flanked forward in Berlin this past spring or whenever it was not too long ago there was a talk by Alibaba and how they're basically converting all of their flink jobs to just use flink sequel api as their kind of scripting language so i here i this may look familiar to many of you have you worked with spark you could either write this query and just real sequel like this or you could use like the spark DSL that basically gives you sequel like operators over you know whatever data set you're using ETL is a really popular one and this is I think a great use case for Kafka streams for example and it's not the only use case by any means but for example if I'm ingesting say raw logs as strings into a Kafka topic then I'd really be nice to parse that into some nice record format for downstream consumption so people downstream aren't all doing the same parsing and cleansing so that's a nice way to like you spark or rather Kafka streams where I'll just spin up a job that's going to watch that raw topic parse it into some you know clean format right into a new topic and then downstream consumers will then you know read that one so obviously you have a lot of options if you're doing that if you're doing something more complex like a data flow then it's that's another case where you want a fairly rich API and most of them now actually give you something that looks a lot like the Scala collections API like this one that this is part of the inverted index algorithm for SPARC not specific to streaming I just happen to love this particular example because it's it really showed like that the yellow is that all the function calls and if you you know once you get that toolbox under your belt about one day when do I use flatten map when do I use reduce and so forth then you can just string together these stepwise computations really quickly and it's just for me it's a wonderful way to work or I can focus on the problem and not have to think about ceremony about the api's I'm working with another kind of processing of course might be machine learning so here's a problem we've heard a lot of people tell us about is that and I kind of almost hinted at this earlier so I might be training my machine learning models you know updating them periodically let's say every 10 minutes I want to update my spam filters just as an example but I need to score it with fairly low latency so I might be using one tool for training but a different tool for scoring so how am I going to get models across these boundaries and there are a lot of ways to do it including having the same underlying libraries that are running in both contexts but it's actually emerging as a pretty interesting challenge to make it relatively easy to update models in one sort of context of time and actually and compute resources and then actually use them in a long-running streaming process somewhere else actually I wasn't I was gonna hide this slide because this kind of goes into detail so obviously I'll put these online if you if you want to read this but I don't I won't go into this now near the end of some of these different trade-offs again these are criteria to use in picking the tool you want sometimes you really do want to analyze each event individually like complex event processing and here it's usually desirable to have something that's fairly efficient on a per event basis as opposed to something that's efficient when you aggregate over a lot of events so if I'm doing like ETL it's fit sort of a sequel model maybe I don't need to look at each event individually they're kind of anonymous in that way they're just records but if I'm doing this sort of a complex event processing where the routing of each event might be different depending on some criteria then that's going to affect my tool choices and sort of a related concept is is it actually beneficial for me to have this ability to do stuff in small batches or in bulk to get some efficiencies of scale versus individual processing and the last point would be how do I actually want to run this stuff so spark and flink you actually start these services running in your environment and then you submit work to them and they figure out how to decompose that job into tasks are gonna run in the cluster and they do a lot of stuff for you like that scheduling of tasks like splitting your data into partitions and all that kind of stuff and if your model that you're trying to accomplish fits that their model then that's a really great way to work but sometimes you actually want to do stuff in the context of micro-services where you have much more control over how things are run or how data is routed to different things how it's all decomposed so you might prefer something that's more of a library that you embed in your apps and then you control all that other stuff and the last one is how do i what do I want to integrate with other tools and they're actually the reason this is here is because kafka streams is really designed only to read and write Kafka topic so that's its focus but if you want something that can like talk to various rest points outside of Kafka or talk to databases you might might change your choices for tools okay so with that list of stuff let's let's talk about for example streaming engines and how they fit this model and that was the stuff on the right here the first one is Apache beam and this is Google's data flow that they've actually open source the streaming semantics part on the top but didn't provide a runner an open source so if you're running in Google Cloud you get everything but if you were running outside Google Cloud you need to use a runner like fling for example the reason I put this here and it's the reason it's kind of indicated as a box off to the right is Google's been thinking about streaming for a long time and they have really thought through all of the kind of scenarios that you need to think about if you want to do streaming that's actually extremely accurate now if you're just doing like approximate calculations for feeding a dashboard or something you know you don't need to be too careful about it but if you want to do things like very specific accounting level for your accounting system kind of calculations and not have to do it in batch later on then you have to think about some other problems and they've thought a lot about the semantics and are basically driving the kind of stuff everybody else is building into their streaming engines whether either as an indirect way through being a runner for beam or actually implementing in our own way things like handling event time versus system time just to give you an example of this suppose for the sake of argument that I wanted to calculate exactly how many units of each product I sold in my store on a per minute basis and I wanted this to be as you know close to a hundred percent accurate as possible well the problem is the speed of light is actually too slow for sapore network systems right even if there's no partitions in my network there's always going to be some events in the system that will arrive at the end let's call it the analysis node after this one-minute boundary so I needed to decide okay how long do I wait until I have got all of the data how do I know I've actually seen all the data what do I do though if I actually have like a network partition and I may not get some data from let's say some physical store out there for another hour do I do like a provisional calculation do I have a mechanism for sending a correction like you know just or maybe a retraction and then a new value you know like like you might do in an accounting book for example so they thought about all this kind of stuff and they've you know defined semantics in beam for expressing these kinds of this kind of precision when you need it so they're kind of driving I think in a way beam could be the small talk of our time so here I'm dating myself again your small talk was back in the 80s was this programming language that everybody talked about it was a very influential object-oriented language but almost nobody actually used it instead we can't we suffered through C++ and things like that and the Java came along beam could be one of those tools I actually said this in the talk recently and somebody responded on Twitter know we're actually running massive beam jobs so I know that that's I'm sort of exaggerating a little bit but it could be one of these things it's more influential than actually used as my point but if today if you want to run beam workflows flink is probably your best choice apache apex which i won't really talk about is another option for running beam workflows it's until it's designed for low latency it was written for that purpose but also at fairly large scale volumes and they have a new and evolving sequel dialect if you want to Grahame that way and also machine learning tools that are growing spark streaming is probably most familiar to everybody here and and you may know that spark actually started as a batch mode system like MapReduce and they came up with this clever hack for doing streaming which is well what do we just capture data and fix time windows and then run a little batch job over that data and that's the mini batch model that emerged that is slowly going away finally the new structured streaming library that has been in spark now for a little while is slowly replacing the internals so that you can get down to lower latency but if you're using the older and even to some degree the newer library today you know you're limited to about maybe half a second or a few hundred milliseconds latency so like that credit card off example I gave you're not gonna be able to do that with spark streaming but it's a great tool for training machine learning models and doing ETL and things like that where you have a little more time and then they have it you know really rich sequel and machine learning libraries okay so I kind of alluded to this there is actually work to make spark a runner for beam and there's also you know this is sort of true streaming engine that's coming so now let's go to libraries so if I want to just write my own micro services but have streaming options in there one of them is akka streams this if you know akka it's like an actor middleware system but that's really kind of a low-level programming primitives if I'm really thinking of data flow then I'd like to be able to write things as you know ingest data do this transformation join with this data set etcetera and that's what the aqua streams API is designed to do akka itself is really good for low latency and per event processing like if I need to do complex event processing as opposed to like bulk aggregate kind of stuff but it's also not designed though to do like sharding of your data into partitions if you have too much of it you'd have to do that yourself so there's a certain like upper limit of the amount of data you'd want to pass through akka streams without doing some engineering on your own whereas a spark streaming job we never flank for that matter we just sort of transparently scale up with the size of your data and caucus streams is kind of similar it's also a library you embed in your application it's very focused on Kafka obviously and they've done a really great job I think not trying to be everything to everybody like you know not trying to talk to every possible data source and sync but instead focusing on common use cases like well if I'm just serving a dashboard maybe I just want to see the last value for each key or maybe I just want to do roll-ups in my Kafka streams app so they thought a lot about those kind of common use cases that have done a good job implementing them they also recently announced exactly once I put this in quotes because anyone who's like a purist will say you can never have exactly once and you know in the general case but within a wide class of assumptions you can achieve exactly once effectively what it's doing is at least once message delivery with automatic deduplication is how it really works so it's a really nice tool for if for meaning you don't have to think about that kind of stuff you can just write it and most of the time like to a very high percentage you won't lose data but once again there's always ways you can lose data and distribute systems and actually this is another slide that shouldn't be here it should be earlier but the nice thing about SPARC those if you've written your application to do streaming that you can take that business logic and easily repurpose it for batch so that you could do both and you could even do the lambda architecture if you really like it I think most of most people believe now the lambda architecture is kind of going away because it's it's it has one big flaw which is if you're using different systems for batch and streaming then you often end up having to rewrite domain logic twice you know once for each case and it really is designed for the situation where I can only do approximate calculations in the streaming side and I'll use the batch side to do the more definitive calculations later on once I'm sure all the data is there we really are as a community trying to push this idea that we could do everything and streaming you know with some exceptions and not have to have like two systems running like this okay the last bit I want to talk about is the sort of conversions that seems to be happening with microservices and fast data this sort of slice the beef looking rock is a kind of a cool thing it's actually in the Ansel Adams wilderness outside of Mammoth a Thousand Island Lake I think it's called so I guess the question then okay I you know put up this diagram how does this actually relate to something like this this is like a contrived microservice example for maybe order processing where I'm doing the right thing by having a separate micro service for each concern and they have their own storage and they send messages to get services from other things well I think there actually is a lot of synergy between them first of all from the application perspective they really are the best way to use them is one route is have single responsibility applications whether it's a micro service that does one thing or even the data job so I might be running something big like spark and Kafka but I tend to be focused on writing one app to do like accounting of yesterday's activity and the website you know one application that's feeding the dashboard and so forth so we can still support the same kind of design goals of like single responsibility and separate evolution and so forth and both of them have to support data that's never gonna stop arriving unless we're in really bad shape which means we're out of business so they have to be you know the support all of the usual reactive so-called reactive concepts of you know resilience and scalability and durability and all that and kind of going the other direction anybody that starts out with like a three-tier service architecture if they're at all successful eventually data will become their dominant problem this is certainly true in this up building with Twitter you know they started out as the three tier Ruby on Rails fail well application and they had that Justin Bieber problem where whenever you know Justin Bieber tweeted something they had to suddenly send it to over he was like the first guy to cross a million followers I mean I've been following Justin since the beginning I'm real so thank you and you know that was that that really forced them to you decomposer architecture no micro-services and these days of course they just have massive data pipelines that they have to run even though they still have a lot of you know small services that look more traditional so if you came from like a Hadoop architecture world and maybe an old service this world you had some overlapping concerns but mostly you kind of focused on different problems and I think today it's looking more like these architectures are much more aligned together they have to look the same they have to solve similar problems and they need to be more integrated I need to get if I have a streaming pipeline I need to get data from other services to enrich my streams and so forth so I think this is kind of a long-term direction for us and of course I think Scala is a fantastic tool for this just you know throw a bone - Scala I showed that you know excerpt from a spark inverted index example that I just love that example so much because I remember writing that in MapReduce and then writing it and in spark and it went you know from here to this and and I could really think about the problem and and it's just a really good language for thinking about data data flow problems in general okay that's pretty much it I just a quick shameless plug we did release a platform recently the kind of bundles all these tools it's it's you can tell it's commercial because this is a color diagram you market who did this actually this is anyway and we talked about all these pieces here and then of course what you do these days in a commercial open source is you provide all the production tooling around an open source core that's pretty much what we've done and we did actually pick DC us from mesosphere which is the commercial version of Mei sauce is kind of the next generation capability for managing these very diverse workloads you not just spark jobs but databases micro services etc okay so we do have a little booth out here with IBM and I will sign that little book at 12:45 if you're interested otherwise there's actually a link for it here and that's it any questions I did say heckle on Twitter earlier so if anybody wants to heckle that's a good time for that too all right well I'll be around the rest of the day thanks a lot [Applause]