BDSBTB 2015: Jay Kreps, Apache Kafka and the Rise of The Stream Data Platform
Recording: BDSBTB 2015: Jay Kreps, Apache Kafka and the Rise of The Stream Data Platform
right hello right okay so this talk is going to be kind of a origin story talk which you know i think has already been alluded to so you know it'll be sort of like Batman Begins for distributed commit logs but maybe less exciting so so as Alexa said I'm coming out of a company linkedin where I worked on cough cup for a long time one of the reasons I like to talk about the origin for any system I'm involved in is I think it tells you a lot about the problems people are trying to solve why it is the way it is how everything turned out and then also where it's going right there's usually some whole mindset which is embedded in any of these technologies and if you come to understand that mindset then I think you can kind of understand what the technology is really about and so so that that's the purpose of this talk hopefully it'll be useful even to people who you know have some knowledge of coffee already are using it you'll hopefully add a little bit to your understanding so so I think we started working on Kafka or this story starts roughly in in 2009 I was on the data infrastructure team and I was responsible for a lot of the Hadoop rollout I'd previously had sort of a promiscuous relationship with different open source projects and had done like a distributed key value store and other things in the company the the thing we were really trying to do was get a complete copy of everything happening in the company in Hadoop we had also like a relational data warehouse which was Oracle at the time and became teradata later so this is that kind of you know data lake vision of having a complete representation we didn't have spark at that time just because I think it didn't exist but that was really the goal we were centered around and this turned out to be much harder than I thought and so so you know we had we had planned his you know spend a few weeks on getting data into the cluster and then spend a lot of time on processing and fancy algorithms that would exploit this and of course like I think as a lot of people had found the plumbing problem is still not a hundred percent so then then we spent more and more iterations on that our initial tempt in that area was really to just kind of gut it out like just do integration with different systems you know get it into some kind of good form in the Hadoop cluster and then hopefully that would solve the problem and you know the feeling was if we just worked really hard with this we could like get through it and I feel like a lot of the working like et al or data plumbing has had that mentality as if you just get enough people and they all just like right enough scripts and it will be a soft problem and you know it turned out that really wasn't stalking the problem after spending a fair amount of time on that approach we realized like we had about fourteen percent of what was really happening in the business like we did some kind of let's write down a list of everything happening in the business and let's try and figure out what percentage we actually have a representation of and we were really only solving the problem for Hadoop there was a bunch of other systems in the company it went graph databases and search clusters and relational data warehouse and everything we did for Hadoop was really went off for Hadoop so into the kind of problems we had like the data coverage problem was one just like the many integrations with different systems dealing like messes of data formats and then probably the biggest one was just you know the whole company was changing all the time it was evolving people were writing new code and keeping up with all that change was a big challenge and so this this kind of led me out of this area of infrastructure which is always solving these well-defined problems in building you know core abstractions and into this area of like data and how it flows in a company and what that is all about and so I spent a fair amount of time just doing a you know inventory of what was happening at LinkedIn what was happening at other companies how did they solve these types of problems what things worked well what didn't work so this was the state of data flow at LinkedIn in roughly 2009 it's not super pretty and it got better but I think it's worth just going through what was happening and and what were the problems with those so we had a bunch of different types of data we had like data in databases which is probably what many people would think of as all the data but but it wasn't the only thing for us we had relational databases where there was multiple mechanisms for getting data there was like real-time polling for changes there was like log replication there was like CSV dumps these these all happened on like different latency scales somewhere fast some were high throughput there was also like key value stores and we had some mechanisms to kind of shovel data in and out of them but for a web company there was other stuff there was user events and by events I mean kind of like logs I mean like what's happening in the business like what are people clicking on what are they searching you know what's happening and I think for a web company this tends to be the kind of like clicks impressions join searches but every company has this type of data and it is kind of a core thing which is a little bit neglected by data infrastructure right this is what is the business doing and so you know in a different type of business it might be stock trades and stock ticks and market information and that type of stuff right but but for a web company it's going to be you know clicks and impressions and our approach to this was a pretty low tech at the time which is I think we copied all the logs to one place and at the end of the day we basically like our sink to them over and loaded them and usually whenever you see our sink in some system diagram that's supposed to be highly reliable and robust and you know the critical foundation many things usually usually there's going to be some paint so we also had and I should mention that was like a purely offline thing at the other day data got copied over and it really was only available to the data warehouse and then to Hadoop by some egregious hack and then finally we had these kind of like logs in operational like app metrics like jamek stats and unix stats and those were kind of very real time but only available in limited systems and then we had kind of a bunch of different messaging you know things primarily activemq but they were just kind of here and there like some application would have a messaging setup and some other application would have a messaging set up and there was nothing shared between these it was really just a way for that application to kind of talk to itself and so each of these things like they kind of made some sense taken on their own a few of them were really bad but but mostly on their own they kind of made some sense but when you put them together it was really kind of a big mess so it was you know something like this but actually much more complicated right so in reality there was hundreds of databases there was about 300 individual services this was replicated over a number of data centers and so you can imagine just like going into the multi data center world was pretty awful if you think about how data flows and how that had to be like improved and as each of these kind of pipelines grew with the company it just had its own problem so we were basically solving this scaling problem over and over again and then the team i was running that was you know in part focused on this hadoop project you know we we were basically having to scale up to handle all these different data sources and integrate with these different things in the reality was we just didn't cover a lot of what was you know in the business and so that was totally unavailable for analytic processing whatsoever and when we did cover something it was a bunch of manual work and it was you know boring to do and it was hard to keep it working and so coming from an infrastructure background I guess you always you know hit a problem with the mental hammer that you understand I was looking for an infrastructure solution an infrastructure solution is usually one where you spend a lot of work upfront and then a lot very little incremental each individual problem so I wanted it to be the case that we could kind of cover a lot of what was happening in the company without this kind of like gutted out really hard you know data set at a time you know like almost consulting team we would have to build to do it otherwise and and to the the thought process we went through was really take all these different types of data and all these ways of shipping data and see if there was some kind of common infrastructural pattern and the the idea we came up with was trying to represent this stuff as streams that a stream is really actually a good core abstraction for you know data in a business you could think of you know clicks on a website as you know a stream of click events you could think of log files as a stream of log activities you can think of changes coming into a database and this one's a little weird for people but it's actually a really powerful representation so you can think of changes coming into a database as a stream of you know updates or final States to that database and you you could actually think of the integration with online systems like how do you feed a search cluster or how do you feed a replica database or how do you feed a cash you could think of that as something that subscribes to that stream and like indexes it in some way but you could also have like applications that tap into these streams and give you some kind of analytics on them and and you can think of like this whole warehouse analytical world the kind of data warehouse area broadly speaking which might include Hadoop and relational data warehouses or red shift or you know all kinds of technologies you could think of that as another subscriber maybe on a slightly slower time frame and being able to come up with something that would act as at hub and would actually you know interconnect things on these different timeframes that was really the first goal the second goal was to support this area of things that would kind of plug in and processed in real time because usually your goal isn't just to get something from point A to point B it's usually to get it from point A combine it with you know things from point B C and D do something with it and then get it too pointy okay so that was the problem we wanted to solve and this is what we started out thinking about we set out to solve this problem really with off-the-shelf messaging systems we were running primarily activemq at that time we were interested in running rabbitmq because it was written in Erlang and we thought that that would be cooler oh and we thought you know like this we didn't know anything about messaging systems it was basically a team of people who'd worked on distributed databases but we thought well you know this is clearly a solved problem these systems have been around for a long time so we'll try and use them and maybe they don't have like a perfect distributed setup but maybe you could build some kind of sharding layer on top like you're like the equivalent of sharded my sequel and we could build our like stream data platform that would hold all this streaming data on top of these systems so tried that and it didn't go great so we spent a chunk of time on that project and we found it was just basically a terrible fit for what we're trying to do so for these high-volume log and activity and metric streams the throughput was just like not not sufficient the integration with batch systems like Hadoop was not good enough primarily because the way these messaging systems handled persistence was not good enough so you know if you brought down your Hadoop cluster for maintenance and brought it back up a few hours later you were effectively storing the entire activity stream of like a very large consumer school website during that time period and the messaging systems were not meant to do that they would have said this very small buffer when that filled up they would basically back pressure but you can't like back pressure your users and be like come back later we can't log your stuff so that didn't work finally they were like a pretty poor basis for stream processing for any kind of richer application integration that doesn't just process like a single you know message at a time and the the reason for that is a little bit involved so I won't go into it in great detail but I'll touch on it the kind of ordering guarantees were not very strong and that was a big problem for and indeed like the delivery guarantees in general and that was a big problem for any kind of important database changes so if you have a search cluster that's you know indexing data coming out of databases it's very important that that search cluster end up with the same stuff that was in the source database if you rearrange the updates out of order and maybe you update your account twice and the second update comes in first and the first epic it comes in second then the search cluster is not represent it's permanently out of sync and that's probably not going to work and so getting things in the right order and not losing updates was pretty important and then being able to partition and scale this out was really important so that that kind of led to our second attempt at this problem which was to build kaka this is our kind of reinvent the wheel approach and you know naturally we gave the same time estimate that all engineers give when they don't know how long something will take which is three months and we're still working on it so that was wrong but we did have something out in roughly five months that worked and then there was a lot more to do to make it better over time and so I'll give kind of a 30-second overview of kafka I think a lot of people in this audience probably know that you know at heart it works like a messaging system there's a cluster you produce messages into it there's consumers who subscribe to those internally it's really different so it doesn't really work at all like an activemq or whatever internally what it's maintaining is a log of changes and this log is you know each of these rectangles is meant to represent a record and records are kind of occurring over time and the old records are on the left and the new records are on the right although I guess that's just a convention rights always go on the end you don't change the old stuff and readers are always at some position in this log going from left to right and so this this abstraction is almost too simple to even be worth talking about but it turns out to be really powerful so first of all you can treat it like you know you can use it for batch processing something then start at zero and proxim is really fast to the end so you can actually do batch processing right off of the log you can also subscribe to it in real time and treat it like a messaging system and you can do anything in between so you could have you know a Hadoop cluster where you replicate data to it you know into HDFS mostly in real time but if the Hadoop cluster goes down then that reader is kind of stuck at its position when it comes to backup it catches up pretty quickly and so so it's kind of nice that you have this control over where you're at it also turns out to be a really good basis for publish-subscribe messaging which is not obvious but if you think about two systems or applications maybe one of these is a application one is a new cluster or something they're each at one of these records and I've kind of given each record some kind of logical number in kafka terminology this is offset and you've got a reason about the state of this system at least for this partition of data by what offset it's right up to so in sometimes this system is at time seven or offset seven and this is it time eleven or offset 11 and so you can kind of reason about how these different parts of your company are progressing forward in time as they replicate these changes and that's pretty much everything there is to know about Kafka except that if you take a bunch of these and you put them together that's basically a Kafka topic and a topic is one of these like feeds of data and so these partitions are you know internally replicated for fault tolerance when you write your writing to you know a leader which replicates the followers and then it tells you your message is committed when it's committed we've guarantee you it won't be lost if the leader fails and so each of these kind of logs is fault tolerant and then they're spread over a cluster to kind of scale out the amount of data that's it's handled and so our goal was really to be able to combine these different use cases and that kind of came up with these different things you have to really achieve to make that work so for logs or operational data you really have to have scalability which is kind of comparable to a file system maybe you're not quite as good but you should be not too much worse or it's not really going to fly and so that means really hundreds of megabytes per second per server in throughput and being able to store lots of this log data per server and this was really kind of where a lot of the messaging systems fell down was actually on both of these dimensions we want you need to be able to provide strong guarantees so that you're not losing data in flight between things or definitely some uses like some monitoring systems are okay with the lossy pipeline maybe but a lot of use cases actually aren't and so if you can't hit this you basically aren't going to be able to handle a lot of use cases and then finally we personally wanted something that was distributed by default so something that was built to run as a cluster something that you would think about as a cluster when you wrote you wouldn't think about oh I'm writing to machine you know XYZ 123 you think about I'm writing to the cluster and presumably they're sending these rights and I think if you get that right it actually allows everybody to operate at a much higher level of abstraction than if you really have something that feels like a collection of nodes that you interact with and manually configure things for so this was our goal to be able to unify these different pipelines and and I think we did we at least hit those requirements and so this is kind of stream data platform idea we ended up building it using Kafka took a fair amount of time at LinkedIn we you know moved each of these use cases over one at a time so you know user events and then logs and operational stats and then database changes and then a lot of different like application-specific messaging and processing things that would kind of show up in this area and the whole kind of offline pipeline and and at that point we started to focus on you know our second our second idea which originally was more of a hypothesis like a venture bet which was that this stream processing area would actually be interesting or important and if we captured a bunch of data as real-time streams it would be useful to process it as real-time streams and as you can see in that picture before a lot of data was not available as a real time stream and if data is not available as a real time stream you don't process it as a real time stream so if you if you have data only via some kind of nightly CSV dump then you're going to process the nightly guess if you dump at the end of the night and this is something which is very obvious but is actually often missed which is you know if your if your data collection is batch oriented then your processing is going to be bachelor anted if your data collection is streaming in some sense your processing may be streaming it may not be you know that's up to you and so you know we believed that if we somehow made the entire set of everything happening in the business available as a stream people use that it turned out that was true the example I used to illustrate this is the US Census if you know how the US Census works I think the program is in the US Constitution if I found correct and it works by basically every ten years you will kind of walk around to each house and you count the number of people in each house and then you add up all those counts and that is the total number of people that is obviously a very batch process the streaming equivalent of that is going to be journaling births and deaths and so if you journal births and deaths then you know a you know how many people there are in the country not every ten years but continuously but more interesting Lee you actually know how many people were in the country at every previous point in time which is kind of an interesting observation rather than just knowing every ten years you actually know every point in time the state of the world and this kind of understanding actually led us to realize that stream processing is a very general thing when you when you talk to people about it who are aware of kind of this current state of the world with stream processing systems they often think it's this kind of weird law see you know thing that just fits in a little niche and it kind of doesn't get the right answer but it's faster and the batch systems get the right answer but they're slower and that's the trade-off that is you know probably a correct statement about a lot of the you know current state-of-the-art systems but is not a correct statement about what's possible in this area and this area is moving real fast and so you kind of want to root yourself in what's possible in the area and you know what I would say is actually these stream processing systems are actually potentially a generalization of a lot of what people do in batch processing and if you're aware a lot of what happens in the back end of companies is basically big batch jobs that turnover once a day I mean a lot of you know corporate America works um and you know if you haven't ever cured period under the covers I can tell you to so and so the way to understand this kind of idea that it's a generalization is most people are familiar with request-response processing this is your classic like client-server thing you know there's a rest service you give a request to a service it gives you back a response for every request there's a response it does it really quickly or maybe it times out your request batch processing I think everybody knows this is you know you you have the big table with all the data you do a big scan and compute something and so you kind of take all the inputs and you give all the outlets right and stream processing is just if you if you give a knob that goes between all inputs all outputs and one input one output and you let the program itself control that so the program kind of operates continuously and controls what's output and it turns out that a lot of what people think are restrictions on this area are actually just restrictions on the current technology and the technology is getting better real fast and so I think this is a very exciting area there's actually a really great O'Reilly blog post on this by one of the people working on google data flow which is one of the good stream processing systems so if you're interested in that whole idea around stream processing you can go check that out I think it was posted in the last week roughly so what does what does Kafka have to do with stream processing when I said you have to get some kind of stream of data to be able to process it but you know the way you can think about stream processing and Kafka is kind of like UNIX pipes right so anybody who's used UNIX is kind of familiar with these pipes you can kind of take any program you can throw a pipe down and you can pipe it to any other UNIX program and you can channel a lot of these together that was obviously a fantastic integration mechanism for the days where everybody logged into a central computer and there was just the one probably in the modern world there's you know programs are first of all distributed they run continuously and so they are going to communicate over networks a little different than what you need out of UNIX pipes but the concept is the same which is you know you can have these transformations which take the streams or continuous feeds do some kind of magic maybe they're taking two different feeds and joining them together maybe they're aggregating maybe they're filtering maybe they're transforming and they publish out new feeds and you can have lots of these in some sense this transformation is just code there's a bunch of frameworks though that will make that type of transformation a lot easier so the relationship between Kafka in and these frameworks it's kind of like Kafka's and sometimes giving you streams these frameworks are in sometimes giving you processing and together I guess you'd have stream processing this is this is my math formula so okay so what are the frameworks you know there's a bunch of them storm was probably one of the most popular earlier on spark streaming is really popular down super awesome there's one that came out of LinkedIn called Samsa there's actually not represented on this slide there's a system called flink all of these actually integrate pretty well with cough gun are really commonly used with Kafka okay so if you put that together if you put together the stream processing with a lot of the other things in that kind of stream data platform picture this is a type of architecture you end up with this is what we ended up with I'll illustrate a few things about how this works so first of all we could capture a lot of what was happening in these applications in the business coming out of databases and that would all kind of go into the central Kafka cluster in reality this picture of course is more complicated than any you know has all architecture diagrams are they they hide things so this would be replicated across data centers and this stuff was all on its own analytical data centers but I'm kind of showing the the idea rather than the complex reality so you know in this kind of central platform you could replicate all these different types of changes monitoring data database data event data you could take anything that ended up here and you could transform it into new stuff and that new stuff would look like you Kafka topic just like the original stuff so from the point of view of the subscriber they don't know who's sending the stuff they don't know whether it's like original stuff coming out of an application or some transformed thing and then that said two things so first of all a lot of these you know real-time processing things we feed back into the site so we go back into some database for real time serving that was super common you'd go into some kind of user facing things so like if you go to the LinkedIn homepage there's a whole bunch of updates about your network in some sense that's a big real-time processing flow on top of these update events that are coming out that's a kind of news feed on the homepage it also fed this really rich set of like real-time analytic capabilities and this turned out to be a collection of systems rather than a single system so the whole security and fraud area was basically a big Kafka processing area this is detecting any kind of bad things abuse spam scraping whatever kind of abusive things people are doing a lot of our real-time analytics and monitoring was in that domain right so these these were all systems it would take these changes replicate them into some kind of store you know so some kind of time series store or some kind of search store and then you could do different types of queries so you could you know you could do like log search or we had a fantastically cool application that took all the different calls to services in the real time stack all the errors that occurred anywhere in the business and all the information about the devices in the data center and it basically did like a giant distributed join against all of these and then you can compute like for each you know user request what machines did it hit what services did it call and then how much those machines cost and then you could say well if this machine costs you know two thousand dollars that would divide up over all these requests and so you could say well the cost of this page view was this and the cost of this page ooh this is pretty cool and all of that actually kind of sprung up in an unintended way in that we knew we needed to get stuff between these things we knew we needed to get stuff in this area but we didn't really realize like how much there was in this intermediate area so this comes started out nascent like it was more just like a pipeline and it turned into a whole platform it was cool and so that platform ended up getting used at pretty fantastic scale so so you know a trillion messages written per day since it's each one of these feeds is multi subscriber any right you do is probably going to at least at least to dupe and at least a few other things you know fairly large amount of data in the kafka clusters and this is kind of the backbone for a lot of the data stores like how data got into surg how crowded the social graph that newsfeed kind of homepage and actually working on making this the kind of commit log for the primary distributed key value store which is a pretty cool project that LinkedIn is doing and it became the basis for all that type of stream processing so that was the that was the journey at LinkedIn it turned out linkedin wasn't the only person using this we open sourced it as I guess you guys know since a lot of you guys are using it and it's it's become pretty common across a lot of big internet company is also a lot of enterprises outside the kind of Silicon Valley crowd which has been very exciting to us at about nine months ago the team at LinkedIn left and you know we decided to start a company around this we felt this kind of view of data as a continuous stream and having some platform you could plug everything into that that was actually a big deal it was certainly the biggest thing I saw come out of the kind of infrastructure area you know in my time there and we felt like hey this is something a lot of people could benefit from but it's not something that's going to kind of build itself requires I'm going to help and so so the company's been around for nine months we're basically kind of putting together the rest of the platform i felt linkedin had beyond just apache kafka as well as you know doing a ton of kafka development to get all the security features and you know make the management of cluster easier and all that kind of stuff if you're interested in this area there's a number of resources that have more information I realize this is kind of a light breezy overview so we have a blog at confluent which has just a ton of writing around streams and streaming data so you could you could probably spend several days just reading news blog blog posts in particular there's a blog post we did which was just on this stream data platform concept which is like having a central platform for streaming data and how that works and a lot of like practical aspects of how to do that whereas this talk was more you know brief overview so you can check that out there was a long blog post on this log concept which O'Reilly you know wrapped up as a book and there's a there's a ton of a patchy Kafka resources here and you know like everything everything is on Twitter of course and that's it so thank you very much