data.bythebay.io: Jay Kreps, Building a Real-time Streaming Platform Using Kafka Streams...
so I guess we're in the basement talking about data plumbing and there's like data plumbing I'm sure running right there okay so I'm gonna be talking about Apache Kafka I'm going to be talking about streaming platforms and I'll be talking about a couple of new things in the Kafka universe Kafka streams and Kafka connect so how many people have heard of Kafka probably a good portion of people in the plumbing data basement here okay good so my plan was aha okay the down-arrow doesn't go next okay so my plan was to just gonna start pre Kafka and talk about data streams and event data then then give kind of a super brief overview of Kafka because I suspect a lot of people are already a little bit familiar then I'll talk a little bit about how you can get streams with data into Kafka and how to process them with Kafka streams and I'll talk a little bit about how this kind of ties together this notion of a streaming platform so I'm going to start with streams and I'll try and address the question of why streaming you know I guess I'll start and actually make a bold claim that you can kind of think of all your data as event streams and some of these are kind of obvious but let me start with an example and actually say what I mean by event so by event I just mean some kind of record maybe it has a timestamp associated with it maybe it has some different fields this this event is a product view and it's in JSON but it could be represented in any way so so now I'm going to run through a couple of examples of different types of data and let me try and make the case that this this idea of events and streams of events is particularly powerful so the first and maybe the most obvious is some kind of sensor data so you know this is happening out in the world where we're collecting you know sensors that are sending back like these kind of event records but it's also happening in people's applications if you have if you have metrics or something you collect off your servers then you're plenty familiar with sensor data and of course the log files are naturally events like you can think of each line in a log file that's being an event something that happened maybe as a record it's a little bit messy but you can say okay one of these pageviews was in Tibet but maybe a little bit more interestingly you can actually kind of think of databases as being streams of events and this is actually a very unusual way of thinking about a database probably when most people think about a database they think about a table and probably when most people think about a table they think about something that's you know kind of rectangular and has some columns and has some keys in values so for my example I'll try and walk through how you can think of a table as a stream and then I'll tie this into the rest of the tacos as we go on but what I want to demonstrate here is that even things that aren't most naturally thought of in this way are actually pretty applicable when thought about as as event streams so how could you think about a table as an event string well if you think about it a table you know this this is my table I have some keys and values but this is really just my table at some point in time like if I look at it I might see this but if I look away and look back I might actually have some different data there so really a table is more like this at each kind of point in time I have a different value for my table so maybe my table you know at time zero is this at time one maybe it gets updated again at time 2 it gets updated again so really what I have is is actually a sequence of tables and I could actually represent those sequences of tables by the sequences of updates right I could I could instead of capturing like a snapshot of the whole table at each time I could capture just the update and if I had this stream of updates I could think of it as being in some sense the equivalent of this table as it evolved over time and if I had this stream of updates I could actually recreate the table so if you gave me the stream of updates I could make a table and if you gave him the table I could kind of capture this stream of updates and turn it into a stream right and so this this mechanism for translating between a table and a stream this is actually this kind of deep thing in data systems people sometimes call it the table stream duality so I guess if you if you want to sound smart and feel like you learned something at the conference you can say well there was a bunch of plumbing stuff that I learned about this table stream duality okay so this is my case for streams being super brought they cover kind of natural event like data they they cover kind of messages between applications and they cover also like stateful things like databases so what are they useful for well there's really kind of two uses for these streams of data one is building Teta pipelines which I guess is our subject for today and the other is being able to kind of tap into these pipelines and be able to transform the streams or process them or react to them in some way so you can build something comes like a service that it taps into this and does processing and I'll talk about this type of stream processing in this talk so how does this actually work in companies how the companies actually won these functions well in practice I think it's often kind of a giant mess like you have kind of ad hoc data pipelines between things you have you know kind of point-to-point communication you know you have basically a bunch of things that are kind of shoved into a batch processing context or some kind of request response context even though in the business they're actually asynchronous and so I'll talk a little bit about how this idea of data streams can kind of clean up this picture and you know the first thing to get your mind kind of into this mode of thinking is when somebody says data pipeline think of an event stream so all those arrows on that big messy picture before you could think of those as being some kind of ad hoc event stream maybe just a poorly specified one and the key idea behind Kafka and the idea I'm going to dive into is it you could have some kind of central platform for these streams you could collect them in one place they could all work the same way they could be represented the same and this could connect both you know data systems where you're maybe connecting to capture streams of data or connecting to load streams of data but it could also connect to like applications that you know react to the streams or publish messages into it and it could also you know connect analytical systems that do monitoring and so on and there's some you know benefit to this type of streaming platform this type of event centric thinking so I'll walk through an example maybe I have a web application and I want to capture my product abuse using the example I just gave and I want to get those into Hadoop for analytics so you know I could publish them into this streaming platform I just talked about and I could love them into Hadoop and in this model it's not really clear what the benefit is but over time as I have more ways of capturing product views right the nice thing is the Hadoop system doesn't really have to know about it right sorry I have some kind of pub/sub model where Hadoop knows it's interested in product views these things that you know they make them but they don't really know about Hadoop and they're kind of decoupled and if more things end up using the data then they can do that without having to connect to these things as well so instead of everything kind of connecting to everything else you can say well I subscribe to the product views I know what I do with them I do something for the business these things say okay I know I produce product views but not everybody has to know about each other so this isn't anything really new it's just kind of like pub/sub messaging but if you can do it at the kind of scale that you would need to really run a whole company around it becomes kind of a different model for treating data and this this is kind of this streaming platform idea that you could capture all these streams that you could have them available to everything else without everything that creates streams of data having to know about everything else so that's that's kind of my introduction to event streams now I'm going to dive into Apache kofta kafka is a system that was kind of motivated by that worldview and you know it is kind of a at heart meant to be you know the thing in the center there the thing that collects all the streams that kind of multiplex system and you know it is kind of a cross between a pub/sub system like a messaging system and maybe something like a file system so it's a little bit different from a traditional messaging system and it actually stores a persistent log of what happened so the fundamental abstraction and Kafka is this type of log and it's basically a sequence of events it's a sequence of Records and each of these records is given you know some number and this type of log it's it's more akin to like a commit log than like an Apache log and you know it's it's readable so it has an API to subscribe to it and it's writable but when you write you just append and these events are all immutable so it is you know a system that's built around directly modeling this idea of events and making them something that can you know fan out and be subscribed by lots of processes and this turns out to be a pretty good implementation of scalable publish/subscribe messaging and this is basically Kafka's data model it's you know a Kafka topic which is like a feed of data maybe all of those product views and the example I gave it's basically just a partition to set up locks and writes are gonna be multiplex over these logs and reads are going to be you know off of this set of logs and their samples even this this notion of replicating changes so a really common thing to do with Kafka is to publish database updates right so as updates come in to some database it's publishing them into this log and then other replicas systems can take this and so the replicas could be other databases or they could be search systems they could be applications and won't get into that but in this case the you know record that's being captured is the kind of key value pair ously this is the key of what was updated this is the new value for that update and I'm gonna talk more about this type of change lock as I get into it and the final bit of Kafka is allowing you to have groups of processes that consume these streams so Kafka has a notion of consumer groups and this is a set of processes so the a is in this picture are you know a consumer group of processes and the B's are a consumer group of processes and you can have this group subscribe to some feed and this allows the processing to scale beyond the capabilities of one process so and it allows these groups to be fault-tolerance so if one of the is dies the remaining is takeover its work and processing if a new a comes along and joins the group it'll take over some of the work and so those those right there are basically the core abstractions in in the Kafka cluster and the trick of what you know Kafka does is just try and implement this as a full-fledged distributed system with fault tolerance and a strong consistency model and security and all the things you would kind of expect of a modern data system and Kafka is open source and wide a variety of companies in different industries so so this is you know kind of what we were aiming for was to make it something that could work out this kind of company scale in this type of streaming platform and now I'll dive into you know two related pieces of technology that are part of the Apache kafka project one is kafka connect and the next is Kafka streams so Kafka Connect is all about these arrows so I drew these kind of nice lines which makes it look like it's very easy to get data in and out of Kafka but what Kafka connect is is actually our attempt to make that really easy to do so when we built Kafka we built it obviously with api's there's clients in different languages but fundamentally if you're integrating my sequel to capture changes into Kafka you're probably not the only person in the world doing that and it turns out that there's actually a lot of shared problems between these types of connectors and what we wanted to do was build a framework called connect that would allow you to build this type of streaming pipeline and take over as much of the hard part of the problem as possible and so this could be either a connector which is attaching to some data source and streaming changes in Chukotka or it could be on the other side it could be you know streaming changes out of Kafka into some kind of data sync right so I'm going to use this sync and source terminology and the key thing that this framework does is it does kind of a hard work of building this type of streaming pipeline so it makes it possible to scale out these connectors so as a connector implementer you just basically tell the logic for how to read or how to write rows to a particular system and Kafka Connect will handle how this scales out across multiple processes how those processes are fault tolerant to make sure that you're always consuming from the system how you manage these processes if you have lots of you know different data sources and data sinks and then also and this is probably one of the deeper parts of the problem how you propagate whatever schema information you have or don't have about your data sources so if you're capturing data from a relational database you probably have a great deal of structured information about what your data looks like you know all the types for all the columns all the names for the columns if you're capturing data from kind of a key value store maybe you have less maybe you just have some byte erase schema but nonetheless you don't wanna lose that information for all the downstream people and so fundamentally you know what this is about is you know being a framework you can build these types of reusable sources and sinks and deploy them and you can you you know so some of the examples that we have we have a hadoop sink and this kind of demonstrates how you can take something which is scalable like Kafka and something which is scalable like HDFS and you can run this connector and scale it out over multiple processes so the pipeline itself is you know scalable and of course if one of these processes Dyess the work will be taken over by the remaining ones and you can dynamically scale the pipeline but you can also view this for something like a JDBC database so in this case you know it's just probably a single server relational database and so you might wonder well why why do you want to have this ability to kind of scale out over processes but what we found is most people don't just have one relational database they usually have a number of them and if you want to actually kind of cover all of these in a way that's operationally efficient you don't want to be spinning up new processes per database and you don't usually you would have to have at least a pair for fault tolerance so this is the way you can come set up this Kafka connect cluster you can dynamically via the rest api add these database sources and it will stream data from you know the database into Kafka and we've collected I think a few dozen now of these different connectors out in open source or once at the community of built or whatever and so you can actually connect a pretty large set of things to Kafka in this kind of off-the-shelf way without having to actually write any code so that's Kafka connect now I'm gonna dive into the next new feature Kafka stream so Kafka Connect was released in the 0.9 release of Kafka that came out at the end of last year and we've been building out that connector ecosystem since then the 0.10 release is basically about to come out so it's being voted on right now should be you know the final release probably by early next week and one of the major features and that release is Kafka streams so if connect is how you capture streams of data streams is basically how you process it how you build applications that reacts to these streams transform them process them and we had kind of a lot of experience working with this type of stream processing framework the team that came out of LinkedIn had built Samsa and we really you know had a very particular thing in mind we kind of got out of our experiences when we were designing Kafka streams and I'll kind of walk through it but I want to kind of start just maybe from first principles and you know ask the question what is stream processing and I'll try and answer that question I'll lead into you know how I think that that motivates our solution so the reason I ask is because people tend to have a lot of misconceptions in this area they tend to think it's an area that needs to be very complicated that tends to be kind of transient and approximate and Loxy so it's fast but it's kind of you know getting the wrong answer and I actually don't think that that's true you know I think I think stream processing is basically really just a kind of third paradigm for programming like one way to split up programs would be how they get their input and how they produce output so most people are probably familiar with request response programs which is like a web server or web service that's a request response process and most people are probably also familiar with batch processing probably if you had a computer science degree the first program you wrote with some UNIX utility that took in data and processes with it and so these batch processes this might kick off kind of throughout the day and stream processing is going to be kind of our hero and I'll contrast it with these and rather than thinking about this as being kind of a limited form of one of these two I'm gonna actually kind of argue it's a generalization it's actually something that can do a lot of what request response processors doing a lot of what batch processors do and here's how so when you think about request response processing you're kind of giving one input which is the request and you're getting back one output which is the response and when you build something like this you kind of run it and it processes you know in some sense the stream of future requests that come in it doesn't you know if you write a web service it doesn't go back and process old requests that might have happened in the past obviously right so this is about you know one input one output future requests that's processing is kind of the opposite you know in a batch process you get all your inputs up front you kind of churn through them and then you produce all your outputs all at once and inherently when you're running a batch process is about all the requests or the inputs which have already happened so it's about processing past data and stream processing in my definition is just going to be kind of generalizing those two things so instead of saying well I'll take all my input up front and I won't give any output until I get all my input I'll say no I'll let the program pick as much input as it wants so it can choose how quickly it produces output and you can choose how much output it gives so it could choose to just take one input and produce one output in which case it's kind of like a weird request response processor and it could choose that it kind of takes all its input produces all its output and then goes to sleep until tomorrow in which case it's kind of like a batch processor but most usefully it can actually do anything in between which and I think this is the most critical part it actually lets you kind of control the you know frequency of processing and the latency that the output gets which actually dramatically you know I think makes a lot more simple these kind of you know data intensive batch processes if you can kind of make them much more real-time and integrate them back into the user experience into the customer experience but the second part of this is it generalizes this idea of processing the past and processing the future so if the batch process was all about you know past inputs which had been you know already written down and recorded and the request response processor was all about future inputs this allows you to you know kind of process from the past you know and just keep processing into the future so it's kind of as if you know you had a query and it was able to you know process the data in the table but just keep updating as the table itself was updated and another way to think about this is you know look you can think of you know a stream processor it's just being a kind of micro service which is event-driven and has access to this past data and has you know kind of stronger semantics and maybe a rest service would so this is kind of my view of stream processing I'm particularly interested in it because I think a lot of the stuff that happens in a business can actually be thought of as data streams so if you think about like a retail example since most people understand retail you can think about there being kind of a stream of sales that are occurring you can think about there being a stream of shipments of products from distribution centers to stores and then in response to this you can think about the stream processor recording inventory adjustments or making price adjustments and you can think about it you know producing output streams such as what products to reorder or detecting fraud or doing analytics off of this and so you can think about even kind of common business processes that were super used to as being these types of stream processors and so there's a couple ways you can implement something like this there's a couple ways you can build stream processing one is just build an application and use the kind of Kafka API right and you could consume a stream of data and you could produce back maybe some derived stream and this works pretty well it's actually pretty simple but there are some hard parts to stream processing you know how do you partition this out what are the fault tolerance semantics how do you use stateful processing like joins and aggregates and how do you reprocess stuff if you're if your code changes and this is you know kind of where people turn to stream processing frameworks like storm or spark streaming or you know the one that we had built was called Samsa or flink and these are all these different ways that you can kind of have some clustered processing that you know helps you write an application that processes streams and we built one of these we built sansa and deployed it actually relatively widely at LinkedIn and what we found was people were kind of trying to build something like a micro service but the way we built it was kind of a lot more like a MapReduce thing you know you deploy your app into this cluster it tries to kind of take over and manage your and so although that worked relatively well for kind of analytical jobs it didn't work that well for these types of core applications that were kind of part of the business so we've started from the other side of the problem to make Kafka streams so what it is is you know it's not really a stream processing framework you don't set up like a Kafka streams cluster it just works with your existing Kafka cluster it's just a simple client library we've done a java version of it so far although we'll extend it to other languages it gives you a convenient DSL it gives you kind of the state-of-the-art you know in stream processing features so it gives you advanced windowing and reprocessing and it doesn't batch data so it works with low latency and allows you to do kind of steep full processes like aggregations and joints but I think the critical thing here is you know it's basically just built around the Kafka API and so it doesn't require any cluster of its own it can basically fit into any existing application that you have and what we were aiming for here was to really take this area of stream processing and make it as simple as possible make it something where you could get you know kind of the capabilities and features you would expect out of a stream processor but without the heavy weight commitment to something that's going to own how your code is deployed and how its configured and how its monitored something that you could just deploy in any way any fashion you wanted and that's what we've done so you know a Kafka streams program looks something like this if you're used to the Scala collections library or the Java streams API it's it's very similar to that except it just includes a little bit about the semantics of streams and serialization and you know offsets and so on but this this is the transformation so you can kind of very concisely specify something that does in this case word counting I think because you're required to count words in any big data example but you can kind of very concisely specify how to do that and this will basically transform from input Kafka topics to output Kafka topics even doing kind of stateful aggregations with updates that come you know late and the way that this executes is you know you would embed the Kafka streams app into your application it subscribes as a Kafka consumer and it produces back derived streams it takes that kind of you know processing that you've specified and it divides it up over instances you can dynamically add new instances to the processing and those will take over some of the work and if they you know fail of course that work goes back so what Kafka streams is doing is all this about dividing up work managing state giving you this kind of higher-level API what it's not doing is any of the like how do you deploy your application so we're assuming you have some way to deploy your application and and the result is this kind of works with the the whole spectrum of deployment things from the kind of very uncool but you know very well understood are sink to the you know very cool but you may hurt yourself distributed deployment frameworks and obviously if you're running in something like may sauce or kubernetes you can kind of elastically scale out your process and it gives you kind of the key operations you would expect you can kind of take a stream and transform it you can filter items so you can aggregate meaning like count or sum or whatever and you can join streams together and then most importantly it extends this not just to streams but also to tables as well so I talked about that notion that you could think of a table as a stream of changes a kind of change log and what Kafka streams is going to let you do is actually process those change logs so in this example of retail I gave where I am keeping track of inventory or price adjustments that's probably effectively a derived table off of the streams and Kafka streams naturally supports that so in this change log picture I talked about how you could use Kafka with a kind of log of updates to replicate data to other data systems but but most importantly with Kafka streams you can now take that stream and not just replicate it as it is but transform it into some kind of like materialized view some kind of new view of what the data looks like and have that be the thing that other systems or applications subscribe to and each of these operations the kind of aggregations and joins they work with both streams and tables so you can you know take something which is a change log and you can compute an aggregation on it kind of a materialized view you can take something which is a stream like the product views and you can do the same and you can join them so you can have a stream of product views you could join on table information about the products or about the viewer and this I think is a key thing that is not that modeled that well in existing stream processing systems and this works well with kafka Connect you can have connectors that run in kafka Connect you can do you know transformations and kafka streams and you know get data back out with connect so you can think of it as kind of being you know connect is kind of the e and the L in a kind of streaming ETL and streams as the T but of course you could substitute other things here like you can you know have your application directly write to Kafka you don't have to use connect or you could use the difference between processing system or just a plain application to do the transformation okay so so those were kind of the two new features that I was interested in introducing kafka streams will have its you know first kind of full release in the next week and I'll talk a little bit about how these come together so a few of us had the opportunity of actually doing this at large scale at LinkedIn where Kafka acted as kind of this streaming backbone for all these different types of data and supported kind of a whole ecosystem of real-time things and this is something that's been done over and over again at different companies so when I showed that open source slide a lot of companies have these really big kind of Kafka based data pipelines or data backbones and you know those of us who worked on this really learned a lot out of how to you know build this type of streaming pipeline what's required to operate it how it integrates into things and so the company that I founded after leaving LinkedIn is confluent and we've been building out kind of an open source platform around Kafka that helps you put some of these things together so the confluent platform is open source we're including clients for Kafka we're including a lot of connectors four different systems so people can get those off the shelf and have them test it all together we're including management monitoring tools rest access and a lot of schema maintenance tools so kind of a full set of things that we felt companies were building in-house we're trying to produce some high quality open-source version of these that people can run against and that's it so if you're interested in this stuff there's more detailed blog so on both connects connect and streams so you can kind of dive in and learn more about them I realize these are kind of long URLs so maybe just Google and I think I have a few minutes left so I'll take a few questions if there are any and then I'll hand this over to whatever sex just a very quick question what versions is dreams going to be compatible with or only the newest yeah so the question is what version of Kafka will streams be compatible with it does require the newest version of Kafka which is 0.10 the reason for that is 0 port 10 actually adds a timestamp field to the Kafka messages and that's actually very important for these windowing features to be done correctly so that you can handle late data and aggregate correctly and so on and so directly it would be possible to make it work with older versions but it's a bit of a hack and you have to kind of do everything twice hi this Kafka stream does almost all the transformation that is done by a regular ETL tool like informatica or any other yeah that's a good question so so caca streams right now is kind of a developer platform meaning you know you have that kind of lightweight Java API in that sense yeah it does do everything that something like informatica does but informatica actually gives you a really nice UI if you're interested in that one of the tools confluent is going to be putting together is something which helps a little bit with that kind of streaming pipeline other questions yeah how does Kafka connect deal with like schema migration data migration if what happens is the data source changes yeah that's a great question so so cough cough cough connect doesn't itself do anything other than propagate the schemas but that schema registry that I kind of listed as one of the open source things confluent produces does do that so you know very often you want to have it be the case that you don't just blindly propagate schema information you also enforce some kind of compatibility so that people can't publish invalid messages or something like that that schema registry I kind of pointed to helps with that kind of thing so you can for some topic say well it's a free-for-all do whatever you want and for other topics you can say no look I want this to be backwards and forwards compatible so I'm only going to allow data into this topic if it you know has the right fields and so on allow changes but they have to be compatible changes in either case Connect just allows the schema to propagate so if I talk about that like HDFS connector that makes it possible to you know have a Kafka topic and have updates and new fields kind of percolate downstream all the way to a Hadoop so the hive table gets updated automatically and you don't have people trying to maintain that by hand that's probably a very short answer to a long question from enterprise side the Kafka is not matured because it has a zero dot something version number and not one point something do you know why is that and you know it's there I know it's pretty reliable and consistent mm-hmm do you know why I didn't promote to did not get broken to one dot something oh yeah I mean the the version numbers are just kind of a open source thing so our take was we wanted to get you know finalized client api's security connected streams and so probably in the next release we'll do that but I don't think it really reflects on the kind of robustness of the software it's used at pretty large scale at a pretty large number of companies usually for that to happen has to work otherwise the companies dump it right so I wouldn't I wouldn't read too much into the version number alright last question and I'm happy to take anything else offline Thanks it seems that doing stateful programming is something like this is particularly tricky especially if you're basically of a consumer group and one machine picks off a piece of information that's needed to compute state and then another machine picks off another piece of information needed to compute a state they each don't have enough to compute the proper state and send then write it to another stream do you guys have you know bits and pieces of software that more or less automatically handle that or is is the programmer basically still stuck being careful making sure that all the all machines have enough data to do the right State copy yeah that's exactly one of the problems Kafka streams tries to help with so I think as you say if you were just blindly subscribing you could imagine trying to partition things in some way that would have the right things end up in the right way but you'd have to be very careful Kafka streams I think makes that significantly easier in that it knows about the partitioning model and kind of enforces that it knows about the state model and so what it's doing is it's basically allowing this combination of partitioning and local state to kind of do what you would expect right so if you have a join the things that are being joined have to be partitioned on the same key so that they end up in the same place so they can be joined together locally and vice versa so so yeah I think that that's actually I guess the value add that something like this has is it helps manage the state helps make it fault tolerant if the processes fail helps make the failover very fast and then it kind of isolates you the programmer from having to you know think in kind of as low-level detail about how your data is partitioned at each phase and so cool all right I'm happy to take any other questions I'll fine thank you all you