Keynote: Apache Kafka, Stream Processing...

scala.bythebay.io: Jay Kreps, Keynote: Apache Kafka, Stream Processing, and Microservices

Recording: scala.bythebay.io: Jay Kreps, Keynote: Apache Kafka, Stream Processing, and Microservices

you thanks everyone all right can everybody hear me even in the back yeah I see not all right and um I this this is going to be an interesting talk because I'm I'm not going to do kind of an overview of coffee I'm going to talk as much about stream processing as kafka itself so so do people have like basic familiarity with Kafka like you've heard of it yeah probably like 50 percent how many people use it how many people use it in their company in some way yeah pretty pretty good number cool so what this talk is going to be about is actually kind of the intersection of two things so two things that maybe people mostly would think don't intersect so it's going to be about stream processing and it's going to be about micro services and I think probably your gut instinct is these are just like two different types of fruit right the one is about kind of like a you know real-time map reduce the processing streams of data analytics thing the other is more like yeah we had this big monolithic application we had just split it in two parts and we put some rest services in place and so you know what is this part hopefully that will be the question I'll answer so it turned out that I actually spent a lot of time with both of these two things so you know prior to starting confluent which is where I work today I was at LinkedIn and you know kind of at the very same time we were doing two things one was splitting up a giant monolithic application into hundreds and then you know god help us I think over a thousand services the other was rolling out in building Kafka and making it kind of the central backbone for streams of data and so you know I can talk a little bit about how they you know interacted and that that process actually this process of taking a monolithic application splitting it actually had pretty significant pain it was associated with it it worked eventually so the goal you know the goal of splitting into services was we would be able to scale our software engineering organization and that happened right so we were able to go from a small number of engineers to well over a thousand but the the pain really came in in operations it came around latency it came around uptime and it came for like a really obvious reason which was we had a bunch of applications and they called each other in some giant graph and as you hire more engineers this graph of services that call each other gets split more and more right you're splitting services really to make people productive so it has as much to do with the number of people you employ and you know for whatever the kind of triggering request was maybe you would go to linkedin.com and we would need to make a home page and that would call out to a bunch of services that would fetch data and do different things as part of that home page in order for this to actually come back and give you your home page everything underneath would have to work right and at first we did this in kind of a blocking way where we would call one thing and then we would call another thing and then we would call another thing that was kind of a disaster and eventually we did it in a non-blocking way so we would call things in parallel but still for that chunk of work to be done everything in that graph had to complete it had to complete successfully it had to complete in a way that everybody understood and that became incredibly difficult to keep working as the number of pieces went from you know a few a handful to like a thousand right you can kind of think of it almost like probability like if if you have five things what's what's the chance one of them is going to go wrong at any given time is it's probably not - not too high if we all are diligent and do our jobs if you have a thousand pieces there's always something breaking at any given time and handling that complexity ended up being kind of the core challenge of making services work and we got better at it we got you know a lot of monitoring infrastructure that was really good the service layer got really good we got really good at building non-blocking rest services but but one of the things I think we came to realize and it came in part because we had Kafka and so we have another approach for building things as well was what we were doing was building things that were synchronous right they might be non-blocking but they were synchronous this this request this homepage wasn't done until every else underneath it was done and yet often the work we were doing was not synchronous it was things that could happen in the business later right so if you do some action on a website and it needs to update some of these news feed that doesn't need to happen while we're making the next page for you those things are like disconnected and so the lesson out of this was really to try and come up with some way to make things that were asynchronous actually asynchronous make things that in the business or in the world where asynchronous happen asynchronously in other words we wanted to be able to take some sub graph of graph of this and make it just happen later make it not be part of that interaction and that that's the way you actually lop off you know the hundreds of things that are in a given request context that's how you actually reduce it right and so so this became actually a kind of an important insight and it was true there we ended up with a lot of things that were being driven off of the Kafka stream and somebody would be triggered as basically actions and so the question is like how common is that is there a lot of parts of business that are asynchronous that happen asynchronously and I think the answer is yeah like if you look at other businesses I think they're actually maybe even more asynchronous so if you look at a retail business when you're buying something that's like a synchronous interchange right I'm like here's some money you're like here's a product that's a pretty synchronous interaction I don't really want to give you my money until I believe there's at least a promise of a product uh-huh but the rest of the business everything else that happens maintaining stock in distribution centers moving that around shipping stuffs to stores restocking and inventory management reordering adjusting prices all the analytical backends you know the meat of that business is all asynchronous and that's actually not that uncommon that there's in any in any company there's a chunk of it that happens while your customer waits it's interactive and there's a larger kind of back-end that's happening independently and so the question becomes okay great we like this idea of taking you know our software and splitting it into small chunks we can work on indepen and deploy independently what's the mechanism for building a service that it's asynchronous how do you how do you do that and so you know the kind of traditional answer at least for people who you know have some kind of background with service-oriented architectures or anything like that is you have a message queue right so you basically get a message queue the application put something in the queue and the applications that need to react process that asynchronously so does this work does this make something asynchronous the answer is yeah it kind of works I mean what is the message queue doing it's basically promising to store the data until it's process why is that important because this person can't go away until it has some promise that the work is going to be done right and so the message is actually do do something reasonable but when we were trying to actually put them to work we were trying to take kind of traditional enterprise messaging systems and put them to work in a big consumer scale website it actually didn't work well at all for a number of reasons right one message queues tend to be like one machine so you're like basically putting your thing on one machine if you're trying to do that in a way that scales across a large company with lots of requests you don't kind of one machine you're gonna have a lots of them and all that interaction is very complicated secondly the whole promise of this thing is basically storing your message until it's processed and it turns out a lot of these messaging systems are like the world's worst storage systems it's like the worst database ever made okay so that's not great so kind of the theory of Message Queuing it kind of makes sense but the practice was not great especially if you look at an area like databases where a bunch of smart people have tried to make it really good over many decades with every university in the world participating in that research that has not happened with message queues so our vision was a little different we wanted something where we could actually take these streams of data where we could store them while they were being processed asynchronously something that could both you know combine data coming out of relational databases event streams coming out of applications and it could serve both apps that would process stream process as well as a data pipeline for things that you do this was kind of our vision of what you know next-generation messaging would look like breast and in LinkedIn to architecture maybe the prior state was something like this where you have a set of request response things that all call each other you have a bunch of relational databases that serve stuff that eventually became key value stores you have kind of a big batch thing that rolls over once a day in the back and then the connections between this we're just kind of a massive is just everything kind of connected where it needed to via different ad hoc mechanisms and what we move to over time was something that looked like this where you have kind of a central streaming backbone or message bus thing that connects everything and that's the point where applications the kind of request response domain goes into the asynchronous domain right anything published there it becomes that's the asynchronous boundary right you put a message here applications can be triggered off those messages they can do things they can take some action they can put stuff back it also acts as a kind of feed of data for downstream systems that need stuff so that was the role for this and these applications what are they doing right they're not just taking the message they're actually doing something with it they're doing some kind of processing on this stream and that's kind of what leads me into the stream processing part of this so you know I I'm going to get into this I'm going to say a little bit what I mean by stream processing because I think most people's inclination is it's like we're going to have MapReduce but in you know faster and I think that's maybe like a technically correct in some way but like the wrong gut instinct and it'll lead you to like think about it wrong right so I'm going to I'm going to make an argument that the you know by stream processing I actually mean something more like an event-driven micro-service something that responds to a stream of events that processes it and to kind of flesh that out let me just go through how this compares to the things you know so there's really kind of three paradigms for programming cue everybody's familiar with one is new I think most people are familiar with request response this is you know how your rest service works this is how you know this is client-server this is like JDBC I send a request I get back a response I send another regret so I get back another response it is you know one input gives me one output and if you you know do programming professionally this is there's a good chance you spend your day making something that works this way so much so that you may not even think about it as that thing there's another weird thing about this paradigm which is when you build a request response service it is always there to process future requests so that that sense it doesn't really make sense but when I start a service it doesn't go back and process the past it processes new requests which will be sent to it yeah right like yeah but actually batch processing the opposite works the opposite way in both in both cases so batch processing is all about processing the past data the data that was stored from yesterday the things that already happen not the things currently happening now right and instead of taking one input and giving one output it's going to see all the inputs kind of turn on them and give all the output so batch processing this is your Hadoop job this is you know when you when you took your first computer science class if you did they make you write some command line thing that takes a file and filters it and spit something out that's batch processing and a shocking amount of the backend of big company still still runs off with and and so what's stream processing going to be I said it wasn't quite just batch processing faster it's going to be a little different in a couple ways so like request response processing stream processing is going to be handled you know future requests it's this infinite stream of future inputs they're coming it's going to handle that like batch processing it's going to allow you to go back and process the past it's actually going to stitch those two things together so you can go back and process old things and you can just keep processing as new ones come and unlike the two examples I gave instead of taking one input and giving one output or instead of taking all the inputs which wouldn't really even make sense if you have an infinite stream about inputs coming to you you're going to be able to take some inputs and produce some outputs and the program will get to decide what some means right and that means that you know you could on one extreme be like a request/response processor we've reached thing that comes in you produce one thing that goes out but you could also be like a batch thing where you take really a lot of input before you produce any output or you could be somewhere in between and that that's somewhere in between is what's going to give you control over the latency that that you know output comes with it's going to give you a lot of control and generalizes to areas so stream processing instead of being kind of a weird niche thing is going to be actually kind of a generalization of these two ideas and and to emphasize this what I what I don't mean is something that's kind of transient or approximate or lossy so if you ever hear about streaming algorithms they usually mean something like this I actually just mean a data system that's going to process you know both the past in the future stream of requests that will be unbounded in size I don't mean it will get the wrong answer and in this example I gave you can see kind of the streams if you squint and I think this is true about most companies if you kind of squint it what a company does there's some kind of inputs and there's some you know state machine of processing happening and some outputs of what the company is if you look at this retail example there's these kind of stream of sales shipments of products and then you have some processing that's occurring where you're saying okay you know we're selling out of this we should reorder we're selling out of this we should register price we're not selling any of this we should lower the price there's a whole back end streams of like analytics you know the reordering events of hey get me this that would all occur in reaction to that and so I think this idea of viewing a company as a set of streams connected by processing is actually kind of powerful all right so that's what stream processing is what does it have to do with Apache kafka which I also put on the subject of the slide let me try and say so I showed this picture I said you know we wanted something that would connect different data systems it would allow us to build applications that would respond to streams of events and you know that something was basically Apache Kafka that was the motivation for that system and Kafka I'm going to give like a super brief sketch for the a third of the audience isn't super familiar you know what it is you can think of it as being called like a next-generation messaging system although it works very differently fundamentally the thing it's keeping track of is a stream of events or records or messages I'm going to use those words kind of interchangeably just because people do and you can't make them stop a message for us is going to be a timestamp when it happened it's going to be a key and a value the key and values in Kafka or just byte arrays you would probably have some kind of higher level representation that would have lots of subfields in your value would probably be a rich record it could be JSON basically it doesn't matter for the purpose of this talk this is what the fundamental thing in Kafka is is streams of these records or events or messages and Kafka will organize these into a log or into many logs and what a log is is just a sequence of records right so it's like a commit log I don't mean a log like you know a the requests log for Apache where it logs out to some text file I mean like a binary you know file that stores the records that were appended so always you know new writes getting written to Kafka are being appended to the end of this log write the first write is here each of these little boxes is meant to you know represent one of those records that I talked about each record is given a unique number which is called the offset and that's kind of the point in time that that record happened so lots of lots of different applications could all be writing at the same time and they would just be appending appending appending to the end of this log and you can read this log there can be many readers each at some point in time reading it so you could have zero readers for the stream of messages being appended or you could have many and each each reader can be at a different point in the log so kind of not that not that not that crazy it's actually a very simple structure turns out to be a good basis for published subscribed message here so inside of many distributed databases you would basically see this style of structure being used to distribute messages within the cluster it's usually kind of an implementation detail inside a lot of databases you would see something like this but basically in messaging terminology you would call the writer a producer and the readers consumers and if you made them separate processes now you have publish/subscribe messaging so this is the fundamental data structure in Kafka it turns out this is really important for stream processing so stream processing I said we were going to be able to kind of unify with batch processing a little bit right so how does that work if I have a log of messages like this it's clear I can process this in many ways right the reader controls where they're at so you could have a reader which starts at the beginning runs to what the current end is stops and is done that's basically a batch processor right you could also have a reader which is kind of an incremental batch processor where it wakes up it processes everything that's there so far it produces some output it goes to sleep it wakes up the next day it processes the new stuff that goes back to sleep you can also have something which is very real-time which is basically subscribing to things as they come in and processing things as they occur right and so the key thing is we can have one data structure one view of our data which includes both past stored data new stuff coming in as as reality happens we can unify all those in a single ways so the logic you write that processes the system that does processing it doesn't change whether you're talking about the past or the present or the future things that haven't happened and a particular important application of these types of logs where you would see them all over is in databases so if you've heard of like the my sequel bin log usually you know unless you're like a DBA it might just be something that's like currently hurting you like oh we had some problems we've been log replication like under the cover a lot of databases are basically keeping a log like this and they're using it to replicate the sequence of updates that occur and this is common in distributed databases it's common in relational databases so you have some kind of source system which is you know basically keeping a log of the modifications and those can be replicate it out to derive databases which just apply those changes right and so these you know replica databases are going to be in sync with this but maybe just a little bit behind they may not be quite caught up but they'll go through the exact same sequence of changes so this mechanism has been around forever and this support for you know replicating a log of changes is actually very core Kafka has built in special support for this type of semantics of blogs and in Kafka the primary data structure is called a topic that's basically a feed of events that's a stream it's basically just a partition set of blocks so for people who already knew Kafka that was probably boring for people who didn't raise their hand before that was probably not enough information but it'll set up the rest of what I'm going to talk about so Kafka basically has four api's that let you deal with this type of stuff it has a producer which I kind of talked about which is allowing you to basically write and append to these logs it has a consumer which allows you to read so you can read you know usually you would read from left to right from old to new but you could always go back and read again if you need to which turns out to be very important as a connect API which allows you to kind of suck data out of existing systems or write it into existing system so it's kind of like the opposite in the push-pull dynamics and then it has a streams API and the streams API is what's allowing you to do processing on streams and that's what I'm going to talk about for the rest of this talk this is relatively new it's about maybe nine months old ish so in this picture I showed I said well we want to be able to have applications or stream processors that somehow react to the stream of things that are happening maybe they're taking changes that are happening in relational databases and transforming them in some way maybe they're reacting to streams with events published by apps and they are outputting some kind of derive stream so this is kind of what the streams API was motivated by let me kind of go through the state in the absence of this of how we saw people doing it I'll talk a little bit about how we were motivated to this API so the first thing we saw people doing was you know kind of bare-bones processing of streams of messages they were basically using Kafka like a messaging system there was a way to produce and consume and so we kind of call this like hipster stream processing because it's like kind of back to basics you don't need much hipsters since this is San Francisco everybody knows but you know they're going to favor the more primitive tool right so if it's a bike it's a bike without gears if it's a computer it's actually a typewriter if it's a PDA I think it's like a little pocket full of cards on paper with fountain pen the nice thing is those tools actually work pretty well they're not complicated they don't fail in complicated ways if you've ever ridden a fixie the changes come off as much and that that's actually true for this approach to processing as well so what people were doing is that you know you write an app you basically consume some stream you do something with that stream of data and maybe you produce back some kind of drive stream and so this is a way with without you know any particular support in Kafka for processing you can actually do stream processing and the only problem is you know a little bit like the fixie bike kind of eventually come to a hill and then it's like oh this is why we have gears and so the hard parts in the hills in stream processing are going to be a lot of questions around state so what happens you know if if every input produces some kind of derived output in my output stream that's easy but what if I need to like count things over a window of time and what even is time if I have stuff that's happening in the past that's combining with things that are happening in the future what does it mean what what am I even using as the clock that's deriving time and how do i how do i chop that up into windows I can process and then what do I do if I like change my code and I need to go back and like recalculate my results in some way maybe I've made some kind of fraud detection application I've detected a set of fraudulent people when I change my rules that calc you know calculate fraud I need to recalculate the set of fraudulent people how do I do that and then a bunch about how does this work when it scales out what are the semantics of this fault-tolerance work these are kind of the hard problems and as you see people kind of dive in with these problems they would sometimes get these right but they would often get them wrong that's kind of the definition of a hard problem is you don't necessarily get it right on the first try so the other the other approach for stream processing with Kafka is there's a whole bunch of frameworks that do something kind of like MapReduce on streams in some ways so there's SPARC has a way to do stream processing you know kind of by dividing up the stream into little batches and processing them there's a system storm we had built one at LinkedIn called Samsa flink is something newer they're all to varying degrees attempting to do this and the challenge we saw with a lot of these was just they were relatively complex when you actually looked at the full set of things you would need so you know if your idea was hey we want to take our big monolithic application we want to split it into chunks we want some of these to be asynchronous and we want to use stream processing to like build the asynchronous ones you can't introduce that much complexity before you just like fall over and die right the whole point was you're trying to make it easy and agile for people to work and the kind of standard stack for a lot of stream processing things with something like Kafka which would fork off data some kind of distributed stream processing framework that would really deploy your code so maybe you would write a storm job and you would deploy it into a storm cluster maybe you would have some kind of separate distributed database for keeping track of any kind of state that would occur so if you're doing count so if you're joining data together anything that covers more than a single row you probably need some kind of database that's output is going to and then finally your app would work with that and then if you ever want to be able to go back and kind of recalculate things or if there's any kind of inaccuracies in this flow maybe you would have a whole batch system that would do the same thing so sometimes this would you know some variations at least on this would be called like the lambda architecture so maybe you would run some kind of daily batch job that would recalculate things you would kind of reimplementation Spann in some way and you would produce out put that on so this works people do it but kind of you look at this and you say okay yeah we have all these services we want to migrate a bunch to this it's just going to be relatively complicated right so our primary goal in the streams API was to try and simplify it and make stream processing into something that could be a mainstream approach to application development that's simple enough to build and run things and carry a pager for them without you know wanting to hang yourself so we basically like this picture we just want to give you more convenient ways of processing the streams of data so the the key operations that the streams API is going to give you everybody who is used Scala would be familiar with them they're their kind of things you would see in something like a scholar collections library I guess lots of languages have something like this where you're able to chain lots of operators together and perform some kind of transformation on the data you have so what's different why wouldn't you just use the Scala collections API we're really trying to do a couple of things so first of all the scope is now going to be across many instances of your application so obviously if you have a collection in Scala it is just like you wouldn't even think of it but of course that collection exists in memory probably on a single machine almost certainly right our definition of a collection is going to be across a pool of machines right when you call map that mapping is going to happen across many instances of the application secondly the collections are all unbounded in length right so the list has no end and so you have to have very careful you have to be very careful with a lot of operations you would take for granted where you call you know size on a collection right it's an infinite collection and your approach to calculating the sizes to traverse and count that's going to be a slow operation so the core the core operations are going to be something like map which you know you take an input stream you transform it in some way and you produce an output stream filter you take an input stream you get rid of some stuff aggregate you take an input stream and you combined records in some way counting something etc and join where you're either joining two streams together or you're joining side information to a stream of events so these are kind of our core concepts there's a bunch of variations like on these a Kafka streams program I'm going to show a an example this is Java code there's basically some configuration and setup the actual logic and then the end the logic is doing this is Java code by the way so apologies but that was screenshot I had the Java code is basically doing word count right and it's doing word count on a stream of documents it's taking them it's splitting them up by some whitespace thing it's counting the number of occurrences of each word it sees and it's outputting that to some output topic in Kafka right so what's kind of interesting or or new about this what's new about it is you're able to I mean first of all nobody cares about counting words but you're required if you have any kind of big data system to give an example that involves counting words so what does this do for you how does it help you well basically what it's allowing you to do is take this and run it over many instances of the application the the individual application is nothing fancy like you can notice this is just a main method so it's just invoking this library which is allowing it to coordinate what work it's supposed to do and what works the other instances of the application that are currently running need to do and if one of those instances dies it's going to take over some work and if one of the instance a new instance comes it's going to donate some work for it to do so it's handling that coordination this is not different than what the Kafka consumer itself would do but it's doing it in a way that actually handles the semantics of transforming data and that's it that's it so so the kind of hard problems are basically that like how do you split up the work and then how do you manage this state that is computed off the stream particularly if that state gets large so if you are counting the number of words and there are many words you need to be able to spill out of memory and do that effectively so so what you get is basically something that kind of in the world of stream processing is actually a cutting-edge stream processing system it has kind of the set of features you would expect but it you know it operates as a simple library and there's three core simplifications I'll highlight so you know the first the first of these is that we're basically using Kafka so what do I mean by that I mean it's basically all the same concepts that you would have in the stream itself so the data model is the same it's these key value pairs with timestamps that that offset which controls what you have and haven't processed your position in the stream is the same concept the management of back pressure comes from Kafka itself the management of scaling out this processor comes from the same group management feature that you know Kafka consumers use all the management of state and partitioning is the same so from the point of view of the Kafka cluster anybody using this streams API it's just like another application using a producer and consumer but we've we've handled a lot of these harder problems for you and in particular the reprocessing capabilities if you write an app and you need to kind of go back and recalculate your state off of the old data that's very easy you basically just rewind it in time and you process the stream essentially in fast-forward right so normally stream processing you would think of it as path processing the current state if you have in my fraud example I gave earlier you have some current set of fraudulent people if you change your code that calculates that how do you get the new set well you just rewind and process reality very quickly you could you know temporarily scale out and have more instances while you're calculating that if you want to the handling of time is built-in so and it uses Kafka's concept of time the offset marking your position the timestamp captured with the message and this will do you know true event time so even if data arrives late from mobile devices from other data centers whatever it will be calculated correctly off the time of the event the next simplification is basically you don't need a cluster so this application uses the library and it acts like a producer and consumer and Kafka does you need the Kafka cluster but you don't need any kind of like stream processing cluster it's not like you set up some kind of magical Kafka streams daemon you don't launch your code into that everything about how your code is packaged configured deployed that's just totally up to you it's how ever you do that with other services so I think that's really important anybody who has ever used one of these frameworks that starts take over the actual deployment of code for you and a lot of these frameworks do it seems simple at first the example always looks simple but then you realize that whoever deploys your code actually dictates a lot about how it works and so it becomes you know it's like you kind of meet somebody you go on a first date and they're redecorating your apartment right away it's a little bit like that where you know I I ran into this with all the Hadoop frameworks where it's going to end up dictating a lot about how configuration works the lifecycle of your code everything about how you program deploy and do things that's really not ideal so I think a big advantage of this is basically you start and stop and deploy programs the way you normally do all this framework does is handle the scaling out of processing and contracting and handling failures and the stream stuff that means of course this works with the full deployment coolness spectrum so you know whether you are basically are syncing your code out onto servers via some shell scripts or you're using something like super cool and like kubernetes either will work with this framework like basically it doesn't matter we don't do that part of it the final simplification has to do with actually supporting operating both on streams of data but also cables so tables of current state why would you care about that well I gave this retail example in this example there was a stream of sales there was a stream of shipments but there's a current state that you're calculating off that which might be your inventory on hand and in order to decide when you need to adjust the price or when you need to real order you need both you need your inventory on hand and you need to know what's selling right now right and and this is actually very common this this kind of current state and we've done this in a very principled way emitted a first-class citizen of these api's to handle that and so I gave this example of database replication a really cool example of this table facility is the ability to take a stream of changes out of one or more relational databases materialize it into some kind of view in your app and basically keep an up-to-date materialized cache or view of things that are happening elsewhere in the company that come together so an example of that might be something like you have many slivers of information about your customers split across many services and databases producing the kind of Universal customer profile or customer 360 view or whatever you want to call it that can that's that's an example of this and that state in this latest release can actually even be queried so that that unified view you could replicate it into an external database but you could also query directly the application and just check its current state and of course I talked briefly about the connect API which allows you to run connectors to other data systems this will work in concert with that so if you think about ETL in the data warehouse space in some sense you know the extraction of data and the loading of data into other data systems is happening with this connect API the tior transformation is kind of the the streams API and so that's how all those pieces fit together in this streaming platform vision what we've been doing at confluent is basically trying to package a bunch of those things together into a distribution that lets people get started so we produce a distribution of kafka that basically has all the clients for different languages has like rest access for things as connectors to common data systems off-the-shelf has some management tools as management for schemas and the compatibility of data and that's it I think I'm almost out of time but maybe I can take a few questions [Applause] yeah so his strange growing into something more like playful it's going to be including the compute processing accessing data per state yeah I mean that that's right so so I think the at a high level yeah both of those things which are things which allow you to process streams of data in implementation it's quite different I think because basically what we're giving you is more a library or API rather than a kind of code execution framework so maybe the use cases are a little different because of that but yeah they're both allowing you to do processing on streams of data so not respective recently so in what I think you know streams over yeah that's right so so the question is is all about what's the difference between flink and Kafka streams or perhaps to any other stream processing there I mean they all work right so there's a whole ecosystem of things that will kind of run on top of Kafka what we're trying to do is give something that is very simple out of the box that's part of the Kafka cluster so you know where I think that that is most appropriate is for use cases which are kind of in this core you know business logic or service area I would say that's probably a little different from the kind of analytic space where maybe you're you're building kind of a you know something that could have been a batch MapReduce job but you want to run faster I've just seen the teams that work on that stuff it's usually different the way you manage that code is usually different where I think we shine is basically you know with the streams API is probably you know areas where you are you know maintaining that like a service you want to deploy the advantage of this approach is basically you deploy it on your own machines you know whatever they are you totally own the life cycle of that code there's no kind of shared processing cluster you have to deploy into your only dependency is basically Kafka next question if you'd say a little bit about the two concepts of the table in stream yeah when you go for a table to extreme I can kind of imagine like three semantics like two of them are very closely related one would be like here's the latest value that has been written to this table second one which I think strictly better is here's the latest value and previous value for that vacated room and the third one I think a lot of people actually want but doesn't seem very convenient lose the ability to occasionally scan the entire table periodical some way of programmatically saying okay actually I wanted to do something that's not just going to update or yeah yeah so the question is all about you know more about tables one of the semantics of them yeah this is super deep area there's probably a whole talk just on that um the semantics in Kafka are you know of the three versions it's the first so what you you can basically think of a stream as having data and the semantics of that are kind of in the eye of the beholder so let's say I have a stream and it represents customer profiles or customer account information right and every time you change your account I'm basically posting to that stream the new customer account there's two different ways I could represent that I can treat it as a stream and if I count records on that stream I'm going to get the count of customer account changes right so maybe if I count by geography I would get something like customer changes by geography if I treat it like a table I'm actually going to be computing this on some derived thing which represents like the current state right what is the current customer profile for each customer which is the current customer account if I count on that I'm going to get like how many customers do I have in California not how many customer changes and the difference between those two is kind of in the eyes of the beholder you might want one you might want the other so we've given you kind of a flexible view of that in kafka there's a whole set of features that do log compaction and stuff in the background to make this work but the key ability is you can take things which are maybe the master data is in a database a relational database you can capture changes out of that which is like oh here's the new value of it you can process that in a way that makes sense or you can have a stream of something which is a pure event like hey we had somebody click on this off of the we had somebody click on this which is a pure stream you can compute a table of the number of clicks per hour per user or something and you can process that table incrementally as well so what you can think of this is you know as being is the ability to compute something derived off of a table like a table in a relational database but keep that up to date incrementally coming back to your original question why do you use the latest state instead of one of the other variations the answer is actually because a that unifies it with the other concept upstream so the records look the same B most systems will only give you that I do agree it's useful often to have the prior state but you can get that by you can get that by maintaining it in your stream processor if that makes sense so if you want it you pay if you don't you don't have to the reason you pay is because you have to keep the old thing and not everybody has that and and so it's better to have you pay a down stream than require it up stream since many people can't do that other questions okay well take anything offline thank thanks everyone [Applause] you