Scale By The Bay 2018: Neha Narkehede, Keynote II: Kafka and the Rise of Event-Driven Microservices
you Thank You Alexi for the kind introduction hi everyone I'm really excited to come back to scale by the way and really excited to see a much larger audience so thank you for you know joining me at this talk so before I get started I know Alexi mentioned a little bit about the audience and your familiarity with Kafka but I'd like to start this talk with just one question which is how many of you have heard of or used and used it in production oh well okay 95 percent all right this wasn't a trick question I actually asked her because I don't have what is and why we built it in the talk so I'm really glad I don't have that so why don't we get started so today I'll be talking about something I'm really excited about helping this and you know this is about how we build applications right like what is missing in the kind of applications we've been building so far and how has Kafka enabled the vision for the change and that change is really event-driven architectures and event-driven microservices and I think those are on the rise and I'm what I'm going to do is introduce you to this concept you know talk about where Kafka is enabled that shift and then also give some examples of you know practical applications in the industry that are moving to this paradigm so why don't we get started so you know the place I want to start is by thinking from first principles right you know what is a digital business and a digital business you know one way of putting it is it's largely modeled in data and software right so if you think about what a digital business does it is really a series of events and responses to those events so what do I mean by that you know some examples are let's say credit card transaction happens some software triggers its processing maybe a sale happens some software triggers its billing maybe you request a write somewhere and some software triggers it scheduling so when this happens do that in response is actually a very natural way in or digital business is built and it functions right so in that sense these events that cause actions in a business are really everywhere right so a digital business can't just be modeled as static tables of state you know you need those tables of state but you also need to model the dynamics of how that state changes over time sort of the what happened part right well that's the reason why events are everywhere in our digital business so events are everywhere you know a good place to start is actually by defining what an event is a little more concretely right so the simplest way I like to define what an event is is actually just something that happened right an immutable record of something that occurred in your business what are some examples well I'd like to take an example of the retail industry because we all kind of buy things in stores and more so online the events and the heart of a retail business if you really think about it are really two things right there are sales of things that are happening and then there are shipments of products that are arriving and the backend of retail you can only think of something that places order successfully right it does of course place orders but there are lots of things that are happening behind the scenes in a retail business it is repricing products it is moving products from distribution centers to stores it is actually shipping and fulfill for filling the orders it is also you know raising alerts when there's some kind of delay in the supply chain so all these activities are also just as important as that transaction processing right to make that sale happen and so these things you can all you know these actions you can think of as sort of stream processing right processing the core events that form the foundation of the retail business and the core events are just the same and the shipment spring so events you know they of course are current retail and that's the example I gave but that is not the only place events occur they are basically everywhere else as well so these are examples of events from the rest of the industry there's probably some kind of sale some kind of invoice a stock trade an aspect of a customer experience you can almost you know naturally think of all of these examples as events why because they're supposed to happen all the time you know these things in a healthy business should not never stop happening so these are just events that keep on occurring but the word events is actually much more powerful than all of this especially for us developers and that is because they're closely related to applications right so events also capture changes in an applications state so for example if you store your application state as a role in a database and let's say that row gets updated that change event is actually an event that is produced by application now the reason this matters is because like going from this old state to new state this change event is actually really fundamental to how you build event-driven applications or how you even build applications in let's say an event-driven world that is actually the idea underlying new application paradigms like event sourcing and CQRS where you keep the source of truth events around to decouple the reads from the writes with the core purpose of having the ability to recreate that state of the application at a later point in time or to recreate the state of one application in another with some kind of transformation now the same notion of events and this is really peculiar the same notion of events that powers these application paradigms like event sourcing is actually also at the heart of database replication and distributed systems and this is the interesting part is if you think about database replication the database commit log is just a series of change events and those change events are changes made to the database and the database also uses this change log a series of events to create a replica copy of itself so that's how fundamental the topic of events is this change log and this stream table duality writes as you can see in this picture the ability to turn tables into a stream of events then replicated elsewhere and do the reverse which is turn that stream into a table that's what I call the stream table duality so these two things the change lock and the stream table duality is actually at the heart of Kafka's design and you know the reason we did that is so that you have the capability of doing stream processing on any kind of source of truth log without changing the source of truth so you can essentially produce any view in real time in a streaming fashion on a source of truth event long so this matters because you know this picture shows databases but what Kafka does is a generalization of this concept of the stream table duality so let's assume that the left side of this picture is still a database the right side of this picture doesn't have to be it could be in elastic index where you create an index on the same source of truth log it could be a Cassandra database where you change some fields or drop some columns and produce a view on that log or it could simply be a graph on our dashboard where you're ingesting data from a Kafka log and you know projecting it as monitoring dashboard and so that you know stream pass is saying that generalization is really what Kaufmann aboard and the purpose of enabling this generalization is really to bring these you know two very different worlds together right Alexi earlier mention that you know the one of the focus of the conference is building data pipelines that is definitely how Kafka started but the reason we built it is so that we could you know bring these two different worlds together one world is applications they're mostly thought of as transactional applications and then the other end of the spectrum is really the data world and this is where all your data processing happens and the heart of Kafka's concept is really to merge these two very different worlds together because you cannot really separate data from applications and you can separate you know applications from their data in so Kafka is a unification of these concepts so you know with that context in mind if you take a step back and you think about this events are also the most natural form in which data occurs you know like I said all these examples these are all things that are just happening the only reason that we've been thinking of data as static snapshots is because that's the technology so far and that's what it has trained us to think of data as static stores but not as a continuously flowing stream of information which is really what it is at its core right so if you follow this concept that Kafka is really originated which is thinking of all your data streams what it enables is you know you can represent all if your data are streams of events and that is really the bold claim we're making is whether your data is communication between your microservices as events or it is your applications logs or it may even be database change capture logs these are all you can think of all of these as events and the power is actually this you know the stream processing bubble in the middle the power is that you could actually process the this data as events no matter where it originates from so you could actually join events that are produced by applications with change capture logs that originate from databases and put these two things together to build new kinds of applications right so that's why events are so fundamental is because they're sort of like a universal language for continuously evolving data and Kafka is just one system that enabled all that so turning to you know the next part of the events concept is well events are so fundamental I've been saying that but then where are they right so we haven't really shown up in infrastructure for a long time in any kind of explicit way right so the fundamental kinds of infrastructure and paradigms we know events haven't really shown up in relational databases as a first-class thing or events haven't really shown up in RPC systems in any kind of first-class way right in fact if you look at your code you have to go hunting for where those events are they're sort of implicit right you almost never you know usually never find an object that is named an event so you know this is in part a problem we're also trying to solve it Kafka is to give events this fundamental form in which data occurs a real and permanent home in a business so before we look at you know what to do with events one question might ask is well why have they been hidden right like if they're so important where have events been so one way of looking at it is well let's take a look at what we have so far right like we have tables you know relational databases give you that we have applications we have caches we basically have all these places where data goes to live and data gets stored right in fact much of the logic in a database even is to hide this fact that data is even changing so the explicit representation of events is the part that is missing so in order to make events explicit it's important to kind of define define more concretely you know how they might even occur right so to help with that you know let me show the events take two forms you know one form is that events act as kind of triggers this is most naturally what you might think of events as is you know it acts as a trigger to go do some work right so the simplest way to explain this is if I make a request to you your HTTP service and it goes into some work and sends me back a response I've actually you an event to trigger some work the other way in which data occurs events occurs is actually in the form of data streams right and this is really what Kafka taught us to think of events as is you can actually keep them around the they can live in a ordered series of events as a log and if you did that you know the same event that triggered some work you could keep it around and you kept it around then you can use it in some form or fashion you can use make make filters on it or aggregations on it and you know these two sort of aspects of event processing coming together you know unifying events has triggers and events as data into one concept and allowing you to do stream processing on that is the other thing that Kafka is enabled and I'm going to come back to this concept right so before that let me give you an example of what I mean by events as triggers so let's take the same retail application as an example and say that you know i microservice here application emits an event every time someone buys something right so my events are basically who bought what events I have another application that takes those events and it joins it with the change capture stream from my inventory table and if I join who bought what with the inventory then I basically get the latest inventory count in my stores and that is the stream that these two joins would emit and then I have another micro service that just ingest this updated inventory count stream and if it falls below a certain threshold then it triggers some kind of reordering of inventory right so if you think about this whole flow it's a bunch of applications that take some stream and they do some work and maybe produce some events or just trigger some business logic so this is basically you know using events as sort of triggers you know you just send some information so some downstream application could those could go do something so these are event-driven applications now the second example I also said is well events could also be your right so what do I mean by that in the same company let's say we take a pretty common use case that I basically say everywhere is you know building this customer 360 application right so you data about your customers could be you know they could live in many types of databases and many types of systems and the job is really to unify that and join that so you can create one 360 profile to know about what your customers do so if I were to build that application then I would take the same sort of stream like who bought what but this time I'm trying to glean like customer buying preferences right behavior of the customer from it and let's say this you know this data pipeline is basically taking that and joining it with change capture streams from a bunch of customer databases and there you go I have a unified customer you know 360 profile stream and let's say be stored in some kind of downstream Cassandra database that you can stream and serve it from now if you look at this picture right this looks now much more like a data pipeline it looks much less like a bunch of applications right but if you closely take a look at it and take a step back you would realize that these two are actually equivalent right in both cases have taken in streams of events joined it with a change capture stream from databases to produce a new stream of events the fact that you can take data from applications and data from data systems together and do some kind of stream processing on it is really the crux of the event of an architecture so this the question really is well if stream processing is the general paradigm that a bunch of digital businesses are now getting built on then where does it happen right like where can the stream processing happen where can these streams of events even live if they originate in all kinds of places and so the answer to that is that's where you know the streaming platform is where these streams of events live the streaming platform is where they both live and where that stream processing happens and Kafka is one such streaming platform right this is basically our vision for how a modern digital company would get built is you would have streams of events coming from all sorts of places from the applications from your logs from your monitoring systems from your databases and you could have all of those streams of events be fed to all these other systems and also be used to join with each other to create new streams of information to create new vendor in applications now this is also just you know a very different way of thinking about communication in your company right so this in fact just flips it on its head and turns it inside out instead of me having to go look for all the different things that have to respond to my event I just flip it around and I say this is what happened in my area the business these are the streams of vents and anyone who's interested in those events could just you know subscribe to it and go to that work and build new applications and this is really a much more scalable way to even build your company because it essentially decoupled different parts of the business different business units and even different kinds of people from each other it's really different in the sense that you really don't have to know who else is interested and wants to consume an event all you say is this is what happened and events sort of are the way of communication right in a business so for a long time you know events sort of lived around the periphery of a business right they actually never were found a front and center and that is actually now changing quite a bit and it isn't changing because you know events are a great idea and they be coupled microservices and people and they enable real-time so let's all do events it's actually happening because there are all these different trends that are now happening in all sorts of businesses whether it is the rise of IOT or moving from monoliths micro services or the emergence of mobile and machine learning adoption of public cloud if you look at all these different trends you know one thing that they all have in common is the concept of having access to high fidelity data and having access to that data in real time and that is one of the major reasons that you see this rise of adoption of Kafka and this rise of adoption of Kafka for very serious konna mission-critical applications is there almost certainly used to enable some of all of these trends in that business some examples that I get to see that I thought were pretty cool is you know event-driven architectures are in the heart of very traditional car companies now their vision is to I think likely become Tesla but really to collect all these you know sensors sensor information from about hundreds or thousands of sensors on your car's onboard processor and trigger lots and lots of applications and a business from driver alerting to traffic monitoring to you know repairs and avoidance of accidents and repairs this is also happening in the banking industry so event-driven banking is essentially transforming everything from making your credit card processing real-time so for some time I didn't know that it they only process it only twice a day with Kafka they are gonna process it event at a time this is also transforming how fraud detection happens so you know using Kafka to essentially track a customer's actions across the globe and detect and alert frauds so this is really happening in a bunch of different kinds of industries for real you know this I would like to share that the last is really you know where does a business event go if it becomes to leave interval right not in some corner of the business but truly you know bunch of your applications are written as event-driven applications this is much of how Silicon Valley's companies are even built is that there are a bunch of possibilities that are unlocked if you know business is truly event-driven first off everything is an event from the get-go right so whether it is logs or business events everything is available as an event it is available actually instantly instantaneously to all the applications and data systems in a company you have the ability to query data as it arrives event at a time and respond to it versus when it is too late all your data is available across all your systems so there's a single source of truth rather than all the different kinds of data pipelines that could diverge over time and the the best part of this is you know this is possible by actually simplifying your data architecture by removing all the bespoke connections and cleaning it up with one platform that could scale to the you know needs of an entire business and organization is by putting a streaming platform at the heart of a business and that is really the power of streaming platforms so with that I'm going to end the talk and actually you know have a few minutes of questions if we have time Hey yeah that's a let me repeat the question the question is you know what what are my thoughts on long term event storage so the use case is if I want to backfill a data store let's say I have a new data store that I want for it to catch up from the history of the beginning of time then how do you make it happen so a little bit of history on this for those who are not aware Kafka started by enabling a couple different kinds of retention policies so one was you know just forget anything older than X you know hours or days or weeks right so with that the question is well how can you keep data for an infinite period of time because you will just run out of disk space or you know Kafka brokers may not be designed to store all the data on one broker but over time it evolved and enabled a feature that databases have which is what we call log compaction so in if you turn that on then a Kafka log actually behaves like the back end of a database like will change capture log where it compacts the latest value for every unique key right and so with that feature you can essentially say that Kafka can store infinite amounts of data because it only scales to your key space which is not very large right so that is very widely used actually today as a feature where businesses turn that on for any information that has keys in it and they just keep it on forever because Kafka takes care of compaction so if you had such a log then you could actually store your database change log the customer updates in a Kafka log and just keep it on forever the advantage is that actually Kafka is built to you know enable this kind of you know ordered series of reads right so you can take an elastic store and point it to offset equals zero and it would catch up to the history of all periods of time on your customer information right so that's the most common use case but off late and going forward there's another use case where people just want to keep data in Kafka forever because you know retaining more data wouldn't actually cause issues in caucus performance characteristics so a thought on that is this something confidence-building is in our concept of tiered storage right so this has been around actually it's not a new idea it's been around in file systems for a while is cold storage goes to much cheaper object storage and your most recent data lives still on Kafka brokers so what that would enable is essentially if you use Kafka in a public cloud then you get truly infinite storage whether it is keyed or non keyed information you know why would people want to turn it on one is to make Kafka more elastic so if you want to spin up more Kafka brokers and have it load balanced information it has very little data to move because rest of the data is somewhere else right and the second one is convenience as an organization starts to adopt Kafka and it adopts more and more of it it just becomes a convenience thing because it becomes that source of truth data storage system and data movement system so then people just want to keep it around so they could actually seek to the right point in the log and build whatever application they're trying to build so it's debatable whether you want to use that or not but that's happening any other questions all right well I'll be wrong for a couple of minutes if you want to chat later thank you very much [Applause]