Scale By The Bay 2020: Viktor Gamov, Event Streaming with Kafka Streams

Scale By The Bay 2020: Viktor Gamov, Event Streaming with Kafka Streams

Recording: Scale By The Bay 2020: Viktor Gamov, Event Streaming with Kafka Streams

[Music] uh thank you i'm uh super excited to be here uh first time uh first time speaker of this awesome conference long time follower uh and you know i guess uh 2020 dreams come true always always make sure that um you know what you you're wishing for right so let's uh let's get this going uh folks it's uh really exciting i'll try to um feed um as many as much as possible material for for this presentation um for this short time we will like um we'll have a lot of things to cover so without further ado let's do uh let's do this all right so just to make sure that everyone sees the slides um hopefully um i i'm not gonna be checking this the uh these uh the chat um like actively because i will be focusing on doing the presentation but hey um here i am okay so um it's not my projector right it's a screen share so hopefully you'll see the skin share and everything is working um and uh um like everything is clear okay so uh this is talk about kafka streams it's uh i call it kafka streams 101. uh if you didn't know anything about kafka if you didn't know anything about kafka streams it's it would be a good um you know talk to to start with um in my my goal in this talk is just like give you idea um and maybe provide some of the research maybe definitely provide you some resources where you can start learning uh things around stream processing and in kafka and then so far uh so far and so on um i work as developer of the kit here at confluent and uh i like to steal this line from my fall uh fellow developer the kit kenny bustani i build highly scalable and highly available hello world applications i think it's appropriate for this type of con for this kind of kind of conference so hopefully um hopefully this is going to be um useful in entertaining for everyone so let's start with this one we're going to be talking about stream processing string processing is a process of um having your processes or like dealing with this event as they arrive into the system meaning that we're not waiting to accumulate them and after that you know the run this processing at the end of the day uh we're not waiting until um all these events will arrive we don't know when the all of these events will arrive so that's the difference between uh systems like batch systems where you um waiting for this um you know events to arrive and have to do something it's even different from the system that do so-called like micro batching when they waiting for a little and after that they apply the computation in uh in stream processing you are passing this message through the computation uh bag or precomputation pipeline and uh you're doing this for every message and it's up to your processes up to your application that will actually tell you what you know what's going to happen with this with this with this data so we're going to be dealing with this and more realistic example like we do have a lot of events that come in from the mobile devices that come from other systems or from different um sensors maybe even uh from many other place so in order to capture those events and like uh to start uh making sense of it uh we usually capture those in event streaming platform um and uh these days uh patchy kafka is usually you know the default choice for for a streaming platform it provides known uh capabilities for scalability it provides known capability for um you know durability of the messages and so far and so on so we do have these events we captured those events in kafka kafka provides our a buffer that allows our processing logic that will be implemented in in this particular case we're going to be talking about some of the java apps and the java particular things but not necessarily needs to be java something different there's plenty of different tools in different languages or even i can be tools that agnostic from any programming language like k-sql that provides you like sql-like interface to deal with this so and another important thing that's usually um not the part of this processing logic is the server logic this is essentially if you if you've been in this world of the data engineering for a while you know that this architecture is very much looks like um it's couple architecture right so there was a lambda architecture we have multiple layers the speed layer there's batch layer but this is cup architecture where you are fully crossing a layer now a serving layer usually it's something uh using some other system like when you do the processing you'll follow this into the system like uh like uh angel cast the red is uh cassandra elastic something that allows you to fast access for random data but kafka streams as a matter of fact provides these capabilities that you can combine this um certain layer in your processing layer together uh we'll i will talk about this just also a little bit now so um let's talk just very briefly for those of you who don't know what's the patria kafka just really briefly i will introduce to this concept of event streaming platform all right so in the world of event streaming platform um we do have uh kafka brokers that essentially doing the work around storing your data capturing this data making sure that order is endpoint uh making sure that data is replicated properly um as of today as of uh what data is today today is the 12th of november 2020. kafka still depends on zukiper there is a ongoing work around making kafka lighter and or maybe making kafka smarter to put more uh the consensus logic inside the kafka brokers and get rid of the additional dependency that simplifies deployment in in many cases especially in the cloud native world where um distributed system needs to be you know dependent on less less dependency as possible so for now it is there so a usually in order to deal with this uh data in order to integrate uh the system um with kafka you are using the library that talks native protocol and this protocol is highly efficient it's binary protocol runs on top of uh tcp uh tcp ip you don't need to uh implement any kind of like a load balancing logic because kafka as a modern distributed system uh also follows like similar ideas that many other distribution follows in in terms of like how to access to particular data and i'm talking about some sort of version of consistent algorithm right when the clients will have a connection to all brokers and after that um on the data they will figure out which broker to choose to write so next thing is that um kafka essentially don't care what kind of data you stored there you don't care it doesn't care if you put their strings json pictures files i do care and i'm against this but hey um we can have this conversation so um one of the things that uh people started realizing that is java serialization is not the very portable so that's why if you're serializing java object into something that you want to read for example in go language that would be really tricky so in this case some of the intermediate format of serialization should be chosen and this format can be protobuf this pro format can be avro even json maybe okay because people still convinced that the json is a good language for inter system communications for some reasons they say oh it's a human readable i um it's a machine readable but the format is allowed you to use and read this using this editor so maybe some other tools that allows you to read for say protobuf payload that also would be human readable but anyways schema registry allows to store this information and provide the schema for applications when application needed so in this case if your application go in and receiving some message you don't need to have um a whole full-blown information as a part of the payload you can have a tiny payload which is going to be binary compressed or like binary generated enough that scheme register will provide um the the rich viewed on this one and uh on the client you can decompress it and uh or like deserialize it and get the right result uh the native libraries user usually considered as a kind of like assembly language for um or for for kafka applications and i will talk about this and you understand why uh native library producing consumer is assembly language for this kafka streams is a high level api high level api that allows you to focus on what you actually want to do with your data so rather than okay so i'm reading the message and after that i need to write the filtering logic or or transformation logic like over and over or maybe worry about the story state where to store it and so far and so on so kafka streams um we're going to be focusing on the graphql streams today there are some other tools like sql db that allows you to write the application without knowledge about java because it's just like sql you deploy the sql it will run this and it's actually full-blown stream processing application there are some applications that are not necessarily able to deal with the native protocol so for example you want to produce message from curl for some reasons you want to write the message to kafka from curl or you're parsing logs and you want to do this so you can grab this and uh pass it to raspberok so restaurants essentially what it's exactly what it sounds uh it is interface that allows to produce data to kafka without um writing like native application and there's kafka connect that allows you to integrate with any other distributed systems or any other systems for that matter like databases messaging queues and so forth and so on um that allows you to um bring data in and takes data out like when you do processing data stores and kafka back and after that it flows back to um to i don't know data warehouse or some of the uh uh search um storage like elastic or so far so let's take a look quickly on how this application looks like so the producer application how does it look like so the typical idea typical cycle of kafka world we read something data from kafka so we're getting this event uh one at a time so this is why we have this uh infinite infinite loop um and it's okay you know you can say look victor we have a infinite loop um one of the things that uh justifies it uh and uh this infinite loop is actually not a bad thing uh and uh it's a it's an architectural pattern um and uh the good analogy that i heard and i'm using this all the time now um is that kafka consumer is like a shark have you ever thought that if what's gonna happen to shark you shark stop moving like have you ever thought about this like how uh shark will be sleeping it doesn't because if shark will stop it will die same thing happens with consumer if this uh the constant pulling logic uh will stop a consumer will uh will will die but let's uh let's break this code uh real quickly so why we need to have a hashmap because we want to deal with a stateful stream process meaning that we want to have ability to calculate current step based on some previous results all type of aggregations all type of transformation that might have information from from the past will require on count calculating minimum calculating maximum calculating averages all these things will require state um usually systems provide the pluggable storage solution for example you can bring some database but in my this example i'm using like a super super fast somehow fast maybe not super fast somehow fast hash map that allows to um deal with this with the storage and the reason why we're using this and the reason why we need this is um after we subscribe this we run this infinite loop of uh pulling messages from kafka topic and only one thing what we need to do this is actually one logic that we we're doing um we need to improve count so same it's a like similar very simple example of you just want to have a count because it's very simple example of uh stateful processing um and after that once you're done with this once you calculate this you can save it somewhere in this case you have a producer so again logic is simple we have a consumer that reads data process data and produces back so this is what a cycle of kafka application uh looks like many people have this idea that um end-to-end kafka applications should look like producer kafka broker consumer no it's actually another way around so it's a consumer that constantly reading data it's a processing that runs inside the consumer and producer that runs as a as a as a thing that writes a result back to kafka so and after that we need to tell consumer that hey we're done with processing and usually people find with that usually people love this um usually people who probably get paid uh for lines of code that they develop they they love this type of code and sometimes i'm even having a hard time to convince them because people are seeing no reason by bringing a something that simplifies their lives because they already satisfied with some of the consumer producer pay maybe maybe i have a theory that the people uh who chose this path they are uh framework developers and as a framework developer you always need to go in the lowest level possible so instead of developing your own client you probably rely on a producer consumer api and implement framework around this um all frameworks that you will see these days that talk to kafka somehow they are using consumer producer api anyways like in one way or another or they may be even implementing their own protocol which would be no probably but you can implement this but i don't see reason why you would so let's talk about the the framework of this right so the you have framework developer and how you're going gonna deal with this so let me um um show you uh some of the uh code that does the same thing as we discussed the previous step remember constant loop of consuming data from the topic um and uh doing some of this with this data and let's see how it looks like so this case this is uh i can say if i would just strip some of the configuration for example connection to kafka broker if it's like that this is full-blown application that implements kafka streams so this application also reads data from the topic it does account and the save data in another topic that will have a grouped version um now grouped by key um accounts so there's a lot of things uh that this teeny tiny piece of code does and specifically um it allows you to just in a few lines of code just do grouping uh implement some um some of the very often useful uh some of the operations that you probably will be doing very very often specifically group by key and counts this type of thing that would be there out of the box and after that store this state somewhere in case of new data will arrive and you will need to continue this the calculating this the data based on the new data and also you want to store the results so other systems will benefit from from using this so in this particular case that's how you would do this and this is this this is it this is topology that allows you to focus on um how this message would be transformed rather than focus on producer and consumer you don't see producer and consumer here anymore right it is there and uh this framework provides you with different uh cool things i don't know how about you but like i think it's pretty impressing right when you replacing some of this like a code the multiple slides that using with a few lines of code so um why it is important speaking about the frameworks that the people develop like if you would start um developing uh your own framework based on the some of the stream processing framework what this framework should be capable of doing so these days running single machine workloads are it's possible it's uh many frameworks still doing this and it's it's probably okay however it's not um it's not like future proof because um once the load of the data is grows uh you want to deal with this accordingly so framework should be uh scalable um so how you would do this so there's some of the lego blocks that kafka protocol and kafka consumer protocol provides in order to scale this load for example kafka consumer groups and kafka partitions allows you to parallelize consumption so if you have a topic with 10 partitions and you want to read in 10 times faster you start 10 consumers that will join one consumer group on each topic and each consumer will be reading particular partition so the scalability would would be nice to have um elasticity is also a very good point like once you start scaling you want to scale elastically you don't want to small application uh build multi cluster of multiple machines you want to start slow you want to see how does it work and like you want to learn the framework you want to learn how the things are um how the things happen and after that you can grow from the smaller use cases maybe like you just like want to put some of the data into kafka and start processing enough that you can grow by bringing more data sources failure is inevitable uh there's no way how you can avoid failures however is only question how you would handle with this and there's different guarantees that different system provides how a system will behave under certain failures we know we know from different the lesson learned from the scalability and the design of data systems that network partitioning usually always inevitable uh it's very difficult to build highly reliable network um except if you're not in the business of building higher reliable network but in general when we're talking about general purpose frameworks and general purpose data systems um network uh partitioning is inevitable like if you're running this in the cloud probably will happen to you because you're not managing this infrastructure or even your running is in your data centers you're also not managing your data structure your system administrator might be um might be uh taking care of this but um the the how you can be tolerate to to failures it's important like what you will choose which you choose like consistency over availability or availability over consistency and um this is decisions that needs to be um clearly stated by a framework for data system in order to people to to use in this one stateful is hard so because the managing state especially managing state in in this world of um uh the clouds is also like difficult but it's important you know you cannot be just a very um very good general purpose uh stream processing framework or like data processing for an org without ability to um save intermediate steps or somehow um take care of the operation that will require some of the knowledge of the past um so that's um that's that's very important uh and um it needs to be working obviously uh in uh distributed in distributed fashion now next question and very important question that uh usually uh designers of the system should should ask you themselves where do i put my compute like if you've been here in a hadoop world where did you put your compute so there's multiple things so essentially you have a driver that needs you uh the you you compile your code you put it as jars and now you need to deliver this to to your computational cluster um same thing with uh the spark you compile this you're deploying this in the cluster and resource management needs to be happened so this is very uh good questions where you where your computing is running and sometimes like when you're trying to uh debug system and they're trying to understand like where where is where where are my things uh it's not very clear where's the state i remember i was talking about this kind of intermediate thing that needs to capture what's happening in your system where it goes where the stage is going it's important and important question where it's called those so um essentially this is the question that uh many data engineers and their architects were were asking when they evaluating any type of processing system or data system these days like there would be a myriad of the the the ways how the system will behave kafka streams is a java api for building applications not building clusters not building a um jobs or managing resources it is api that runs inside your application however it provides you cool capabilities for example like if we take this small application that follows the pattern that i was talking about i was talking about the pattern of constantly reading data from the topic because we're expecting data to arrive once we receive this data we do a processing and this processing is constant nothing runs in the brokers even though kafka streams is a framework that comes with kafka it's not it doesn't so it not runs inside the broker so it's not running inside the broker so um this this framework allows you to scale of your processing if you feel that one instance is not capable of consuming messages fast enough and with this built-in protocol of scalability you only think what you need to do is just like spin up another instances of application providing the same application id and after that kafka streams will handle um the coordination and will handle mid-date distribution it will handle which application instance will handle particular partition on topic and so far and so on so i i talked a little bit about um i talked a little bit about this um lambda architecture back in day and this is also like simplified something wrong my slides my dashboard d somehow uh went out but essentially this is a example of lambda architecture there's a processing cluster uh there's some job deployed to this cluster and there's some shared database that would be populated with results and after that there's some microservice there's some other system that will display this um in um in ui now how these uh kafka streams fella change this situation that's pretty much it so your kafka streams application might include other things it might include um some of the front-end elements because it's just a library it's a library that runs inside your application if you run this uh any uh popular microservices framework like micro node the spring boot um quarkus some other things you can use kafka streams there so you're using all benefits of existing java infrastructure or for that matter jvm it doesn't really matter if using um the java language or kotlin language or scholar um i don't i use java or kotlin but hey there is a native api for scala in kafka in fact so um in this pattern is uh there's a bunch of uh good examples um internally like we also at confluent we do have a the commercial product called confluent control center in confluent control center essentially is a kafka streams application it's a dashboard that allows to monitor your kafka cluster but it actually kafka streams application full-blown ui has all nice graphs that showing what's going on in the cluster and importantly with this like where is your code your code can be deployed anywhere like it depends uh on whatever tools and technologies your organization is using you can still follow this if you're deploying this in docker if you deploy this in um uh like the vms if you deploy this kubernetes if you're using ansible and teleport and things like that it really doesn't matter so as long as you know that application can be packaged something that deployable to this particular platform you will be golden so that's going to be fun so uh there's multiple multiple cool things um this uh um it's a great opportunity for me to be there and speak to you about this and i hope i will spark some of the interest about the kafka streams so what if you want to say like i want to know know more about this and i want to take my hands dirty so there's a fantastic resource called developer.com it's a my team and uh developed this uh the website this is the landing page of all your you know learning experience um it's my second favorite place on the internet like if you want to know why it is my second favorite place in internet click to this um link that you have this um like this one that this this link and you will get the idea why it's my second favorite place on internet um as a part of um is a part of this the developer.com from the taiyo there is a fantastic resource called kafka tutorials it actually gives you a 10 15 minutes lesson where you can just go follow along copy paste and you have runnable application no need to install anything except uh docker because some of the infrastructure like a kafka skim registry zookeeper will be shipped in the docker containers the rest of the stuff will require just the java installed on your computer so check this out it's fantastic it's my favorite uh little friend and i hope you will um find this useful um you can always find me in in internet and ask any questions i've always seen twitter i recently started doing a kind of like a weekly live streams in uh my personal channel it's youtube victor game of or confluent channel uh youtube confluent where i am talking about stream processing answering the questions and do all kind of cool stuff around all these uh things that we will be like that excites me including some of the devops capabilities including some of the things around how to deploy this how to solve not the first day responsibilities in terms of like what the app is doing but the second day like monitoring maybe or some of the um upgrades and other things so for that um i thanks for uh for sticking around my name is victor gamov and as always have a nice day i open for all possible questions um and that would be great so question here difference between kafka and the patches park so kafka and apache spark there's no difference it's a huge it's a different system apache spark it's the uh processing system system that allows you to uh do processing with data and apache spark works with kafka fairly well there is a connector that allows you to use kafka as the source for your data kafka is the general purpose event streaming platform that provides the ways how you can um the capture events how you can store those events and how you process those events um look into uh sbtb cloud channel in discord um maybe there are other questions or anywhere else so um yeah thanks victor for the session uh i'm sure uh there is also a special chat room uh that the attendees might go so you can connect with them yes as a matter of fact i will do this right now and after this talk and i will hang out for some time there even though i'm not promised to stay for very long time it's uh even though i virtually with you on the west coast i'm physically still on east coast and it's getting late here [Music] so you