scala.bythebay.io: James Ward, Distributed Commit Logs with Apache Kafka
Recording: scala.bythebay.io: James Ward, Distributed Commit Logs with Apache Kafka
you alright thanks everyone for coming I'm James Ward I'm a platform evangelist at Salesforce so at Salesforce we're big Kafka users but also provide Kafka as a service so that's how I've been getting into Kafka lately so this talk is intended to be a bit of an introduction to Kafka so how many people here have done cofco before okay so some stuff that Jay went through quickly in his keynote I'll go into a little bit more detail on and then we'll do a little demo and and also show how to use acha streams with Kafka so that's the plan feel free to ask me any questions as I go okay so some quick fundamentals about Kafka it's an event stream that's distributed and redundant so it's distributed across the cluster so you get redundancy with the data in an event stream form so uses messaging system semantics so a lot of the the vernacular the OC with Kafka should be familiar to you if you've done any sort of messaging stuff so we'll see that as we go through and one of the things that I really like about Kafka is that it the clustering part of Kafka wasn't an after thought it was really the one of the core design parts of Kafka so it wasn't just bolted on later it's really core to to what Kafka is and then another core part of Kafka is that it provides durability and ordering guarantees so some of the use cases for Kafka what I have seen it used for is things like modern ETL and then CDC so change data capture that's pretty common uses for Kafka data pipelines is is what I think probably the majority of people are using Kafka for is building these data pipelines like Jay talked about hooking together analytics with with all sorts of other integration tasks and those sorts of things so and then we also see Kafka being used quite a bit for big data in jest so this could be i OT devices this could be interaction that's all sorts of different things but we've got tons of data coming in Kafka can provide a buffer for us for for handling those events okay so that's kind of our background let's talk about what Kafka actually is so one of the the core piece of Kafka is something that's called a record or could also be called a message or an event and what a record is is it something that has a name a value in a timestamp it's immutable append only so and persisted so essentially this is a log and because this is built into a cluster it becomes a distributed log and so that's that's what a record is and so I'm going fast but we have 20 minutes so okay so we have producers and consumers in Kafka so in Kafka there is a broker that's a single node in the cluster and then the way that we interact with Kafka is we have producers that right records to a broker and consumers that read records from a broker so it's important to note that consumers aren't actually pushed records they actually read and they do that with up with polling and you can set the polling window it's a little bit different than a traditional messaging system in that way there are some reasons for that that we'll talk about a little bit later so Kafka has a leader and follower architecture for how it manages the distribution of data across the network so usually when you connect to a node you're going to publish to that node and then it is going to be follower nodes that will then get that message and percolate it down so leader follower distribution okay so now is where Kafka starts to deviate a bit from traditional messaging systems we have topics and those are really just a name for something that groups together one or more partitions and a partition is the way that we get our horizontal scale in Kafka so a partition is actually will can exist on a different node within the cluster so this is how we can now have our partitions that are scattered across the cluster and so then we can do our horizontal scale in traditional like sharding way of doing things so the partitions are replicated you set the replication factor and your configuration to determine how many copies of a partition there will be then ordering is guaranteed only for a partition so this is where things get a little bit tricky with ordering is that when you're dealing with writing to a topic really you're writing to to a partition within a topic and the ordering and Kafka is only guaranteed for that partition so the way that it works is you'll have a producer that's producing new records depending new records onto a partition and we'll talk about the the offset IDs in a minute for how that works but the ordering is only guaranteed for a single partition so across a whole topic there's not really an easy way to get ordering there are time stamps on messages so you can use the time stamps to do some sort of ordering but it's not going to be guaranteed like it is with the the offsets okay so then the offsets so the offsets are these sequential IDs that are written assigned to every record when it's written and so you can see in the the diagram that each partition has its own sequence of ideas within a topic so when you work with Kafka you're subscribing to a topic you're publishing to a topic but then there's the partitions underneath where the ordering is applied and the the partitioning across the cluster is also applied so what this means is that your consumers have to actually keep track of their offsets they have to keep track of where they are in reading through the data in a partition so when your consumer subscribes to to a topic it can subscribe to a partition within a topic and say alright I'm going to start reading at this particular offset ID and then continue reading from there so the benefits of this is that you can do things like replay so this is something that Jay talked about earlier where your consumer can actually go back in time and say all right let's start with this ID and if I need to re reprocess any data in the string am I can always replay back to that ID and then and then roll forward from there but usually you would do that per topic because that's where your offset ideas based if you want to do it across the partitions and you have to maintain the offset ities for the multiple the different partitions so another interesting thing that this supports is consumers that are operating at different speeds so this has now become a very common thing with with our data pipelines is that will have different consumers that are that are being able to process data at different rates and you want to be able to handle that without slowing things down and so that's each consumer can be at a different offset and reading at a different rate through the partitions okay so that's our offsets and then another nice feature that they've kind of wrapped around some of this to make for a bit nicer developer experience is this idea of consumer groups so with consumer groups we now assign a name to a group of consumer nodes and then what that can do is that can essentially load balance the the messages across those different nodes and make sure that a message even message is only going to be delivered to a single consumer node within it within a consumer group so we get a bit of load balancing without having to deliver the messages to all the notes through consumer groups okay okay so let's check out a demo quick little hello world demo here of how this all works so let's go see so I've got a local kafka server up and running and then I have a producer and a consumer and we'll go through the code for those but three different processes in this case the server the producer and the consumer and all running on on my machine okay so let's go first check out before we go into the code let's check out the actual demo here okay so you see that what's happening in this case is i have a a web application in this case and it is connected as a consumer of Jews from Kafka so we're seeing those messages come in they're being pushed over a WebSocket so the first number here is the offset and you'll see that i'm getting the sequential ids for the offsets so that's good I've got those numbers but one thing you may notice is that all these numbers are unique so what that actually means is that in this case I only actually have one partitioner I'm only subscribed to a single partition if there was multiple partitions then we'd be seeing duplicate values in those sequential IDs because the numbers are only unique across two given partition so in this case when i created my topic i only created a single partition for that topic and then the second value here this is the actual message body the message value and it's just a random number so so very boring data set here but let's pause this because this is now this is where we can now see the ability to do the replay so right now I've actually disconnected my consumer so it's no longer connected up to Kafka and what i did was i tracked my offset so what was the last offset that I saw and that was this 4637 number and so now if I go in and say alright let's resume this then you'll see if we scroll down there we now got all the did all the messages that were sent from when I paused it and then now we're back at real time getting the messages that are coming in okay so that that just showing that hey you know we can we can roll back in time to previous offset ID and start reading from there and then catch up so in this case there's there in Kafka there is durability settings so you can control the window of time for for how long it's going to keep messages actually on disk in the cluster for so the the default i think is a week but you could increase that up as long as you want you could also increase it based on the number of bytes that you want to keep around and that's how you would decide how long to actually keep these messages in the kafka cluster okay so that's our very simple hello world application any questions so far about six of Kafka or vix basics of Kafka or the the demo here questions so far yeah good yep there was two yeah good yeah so I should have a prize to give you for noticing it so the way that I actually owe the question was there was to repeated numbers in the stream so if we look down find that that number you'll see that there's that 4637 when I paused it we see that actually twice in the stream here so why is that so that's just an artifact of the way that I had it resubscribe is that I know my last offset that I read and so when I resubscribe I gave it that offset so that's where it started me what I should have done is like incremented that number by one or something said said okay I don't need that that first record because I already processed that one so so yeah good catch on that one yeah okay any other questions about basics of of Kafka or how this is all working yeah go ahead so duplicate messages yeah so the consumer when it subscribes it's just going to get the messages from the offset point that it subscribes out which could be the latest offset and then it's going to get just a single delivery of the messages from there so the consumer can actually control how what's going to happen when it's going to actually control have I actually processed this message so that's something that you need to keep track of on your own is which messages have been processed and so what you can do is you can store that offset idea of how far you've processed through the stream you could store that in Kafka you could store it in a database you could store it wherever you want but you do need to keep track of that offset ID so that you know how far through the stream you actually processed and then there's depending on the client that you're using you can control whether it you want to guarantee delivery Atma at most once and then there's some other the other settings that you can control as well yep yeah when I topic is split their conditions so each partition is fully replicated on to the other node so each partition will have a leader and then followers so the partitions are actually replicated from a leader to a followers so each node should have the the full partition yes a subscriber as subscriber can subscribe to to choose one or more partition so it subscribes to a topic and then can be more selective within the topic if it wants to so I guess there's like three different layers of absence shun you can subscribe to the whole topic and get everything on that topic you can subscribe to a topic and then a partition within or many partitions within that topic and then you can subscribe to a topic of partition and starting at a specific offset within within the partition yeah so different different granularity so in this example the way that I subscribed initially was to subscribe to the whole topic so everything across the whole topic and then when I when I reconnected I had to actually subscribe to a specific partition will see us in the code in a minute I had to subscribe to a specific partition because i have an offset ID that's specific to a single partition so I had to to subscribe to a single partition in that case if I wanted to that to work across the whole topic then I'd have to keep track of for each partition what's my offset and in this case just simple example I just was keeping track of a single offset so this example would only actually work with a single partition the way that I programmed it [Music] yep yeah so see ya so it's just taking every time it gets a message so the question was why is it all scrolling down so every time it gets a message it's just a pen prepending that message into a list and so that's why all the data is pushing down okay so that's that's a quick demo let's move on to to the code and where did my slides go okay I guess I don't need slides okay so the the code that i'm using here is using something called reactive kafka which is a library for acha streams acha streams is a really basic source sync system a p.i and that and in the case of acha it's built in the case of acha streams it's built on top of acha but there are other implementations for for what's called reactive streams out there as well okay so this is using reactive kafka so let's go through some of the code for this this little demo so I have a Kafka class here I'm doing some ssl config which is actually not being used in this case that's only being used when I run on Heroku because Heroku requires ssl to connect to Kafka but here's where we get into the interesting stuff first I have a producer settings and the producer settings uses a string serializer so we need to give the producer a serializer for the key and for the value and in this case I'm just using strings so pretty easy there then I give it my bootstrap servers Kafka uses zookeeper underneath the cover so that's actually what we're bootstrapping based on and then similarly we have consumer settings and in this case we need a string we need a deserializer so i'm using a string deserializer for my consumer so i give it that for the key in for the value give it my bootstrap servers and then I give it my group ID which in this case I'm not really using but that's how we group together multiple consumer nodes so that only one no within the consumer group will get each message so now here's where we use the daka streams part is we have both a source and a sink so the sink is a sink of producer record producer record comes from the kafka API and we're going to create just a plain sink give it those settings and that's how we now can can have a sink that will dump messages into then we have a source and a source takes a topic and then maybe an offset and that's a source of consumer record so we look at that offset if it was spit was not specified then we're going to just subscribe to everything on the topic if it was specified then we're going to subscribe to a specific partition using the offset that was specified so that's what happens when I reconnect and then i create my consumer plain source okay so now my producer here's what I'm doing is I'm creating a source tick that's going to tick every 500 milliseconds it's going to produce a random number and then we're going to take that tick source we're going to transform each of those random numbers into a producer record so that's from the kafka api so we need a producer record to be able to write into Kafka and then we just say alright send that data to the kafka sink and so now that's what's actually producing those records to Kafka then on the consumer side i have my web application this is using play play uses acha streams so now it's really easy to wire together the WebSocket wick with the reactive Kafka so what we do is we check to see do we have an offset that was specified we create a Kafka source from the random numbers topic we give it our may be offset then we're going to get a consumer record every time one is it one of those is received so then we turn it into Jason and then we wire together the WebSocket with that with that that Kafka source and you'll see that there's a sink dot ignore and that's because incoming WebSocket messages I just ignore so that's that's how we do that ok so that's our Scala reactive Kafka code that's that's hooking all that together so you can see pretty simple I really like the reactive Kafka library makes working with Kafka super easy so ok so that the code it's all up on my github so github.com / James Ward and you can find the hello play Kafka a repo in there if you want to go through all the code and run it on your own or run it on Heroku or whatever okay I'm just about out of time but are they any last questions about Kafka I know it was quite the whirlwind but i hope you stuck with me yep repartitioning that's the topic I don't know yep does somebody know how Kafka handles repartitioning command line tools so like changing the partition the number of partitions yeah right so it has to be done as far as I know it has to be done manually maybe there are some tools that exist on Heroku Kafka i use the Heroku command line to change my partitions create topics and change partitions on you can change it at runtime automatically using machine learning you can scale it add partitions but not remove partitions ok I'll be around all day so if anybody has any other questions come find me but I hope that was useful thanks for coming [Applause] you