SF Scala: Jay Kreps, Kafka: I Love Logs
Recording: SF Scala: Jay Kreps, Kafka: I Love Logs
Okay. So, um this is going to be uh kind of a weird talk. I'm going to talk uh about Kafka um and I'm going to talk a lot about logs and data integration and a bunch of other things. Um I should probably introduce myself. Uh I'm I'm an engineer at LinkedIn. Uh I'm the you know, technical lead or architect or whatever you want to call it that's in charge of online data systems. So, that's like key value storage and Kafka and the stream processing stuff that sits on top of it which I'll talk about in a second. Uh Okay
So, so the plan is all you know, to start with the specific. I'll give kind of a you know, very brief introduction to what Kafka is. Um I'll talk a little bit uh more generally about kind of like why we built this and and what it's for and and logs which is kind of the abstraction that Kafka provides. Um and then I'm going to talk about something uh much more concrete which is data integration and and getting all your data together in the in the right places in different systems. And and then I'll move on to talk a little bit about stream processing. So, let me let me start with Kafka. Um so, it's a you know, it's a messaging system. Um I I don't know how familiar people are with messaging
You know, you you have some producers. They they publish data. They send it to some remote cluster which stores the messages. You have consumers which subscribe to that and get the messages out. Um it's pretty straightforward. Most people have you know, at least had some familiarity with like JMS systems or something like that. Um and so, Kafka is actually you know, kind of very much in that same vein but but uh but it's built a little differently. And I'll I'll talk a little bit about that
Um and for a long time it was um it was kind of a an odd duck of a piece of software. Um but actually Amazon made a service which is very similar which is Kinesis. And and so, most of what I'm going to say about Kafka you could probably actually substitute in Kinesis and and it would probably also be true. Um and so, I was actually very happy about this because the first uh open source project I worked on was a clone of uh Amazon Dynamo. And it was kind of just a future for future clone of their paper. Um, and and so this is good because then we made something and they kind of they kind of cloned it. Um, all right. So so let me talk about how we came uh to kind of make this thing
Um, uh I I always find that if you want to understand some system, one good place to start is just what were the kind of problems and uh you know, um uh what what did the people have in mind? What were the use cases that they had in mind when they started working on it? And and so what we had in mind uh was, you know, essentially we were we were building out these uh you know, scalable key-value systems. We we were building out our Hadoop infrastructure. Um, and and probably the biggest problem we had with with all this infrastructure in addition to just keeping it working uh was kind of getting data flowing to all of it, right? Um, and um we had actually different ways of integrating data for each system. So we had some kind of way that we are synced log files around and we had some other way that we would yank rows out of our databases to feed it to our search cluster and we would have, you know, different ways for each of these things. Um, and you know, over time I I I got to see this as being more and more of a problem and we kind of decided to take a more, you know, infrastructure-centric approach to solving it, which is which is what Kafka ended up as. So so kind of in looking at all these different use cases, um they're actually really different. Like we we didn't end up with so many different ways of copying data around just because we were, you know, uh stupid. Um, it turns out that, you know, each each system actually had some special needs, right? The data warehouse system was some offline system, but it but it was actually copying pretty high volumes of data around, right? We we we had kind of historically we had some some like, you know, JMS queues that were taking data for for newsfeed
Um, and those were actually pretty low throughput, right? And then we then we had things that were processing data out of databases for search indexing and to put it into our social graph system and whatever. Um and and those actually have very uh very strong semantics, right? If you're if you're trying to keep a search index that matches your database, you better have all your data get through. Otherwise, they are permanently kind of inconsistent. Um and so you end up kind of having these different needs where, you know, one is kind of high throughput, others are low latency, others are, you know, um have strong guarantees around data delivery. Um and so so we we kind of tried to abstract out of this something that that a messaging system would have to do if you wanted to replace all these different things and kind of have a a single pipeline that carried all your data. Um and we came up with a couple things, right? So first of all, you know, it had to scale relatively easily and cheaply. Um I think this is really important like uh the systems that I've seen really succeed in organizations are ones where it's easy to get data into them. Uh any system which is actually a pain to scale and it may be a pain because it's, you know, fundamentally unscalable or it may just be very finicky
Um what you see is you see always some some layer of bureaucracy organizes to keep people from using it, right? So you you know, uh and the reasoning for that is just because it breaks when too many people use it. Uh and so so the systems that I think are successful are ones where, you know, it's very easy to get your data in uh and get some value out. Um and so one of our goals was to make something that was very easy to scale uh and and could handle a fair amount of data per machine and was easy to add machines to. Um the next thing we had to do is we had to give guarantees on the data delivery, right? So um to to handle any of these uh use cases or kind of low latency and handling transactional data, we couldn't be losing records. And and finally, we wanted to build something that was uh you know, distributed by default, right? The the the systems that I think uh work well and are good to operate at scale are ones where people thought through the fact that it was going to be a distributed system up front, that wasn't some later add-on that they tried to to add in. Um and and so so what we came up with was basically Kafka. Um I'm going to talk a little bit about kind of uh the motivation for it. This isn't going to be an internals talk
So I'm not going to say like oh this is how it works and this is how a message is processed and so on. Um if you're interested in that we have documentation and so on. What I want to do is kind of more motivate why you know why I think the applications of this are interesting. So anyhow, we're we're using it at LinkedIn uh you know for for a fairly large amount of data, right? It's kind of the central data pipeline for everything. It's it's a low latency system. Um it does replication between all our data centers so we can get an aggregate view of what's happening. Um and and we you know we have thousands of uh clients that that either are sending data into it or consuming data out of it. Um and it processes millions of messages per second both both read and written
Um and this is kind of the central data pipeline that we use for everything that goes to Hadoop as well as real-time processing. Okay. Uh so it's an open source project. Um you know it's it's with Apache. Uh it has a a pretty good group group of people working on it. A bunch of clients in different languages. And and probably the best thing is it has a bunch of uh I don't know what we call it. We call it the ecosystem of like integration things that plug in so you can get data into S3 or you can get data into different systems or you can stream data out to like web sockets or all kinds of crazy things like people have have really made a lot of these
Um okay. So that's kind of just like a very brief overview. It's a messaging system. We're using it. Hopefully that gives you some information. Now I'm going to step back and talk about um some of the context of of how it's different architecturally and and why I think this is an important uh you know thing for companies to build around. Um and I'm going to start by actually just talking uh more fundamentally about logs. Uh so I'm starting kind of at the very beginning
Um so so Kafka um its abstraction is actually logs. What it provides for you is a structured log. And, um, when I say that, uh, it's a little confusing because most people's idea of what a log is is a little different from from what I mean. Um, most people's idea of a log is like an Apache log, right? So, it's like a text file, you know, it's kind of it clearly has some columns, like some timestamp and some URL and some request code. So, it's kind of semi-structured. Um, and I guess you append data, uh, to the end of this log file. And eventually, I guess you are sinking it somewhere so you can process it in Hadoop or whatever you do with it, right? And this is actually not what I mean. Um, so I actually have a more abstract view of logs
So, so I think logs are great, but but this is kind of, um, uh, this is a little too nitty-gritty view of a log. So, so kind of zoom out a little bit. You know, a log is basically a structured append-only array of records. All right? Um, so you can imagine all that data that was on each line, now we're kind of putting it inside the box. I don't know what it is. It's like some bytes. Uh, maybe it's some JSON, maybe it's some protocol buffer, like doesn't matter, right? Um, each record has some number and records are, you know, assigned a sequence, right? So, there's some order to these things. And, um, you know, it's still a log
You append to the end and you can read it from from wherever you want. Uh, so so that's this is what I mean by a log and this is much more akin to what you would see, you know, in a transaction log in a database or in a distributed system. And this is this is essentially what Kafka provides. In fact, Kafka's data model, uh, you know, the the abstraction that it provides as a system is is really just a sharded log. So, you you have a topic that you publish data to. But all a topic is is just a bunch of, uh, logs. And and data is split up over these logs. And, um, you know, records are are kind of appended to the end and and readers can read from any position in the log
Um, and, uh, so how do how does that work as a a pub-sub system? Um well, it works, you know, kind of as you would expect, right? Um you know, here's your log, new writes come in at the end, readers are actually have some position of what they've read up to. Um and here's where you can kind of start to see some of the advantages of a log, right? Um because because we have these numbers, which are identifying each entry, uh we call it an offset, but a lot of times in logs, people will call it like a log sequence number or something like that. Um that that almost acts as a sort of notion of time. So, if you've read up to record seven, you're kind of at time seven, right? And and because you have a log, you actually kind of have a full record of what happened, and you can kind of capture the state of your system after processing each record, and it would have some some time that it was at, right? And so, this is this is kind of a very old idea, um not not invented by me in any way, uh but it gives you a way to reason if you have different systems, and they're at different times, how caught up are they? Where are they at in this stream? Okay. Um so, this is actually a very old idea, um and, you know, it's kind of at the core of what I would call classical distributed systems theory, um is, you know, logs and state machines, and um you probably a lot of people have actually heard, you know, somewhere about like Paxos. Have people Have people heard about Paxos a little bit? Yeah, a few people, right? Okay, so so, I'm not going to go into a lot of details about these algorithms, but there's basically a lot of algorithms for making machines agree about the order of records in a log. And that's an oversimplification, but that's mostly what people use these kind of consensus algorithms for is coming up with a structured log of changes. And why would you Why would you want to do that? Well, it turns out that that's that's actually a core problem for any distributed system is is getting a bunch of machines to agree on the sequence of updates, just just like that log
Um and so, if you if you kind of peek under the covers of a lot of distributed systems, um you see uh you see a log in there. So so actually, you know, in in uh HDFS I think has some log. I mean, I these guys here would know better than me. Uh I think HDFS, right? It has some log of changes that it's recording in its in its name node. I think recently maybe they've even distributed that so that they have some fault tolerance for it. Um and uh you know, these these newer fancy things from Google like Spanner is kind of very built very much built around some log. Uh and of course, everybody is used to log replication in traditional databases like MySQL where you're you know, replicating master to slave or whatever. So so I think most people are kind of vaguely familiar with the idea
There's usually two things you're trying to accomplish in these systems. One is you want to get data around between things. And the other is you want the two you know, your your replicas to agree, right? Okay, so this is uh this is getting very theoretical, but I promise you I'll I'll bring it back to kind of our our central theme of building data pipelines and getting data around. Um, I'm going to give one example just of how how a log works in practice. So this is a CEO hash table. Um, this is a very important uh application. So we need to we need to keep track of CEOs of companies. Uh and we want to do it on two machines
If we were just doing it on one machine, it would be fine because if one of them crashed, we would just lose all our data. We wouldn't have any problems. But if but if we have two machines, we have to keep them in sync. So in this column, I have I have a sequence of uh updates. I don't know if you can read it because um, it's a little bit late in here to read these slides. But this is you know, a company name and some CEO name. So we've got like Microsoft and Bill Gates. That's a put
And then we've got Apple and Steve Jobs. And we want to carry out these in on two different machines. Uh and we want to end up with some final state which is you know, sort of like the current state where the right CEO is associated with the right company. Uh and this is kind of you know, at the heart of what a lot of systems are trying to do, right? They're trying to undergo a bunch of state changes. And they're trying to end up in the in in some consistent state across across multiple machines. Okay? Um and and kind of the classical distributed systems approach to solving this problem is like, well, hey, let's put all our changes into a log and then our two replicas will agree about the order of these changes and they can, uh, you know, take action on each of these changes independently, but still come up with the same answer. Why? Cuz they made the changes in the same order and they got the same data. Okay? Um and there's a bunch of, uh, kind of names
If you were to go get some book, there's like one style which is called state machine replication where you have a log in front of the system and there's another style which is primary backup where you have masters and slave that feed off a log. I'm not going to go into it. Um if you're interested, you can kind of read about it. It's kind of interesting to know how these work, but we're going to getting into these very esoteric details, right? So, what's the what's the point of all this? What, uh, you know, what's what's the use of these? Um unless you're building a a distributed system from scratch, right? Unless you're trying to build, uh, you know, a database, why would you care? Uh and the reason I think you care is actually data integration which is a very practical problem. So, so I think, you know, there's a problem that, um, I think every company has, uh, which is you have all these systems and you have a bunch of data and the systems are only useful if you can get the data into the into the cool, you know, processing systems and make use of it, right? And, you know, my observation is that the types of data that companies have that they care about has has actually increased. The scope of data has increased a lot, right? So, so if we were talking about this, you know, 10 or 15 years ago, we would really be talking about transactional database data. That's what people wanted to get. They want to get their transactional data into their data warehouse for analysis, right? But I think increasingly people are interested in two other types of data
They're interested in event data, right? Which is maybe tracking like, you know, if you're a website, who's clicking on what and what are the searches and what are the impressions that you showed. And they're interested in kind of sensor data, which is like maybe if your website again, you know, what's the CPU usage on this machine and you know, what's the queue length on such and such a queue in your application. And And broadly people would just call this all logs probably if you're in the web space. With a different meaning than what I'm talking about. And so this data has really exploded and is much larger, right? I think people understand this. But at the same time, the the set of systems that people care about is also exploded, right? There's more more specialized distributed systems. You know, there's there's talks about two of them today, so I guess. There's so many of them we have to go out and and talk about them with people
So at LinkedIn I can kind of tell you what we're running. So we're running a couple of key value stores. We have a graph database that shows you kind of the different you know, the the path length between people and the degrees and you know, the shortest path and all the different routes and how you know people and that kind of stuff. Kind of CS 101 graph algorithms. We have like an OLAP store which does all the reporting for advertisers or even internal you know, reporting and metrics. And so it does kind of big distributed queries. We do search like in many ways that's the key business you know, that's how LinkedIn makes its money is off search. So so search is very important to us
And then we have like you know, metrics and graphs for monitoring. And then we have a whole offline ecosystem. So this is already like a lot of things, right? And they're only valuable in so far as you can get the data you you want into them. And when I started at LinkedIn we actually you know, we approached this in a very direct way, which was you know, we would have a system like maybe we had Oracle and we wanted to get Oracle data into Hadoop. And so we would write you know, some programs to scrape Oracle data out and shove it into Hadoop or we would write something that would copy this kind of data to that kind of thing. And we wrote more and more of these things and it got more and more complicated. And that was kind of the the the for for working on Kafka. Uh and so our realization, I guess, is that um you know, first of all, these pipelines ended up being much harder than we thought, right? To keep running, to make correct, to ensure that we have the same stuff in the same places, and to be able to reason about what state each of these derived systems was at
And so our our realization was this is actually very much like the same the problem you solve within a distributed system. Right? It's the very same problem. You need to reason about, "Hey, what's the state of my Hadoop cluster? What time is it caught up to? Um what what is the state of my search index? What time is it indexed up to? Hopefully much more recent than my Hadoop cluster. Um so and and and so um the the kind of abstraction that we wanted to build and put in place was really, you know, a distributed publish-subscribe log that we could feed everything off of. Uh and and the use case for this is really this data integration problem. Really feed feed all the different systems. Um it really bring kind of a uniform structure to data even when the data comes from different places, so that you don't have, you know, JSONs and CSVs and whatever else, right? Really have some kind of uniform structured data that all flows in a uniform way. And and this actually ends up having a lot of value cuz what happens is you have to do some work to bootstrap the first people to get their data in, but once they do that, many other things integrate to get that data
And once they integrate, there's more value for people to put more stuff in, right? And once they put more stuff in, there's more value for people to integrate, and pretty soon you have this pretty rich data ecosystem. It's much easier to kind of connect all your stuff. Um so I don't know, it kind of reminds me of um Metcalfe's law with cell phones. They say like the first cell phone is very hard to sell cuz there's no one to call. But uh the value of cell phones goes up as as more and more people have them. Uh roughly proportional to the square of the number of total things. Uh which is actually not surprising, right? Cuz the number of lines in this picture is roughly the square of the number of things we have to pipe together. Uh and to give a kind of a concrete example, you know, if you're working on a website or, you know, mobile application or whatever, this is kind of how this plays out for data integration
So, one thing LinkedIn does that people knows we show jobs on the website, right? So, people look at their job. And when you're very small, it's actually very simple, you just show the job. But then you get bigger and things get a little more complicated. There's a bunch of other stuff that has to happen every time you show a job, right? So, the first thing you have to do is people start messing with you and you have to start dealing with scrapers and spammers and people who are trying to DDoS you and whatever. So, you actually have to record and respond to all those type of activity, right? Um the next thing you need to do is like, "Hey, this is a business. People are paying to post jobs. You got to tell them whether they're getting any click-through. Like, "Hey, you know, a lot of people looked at your job
Nobody applied." Uh it's probably not a very good job. Um and uh we're also doing job matching, right? So, we want to be able to recommend jobs. We need to know uh what's the click-through rate on this job, you know, or what's the application rate so we can make better recommendations. Um and we need to be able to monitor these kinds of things in real time um because this is a business and if it's performing poorly. So, so if all of these things had to be kind of baked into the the jobs page, um it becomes more and more complicated, right? But but the way we actually build it is the jobs page or, you know, the job app, it just publishes something that says, "Hey, you know, a job was viewed. These are the attributes of viewing the job. This is the person that did it." Um and all the other systems that have some specialized domain, right? Whether it's kind of Hadoop, which is keeping the data forever, or security systems which are, you know, being like, "Hey, it's not really possible to to view 100,000 jobs in 3 seconds" or whatever, right? Each of these systems just kind of feeds off that stream and does what it does uh without having to be kind of baked into every application it it integrates with. And without actually having to know what the original source of the job view was
Right? Um because nowadays there's actually, you know, if you're a website, there's like, you know, 14 mobile devices and web and mobile web and everything. And so, so having a unif- unified data flow across all those is really important. Okay. So, so now I'm going to extend this and talk, um you know, not just about the the piping, the data integration. I'm going to talk about stream processing. Um and and stream processing I actually think of this as being, um very important and very closely related to data integration. So, so stream processing, I don't know how many people are like vaguely familiar with the term. So, I don't like Okay, so like probably most people
So, stream processing in this context, I'm going to use it to mean something kind of like MapReduce, um but in in real time, right? So, something where you're continually processing changes and you're continually outputting new results. Um and I actually think this is a really important thing uh if you're building out some kind of uh set of pipelines like I talked about, because it allows you to take all this like rich ecosystem of data and transform it into new data feeds which you can pipe into other stuff, right? Um and this is almost always something you need to do when you take input data, right? So, when you capture data, it's rarely in the form that you actually want for different applications. So, anybody who's built like a, you know, like a search application knows the stuff you index is not quite the same stuff that was in your database. You do a lot of intelligence beforehand. If you've ever built something that kind of goes out and crawls, you know, news or web pages or whatever, there's a there's a long pipeline of transformation, right? And so, I think this is a really nice kind of abstraction to support and build this kind of stuff. Um and uh the way I the way I kind of discuss it with people is like, um you know, when we got started, uh when I started at LinkedIn, we actually had no real-time data. We didn't have any Kafka. We just kind of had batch data collection
And at that time, somebody actually came and tried to sell us uh a stream processing system and we didn't really know what to do with it because we just had these batch data files and they said well there's no problem you can take the batch data files and you can feed it into our stream processing system after you've collected them. And we were like well I mean that's fine but we don't really want to pay for that. And it was also all on one machine so it was it was it was not really able to even take all of our files. And and so then later and so at the time we thought well this is silly but but when we actually got real time data suddenly it made a lot of sense able to process some of that stuff in real time. Now not everything needs to happen in real time but a lot of things that actually makes a lot of sense. And so the analogy I use for people is the census. You know if if you ever I assume people are familiar with the census right every 10 years we hire some like you know million people with binders and they go around and they write down they go door to door and they write down everybody who lives somewhere so they kind of ring your doorbell hi do you are you a person they write it all down at the end we count it all and we say okay this is how many people live here in the US and then we do it again in 10 years. And if you ever talk to somebody who's a software engineer they think well this is kind of crazy
This is a crazy way to count people. Why wouldn't you just journal births and deaths? If you did that then you would know not only how many people were there you know in the country right now but you would know how many people there were at every point in time prior. And that that I guess is kind of the difference between a stream processing approach and a and a batch processing approach. And there's there's different pros and cons in different scenarios but there's actually a lot of value to that kind of thing. And you can totally see why they don't do that for the census because it started in 1790 and they were like going on horses and so you know the the horse was the the the equivalent of the TCP IP network so the only thing you could do was collect all the binders and then count them all. And and I think a lot of organizations are actually in that same stage right they're shipping data around in batch files and and so they're going to then process it in batch. Uh but but the first thing you want to do once you collect it in real time is process it in real time. Um so the next thing to understand is actually, you know, stream processing is actually a generalization in most ways of batch processing
So so the the two things that uh people are usually familiar with is they're usually familiar with, you know, web services where you get a request and you give a response, right? And they're familiar with batch processing where you get, you know, a bunch of rows and you process them all and you give out a bunch of rows. Um so if you if you kind of mix up record and row, you know, stream processing is just where uh you know, you get some number of rows and at some point then you're choosing you give out a bunch, you know, some other number of your choice and you get to control the frequency of output. Uh it's not always at the end. Okay? Um so where this kind of comes in handy is for use cases like monitoring, security, any of these content pipelines. That's very general, but anything that's going into a search, anything that you're kind of like crawling and processing. There's a lot of people doing really interesting stuff with data these days uh where they have these really complicated transformation pipelines. Um a lot of like recommendation things, especially anything off of activity data where you're looking at click-through rates or popularity. Um uh any anything related to like news, you know, news feeds of what's popular, trending, right? Okay, so how does this tie back to kind of Kafka in this log-centric view? Well, my my view of stream processing is um you know, I guess a lot of people when they think of stream processing they think of like there's a stream of data and you process it and you throw away all the data
Um my view is actually a little different, right? My view is you have, you know, you have input logs and then you have jobs that transform those and they create new logs, right? And uh so this fits really nicely into that kind of log-centric view I was I was proposing. Um the uh you know, the the um what one of these pipelines looks like in practice is not that different from a bunch of MapReduce jobs. It's just that data flows from job to job without the job ever ceasing, right? The job you know, runs forever and it takes the continuous stream of page views or clicks or whatever and and sends it on to the jobs afterwards. Um and so we we put some effort into supporting this at LinkedIn. Um we have a system called Samza. There's also a system called Storm um and and there's uh Spark has some streaming support and I think all of them have some integration with uh Kafka. I think Samza and Storm is very good. I haven't looked at the Spark ones in as much detail
Um so they'll all kind of feed off these pipelines and do real-time processing for you. So I'll talk a little bit more about Samza. Like the architecture is, you know, it runs on top of Kafka. It runs on top of YARN, which is like the lower-level process management framework. Um it's technically pluggable, so it could be made to run on top of Mesos. Um But uh and then there's some layer that you program to and there's a bunch of jobs and they all kind of run and and just churn through these real-time data streams and produce new ones. And so what the the the real value of this is, you know, in this kind of like log-centric integrated architecture or whatever, um if you were to kind of zoom out, this is what LinkedIn kind of looks like. We have a bunch of storage systems that do graph storage or OLAP and are just they're just kind of derived stores
And then we have kind of primary data stores and we have search, right? And these are all the things users interact with. And all the data streams that either come out of them or or go into them kind of go through this central pipeline. You can take any of this and you can transform it and create some new feed, which can go back into the main systems for serving. You can get any of that stuff kind of projected out into Hadoop for offline processing or analytics, right? And you can do monitoring and graphs on top of those data streams as well. Um and so uh so I think that's that's kind of the role of the system and how it plugs everything together. Um And so if you're interested in this stuff, uh the, you know, Kafka and and Samza are open source projects. You can you can check them out. Um I wrote a much longer blog which actually goes into a great great deal of detail about kind of the architectural style
And then if you're interested just in Kafka and how it works, there's a bunch of documentation um on the Kafka site about the internals and the APIs and how it processes messages and all that kind of stuff which I I really didn't cover in this talk. And that's it. Any questions? So what do you use as your authoritative data store? Cuz it sounds like you use your logs to drive downstream information. Yeah. And so that feeds back into the key value store that I mean it's not really clear what the authority is. right. That's right. So the question is what's the authoritative data data store? We would call it the source of truth
Yeah. So it depends on the data source, right? So for databases it's always the database. Database is the source of truth, right? Log is just an integration subscription mechanism. Um for event data, uh Kafka is actually the source of truth, right? So if you're an application and you're you're publishing events of what happened, you just write it to Kafka. There's no ground There's no other system that's the source of truth. So it depends. So everything else that I described, all of those serving tiers um and Hadoop, those are just replicas, right? So in the database case, they can all be recreated off the database. If you lose the source of truth, then you're really screwed
Does that answer your question? Sort of, yeah. Okay. So what what would answer it more? Uh so so it sounds to me like you answered it with it depends. Yeah. And uh so I think that makes it very difficult to reason about, you know, where the authoritative data is. I guess it depends on a case-by-case basis. No, I I don't think so, right? So there's two types of data, right? There So the question is does that make it hard to reason about, right? So so the authority is always the system of record and there's always a system of record. But for log data, what would that be? It's like an event
It has no It's It's transitory, right? It doesn't It doesn't have any like system of record other than the log itself, right? Um and so Kafka acts as a system of record for us. Um for these other systems, you know, it's just an integration mechanism. Okay. Does that mean then that you would keep the logs forever? Yeah. Yeah, so so um So, one of the things Kafka is good at is retaining data. Um so, a lot of messaging systems are meant for just kind of instantaneous messaging. We're actually pretty good at keeping terabytes and terabytes of data if you want. We don't We don't keep the logs forever
Uh There's two different cases here which I should talk about um and I'm glossing over a lot of details of our internals, but um So, for logs, we keep 7 days uh for real-time subscription. Then we keep uh 2 years in Hadoop. So, if you want really, really long, you know, batch processing look back, you go to Hadoop. Um it's configurable per topic in Kafka. So, people who are like, "Oh, I need 3 weeks." I mean, fine, you can have 3 weeks. Um but the default The default is one. Um and then you for for uh for database data, we actually support um We have like It's more complicated for database data, but but we do support some look back feature so that you can kind of go from the beginning of time in some compacted view and scan forward. Um that's a little bit inside baseball, but Question
Do you process data on the disk or you using replication for data delivery? Yeah. So, so in Kafka, the question is, do you persist data on disk or you just using replication? Um so, Kafka is persistent. Um There There's two, you know, usually when people ask about durability, the question is, "In what situation will I lose my data?" Uh So, so we do two things, right? All data is immediately written to the file system, which is not the same as the disk. Um And it is then immediately replicated to the replicas, right? Now, you have an option, if you are paranoid, of setting an Fsync setting for that topic, which can be either very aggressive, like every message I want immediately flushed to disk, or very lax, like do it at the operating system's behest. If you allow the operating system to do it, you know, in its own good time, then you get great great disk throughput because it can optimize, you know, its write pattern. However, you know, replication only protects you from certain types of failures, right? Replication protects you from server failures, any kind of random failure. It doesn't protect you from like power outage, where all your servers suddenly die. So so if you were to run in that mode, then yeah, if you if you're if you lost power or something, you would lose your last few messages that have not been flushed
Other questions? Another question? Can you give me a rough idea of the type of write throughput you see Yeah, um That's right. That's right. So so the question is what's the right throughput per node? So actually, yeah, I I just wrote and was about to publish, which I'll probably get out next week, a more detailed benchmark on the newer code. But yeah, so for messaging systems, I guess there's two cases. Very very small messages tend to create some challenges. So for us, maybe the threshold is around 100 bytes. So for very very small messages, you're thinking more like QPS. And so for those, you can get an enormous QPS, but the the megabytes per second would not be competitive with like SCP
Once you get up to about 100 bytes per message, it's kind of like whatever your you know, gigabit per second ethernet card will do. I haven't tested on like a really fancy network, but in general, we're able to kind of, you know, work as fast as the disk subsystem and the network card up to reason, you know, at least for reasonable gear, right? So so yeah, it's a very fast compared to uh traditional messaging systems, which for whatever reason seem to send like one row at a time. They don't have any kind of batching or grouping, and so they tend to be much slower. So, one of the reasons people really like Kafka for log data is cuz it's really high throughput. So, so like one experiment we did before we had Kafka was we had these like ActiveMQ's, and we said, "Well, what if we just like ran all our logs through ActiveMQ?" And ActiveMQ may have gotten better, so I don't know if this story is still accurate, but at least at that time, you know, we were a very small web company. Uh we had a total of 300 servers. We we did the back of the envelope on how many servers we would need for that that much log data. It was 300 more
So, it was like obviously not going to be cost-effective. Um, so yeah, we do a bunch of tricks. It's really the same thing a file system does, where you just group together a bunch of small writes so that you get good throughput. Um, and that will basically uh give you very good throughput. So, so yeah, you you know, I would think about it in terms of megabytes per second. You can totally do 100 megabytes per second from one node. Um, you can do that with replication, and it'll still keep up. Um, um, Question in the back
Is Kafka suitable for like real-time performance monitoring, or And if so, is anybody using it? Yeah, so so we use it So, the question is is it suitable for real-time performance monitoring? So, so I assume that what you mean is kind of like um I would call it like sensor data almost, but what we have uh So, so the answer is yes. We run um you know, we have whatever 10 million metrics that we track in real time, and they all run through Kafka. Um, we you know, every service that we have instruments itself with lots and lots of basically counters, um and they all publish, you know, continuous sensor data about what's happening on every machine. Um, I I think it's 30-second intervals or something, and that whole data stream is available and subscribable. It goes into the alerting and graph system, and it goes into other stuff. It goes into Hadoop for offline analysis. So, yeah, you can totally do that. Question in the back
So, you mentioned right off the bat that that Amazon essentially copied Kafka or Kinesis. So, at this point in time, what are the significant differences between the two? Yeah. For starting today, Yeah. which solution would would be the preferred choice for a company? Yeah, I mean, so I don't I don't actually know like a really good feature matrix. I think they they might be at slightly higher latency. I think in practice it doesn't matter. The big difference is one they host for you and the other is you run. Um The good news about you run is like, you know, hey, it's an open source project
You can take it with you wherever you go. Uh you can run it on your own gear. If you're not in AWS, you can move to a different cloud. The good thing about AWS is they run it for you. If you've ever run this kind of stuff, that's a pretty big pro. Um so so yeah, I think in practice, that's like the only deciding factor. If you're if you're interested in having um data data infrastructure that you run, um I you know, I would I would probably go with Kinesis and pay by the hour. If you're concerned about building everything around some, you know, proprietary service that charges you by the byte, then maybe you want to run your own thing
That's probably the biggest deciding thing that I know. So, Kinesis is only you can get one day of data. Yeah. Yeah. Yeah. So you So you basically get more you run is like, you know, hey, it's an open source project. You can take it with you wherever you go. Uh you can run it on your own gear
If you're not in AWS, you can move to a different cloud. The good thing about AWS is they run it for you. If you've ever run this kind of stuff, that's a pretty big pro. Um so so yeah, I think in practice, that's like the only deciding factor. If you're if you're interested in having um data data infrastructure that you run, um I you know, I would I would probably go with Kinesis and pay by the hour. If you're concerned about building everything around some, you know, proprietary service that charges you by the byte, then maybe you want to run your own thing. That's probably the biggest deciding thing that I know. So, Kinesis is only you can get one day Yeah
Yeah. Yeah, so you so you basically get more control and there's more features. I I actually probably don't I'm probably not the most informed on the the feature comparison. Any other questions? So Jay, I remember you mentioned that there are two modes for log storage. There is download and there is persistent. Can you talk about the other mode where log storage is the system of record where the persistent Yeah. Yeah. Yeah, so um so the question is uh there's another mode where you keep stuff forever
Um okay, so this is like a little bit detailed and sometimes confuses people. So we do we do have such a thing as uh compaction in logs. So usually the way you handle a log is you throw it away after like 7 days. After 7 days you say, well, that should be enough or after a year, whatever your time period is you just chop off the end and throw it away. Um but we do have a mode that will allow you to go back and compact the log. Um and the the rationale behind that is for many things which are serving key data, um where there's some primary key, it's actually very useful to be able to retain not the last 7 days but a complete a complete copy of the last value for everything. And and that allows us to act as kind of a you know, an external log for uh you know, a a system that's doing live serving like a search index or whatever and and reboot strap itself. Uh it's also used internally by the stream processing system as a way to maintain state for the stream processors
There's a bunch of use cases for it. Um if you Google Kafka log compaction, you can learn all about it. Question. So because the the client tracks the offset Yeah. I know it's possible, but it seems that Kafka's not ideal for work distribution like some of the other Qs do. Do you sort of agree with that or Yeah, so the the question is um because the client tracks the offset and the offset is that position in the log, um is Kafka ideal for uh work distribution or maybe it's not. Yeah, so um and and work distribution in this case is like a lot of what people would use queuing systems for is you kind of like put a task to do into some queue and a bunch of people feed off of it and it kind of load balances. Um so I would say actually the thing about Kafka, this is kind of a big trade-off between Kafka and other messaging models
So our model is a totally ordered log, right? And we shard that so you actually have a collection of totally ordered logs. Each um consumer gets one of those shards and processes it in order. So so it is indeed less fair potentially for work distribution. The good news is you get your data in order. Which you don't get otherwise. That that I think is essential but but you uh because in a traditional queue everybody kind of fights for the next message, it is more fair I think in in uh distributing load to consumers. Now where this matters is cases where um each item that you have is going to involve a lot of processing beyond the overhead of getting the message. So it turns out I mean at least for our use case that's a that's a minority of cases
Usually it's the throughput that matters more than like you get something and you turn on it for, you know, a minute. However, if you do have that case, I think the the balancing is maybe more of a concern. Other questions? In the back. Do you have a use case where you don't really have any kind of fancy analytics you need at all? You just kind of want to aggregate it and keep it around just in case. Um yeah, yeah. So I mean the way we run this stuff is like as a service, right? So people put all kinds of in there. Yeah, I mean and uh like we try not to care too much cuz unless it's a lot of data, I don't think you really need to stress that much. You know, so so really when you you know, the only people you really need to talk to are your top like 10 abusers
Those people are actually usually doing something relatively important uh or they're just crazy. Um so so yeah, yeah, there's definitely stuff there where it's clearly being published. It probably shows up in Hadoop where we retain it, but there's nothing really deep or fancy going on with it. Like we have I think we at this point we have like on the order of you know, a thousand or so topics. So, so yeah, I doubt that there's some person deeply processing every one of them. But they're all ending up in the HDFS right? Yeah, yeah, yeah. Yeah, but but but again it's like as long as as long as you're able to scale like in general you just look at your big space users. You know, most most things are small and so you can have lots of those
Question? This is kind of a loaded question. Yeah. What issues do you see using Kafka as as for an event sourcing system? Mhm. As in building your entire system event sourced, what issues do you see by using Kafka? Yeah, okay, so the question is if you if you're if you're building in kind of like an event sourcing model where you you kind of retain all the list of changes, which is very much like this log centric stuff I'm talking about, what would be the problems with with Kafka in that model? Um I mean I guess I'm probably the worst person to answer that. So, first of all, I I like that model. I think it's really nice. I I I I don't know as much about like doing application development that way where like literally your application is taking that whole log of changes. I I don't actually know how people handle like partitioning and and scaling data when they're doing that
But I have thought a lot about building infrastructure this way with some external log. And the infrastructure is handling the the sharding. I think it's a fantastic model. So, so yeah, I mean I think it actually works reasonably well for that. The if I I guess if there's probably two cases here. If if you have some kind of high throughput volume of data, that's where I would say we're like solving a problem better than other people. If you don't, I actually think like you know, hey, maybe like my sequel is fantastic and runs well and there's no problem and everybody knows how to run it and it works good. So so that's that's probably the only caveat, right? I mean, I think the only thing where we're really adding value or doing something new is if you're if you're operating at, you know, fairly large scale
All right. Cool. Thank you guys. Thanks Jay. That