Jay Kreps: Confluent and Kafka, SF Scala @Nitro 20150115

Jay Kreps: Confluent and Kafka, SF Scala @Nitro 20150115

Recording: Jay Kreps: Confluent and Kafka, SF Scala @Nitro 20150115

hello everybody this is SF scholar developer series when we talk about interesting technologists interesting companies behind them and today or gases j Krebs he is the founder and CEO of confluence it's a new company which is product izing kafka well-known open source project originally start which started at linkedin and used in a lot of companies especially scholar center companies because it's written skull itself a lot of people use akka and Kafka and also Kafka to feed spark streaming jobs and storm streaming jobs so Jace our guest today and we're going to talk about the origins of this project use cases and kind of future so I guess I'll start welcome Jay thank you for having me I'll so remember i'll pick up on all of our previous conversations when i visited linkedin a while ago also a special about Kafka you mentioned that when you join linkedin you wanted to do all this cool data science right so your original interest was to do data science and then you realize that there is not much or not enough infrastructure define it and so then you essentially became an engineer and you build multiple systems maybe we can kind of apply the history with voldemort social graph so I wonder how does this trajectory happens and it mirrors what happens in Big Data now we have data scientists who want to look at data but its necessary to feed them this data see you a show you when did the data pipeline route so how does the decision happen in and maybe like what challenges people face you know if you want to be a little fancy do you need to know about data by blinds right how how does this whole thing works in companies like living hell it work for you that make sense yeah so I came came out kind of like a machine learning background and I done gentle some related work in a web company prior to going to LinkedIn and so I came really because I thought linkedin would have the most interesting data or any social network would have the most interesting data because it was about people is really a kind of new application for data stuff and that was totally true and the the challenge was really that we were operating at a much larger scale than any of the stuff I had really faced and and so a lot of the kind of off the shelf machine learning techniques were totally inapplicable and then just a process of actually production izing like a recommendation engine or any of the stuff was was really intensive and we really didn't have any infrastructure or support for it and you know we at first we were kind of trying to solve those problems in the context of building products but eventually we realize like hey if we could if we could do this in a more infrastructural way we could do it across all the products and really have a big difference and so that came out in a bunch of different ways I worked on the Hadoop adoption and we kind of moved a lot of our big batch processing that compute the people that you may know on linkedin into that and then we were able to significantly fancy up our algorithms and score right just by just by virtue of adding more data and having more computational power that really kind of gave us a platform to be able to iterate on that and I built a key value store Baltimore that you described it that was really used as kind of a serving to serve the live serving and for low-latency access yeah so so over time the what I came to realize was you know one of the most interesting and unaddressed areas was actually a synchronous processing so so batch processing of course is asynchronous that's like very delayed but but but maybe there's no reason that things have to wait 24 hours to to get processed maybe you could do things kind of in real time in a stream processing mode and I felt this was really closely connected to actually making the data available two different systems available in Hadoop available in you know for surveying all these kind of pipeline questions that were kind of at the core of really making data usable and so yeah that was kind of how I became much more of a data plumber and I'd always had an interest in distributed systems just because it's an interesting area but but that was really I think and it maybe still is actually kind of the most pressing problem I think to be addressed and I think once you know once these technologies get a little bit more mature these platforms get better I think of course that stuff fades into the background you can move up the stack and focus more on you know what's the point of the data yeah yeah so it's always it's always about where the most pain is at any at any time and certainly for these large companies that have this kind of massive data sets it's really where a lot of the pain is today and it's just starting to get I think a little better with some of these Big Data technologies and so the the result of that was we came to focus on building kafka which was meant to be kind of a next-generation messaging system it was very stream oriented but it was built much more like a distributed database more more than a traditional messaging system and it was built to be kind of a backbone for stream processing and so we spent a bunch of time kind of operationalizing that get that gang on to run at large scale and then getting everything that happened in LinkedIn represented as a street right which is actually not that intuitive so most people are aware of course that like a log you know application logs error logs can be thought of as a stream but but it turns out even like database changes and things like that can be thought of as a stream and that's a actually really good way to propagate that kind of change in process that's had to change and and so we we kind of built this facility that ended up being kind of the core data pipeline for the company that fed everything else it was you know the platform for a bunch of really cool applications that were built and we liked it so much that recently we've kind of left linkedin to start a company around it yeah that's great and so I mean we were enjoying first startups where I work before kalke for for a while and so exciting to see that kind of become its own self project and immunization to find it so I wonder uh it's it's a very happy really very happy outcome for open source project right I think this is the best you can hope for when you started within the bigger organization it's motivated by use case and then the kind of gets enough community support and organization self kind of helps calm suspend off I wonder I mean a lot of open source folks would want the to happen so what is special about LinkedIn and at first it let you incubate and germinate this infrastructural projects right obviously like you would probably you had some other things right brayla the process you know in common sou you were able first to have an infrastructure group where you would think is systematically about what kind of but once you need in the first place and they're like you said you had you actually had clouds to convert organization to think in terms of that infrastructure yeah what is it about LinkedIn or about you about your group which made it happen in the first place yeah it was something that came around fairly early after i joined was that you know the engineers of course wanted to open source of stuff they were working on and you know the company was crying why would we do that and we talked about it and linkedin was actually fantastic about i mean the realization is for a lot of these services or websites their core IP is actually gay its data and users it's not it's not actually their code and obviously they don't want to make all their code public but there's a large portions of the code that are fairly general purpose and if you want to convince people to make those parts good you have to treat them like products in the right one of the best ways to do that is to make it something public and so we started to do it I think really just because the engineers wanted to do it and then over time I think it became clear that the things we did that for ended up much better we were able to find better people to work on them the people were motivated to really document and explain them and really treat it like a first class product not just some half-baked internal tool and and that kind of led over time to this culture of really wanting to make you know our best infrastructure open source and planning for it upfront so and and you know I think it's very motivational for people who work on the stuff software engineers hate doing things twice and you know I had I had friends who worked on the search system at LinkedIn and their experience was they've gone from company to company and essentially rebuilt the same distributed search stack and this was their third take or whatever it so so obviously for people like that it's it's much nicer to be able to kind of build it once make it good open it up and be able to take it take it with you as you go and you know I think it's to linkedin's credit that they realized very early on that that was a positive phenomenon and not a not a threatening phenomenon and I think that's become increasingly common you see a lot of companies that do that now so I think it's a wonderful thing for for open source also home so know that you have that open source project how did it go within the organization like that all you know how did the adoption happen I wondering what's the mechanics of it were like any holdouts did you have to do a lot of explaining yeah and how long did it take to convert injuries with the structure yeah so the you know we I'd had this idea really early on for this messaging system that would be much more efficient and we'd be built like a database log and all these other things and you know I had this idea that we should replace our kind of main activity tracking pipeline with that and it was just a lot of work so no one really wanted to do it but then are our main activity tracking pipeline which was really just like we logged out to log files and kind of are synced the log files around it started to just kind of like fall apart under load and it was just not reliable and we were kind of hacking things to make it work and then then it became a much easier project to pit and so so we started with that that was a great use case because really high volume so really made us get the kind of like operability at scale stuff down up front which i think is very hard to add on later on and then we moved from that to you know all the kind of operational data and metrics and you know finally thing we're doing as I left with all the database changes that kind of capture a lot of the like core systems are now built around cough guys I kind of log so some of the storage systems or the starters in databases issue yeah yeah not only that but actually moving the some of the core data systems themselves to use Kafka as their kind of backbone so the news feed system which is a serving system in its own right is kind of like basically a big Kafka log and they're doing a similar project with some of the key value stores do this you want to go to a whole when the old one yeah that's right that the front page feed of activity in your network remember so yeah that's that's basically a big Kafka backed serving system which is pretty cool great great thing everybody saw that you know everybody uses lifting so that's a good thing to visualize are interesting so I mean I have to ask being a subscriber laser how come it was written scholar well the kind of experiences you derive from it and you know now that is open source column what kind of scholar ends what makes easier what it makes harder oh yeah I see that kind of yeah yeah choice going forward sure yeah so it's kind of interesting choice you know Kafka is basically a pretty low-level piece of systems infrastructure so the reason we did it was just because we felt that we weren't really sure that Java was going anywhere at that time you know this is still the Sunday's there hadn't really been a lot of changes or improvements in the language in a long time we were concerned about our ability to recruit top and programmers to work in that environment so you know kind of the pro of Java was it was pretty fast so you can basically build a reasonable system in it which is not true you know I think this type of system and like Python or whatever it just wouldn't be practical but it was still you know still reasonably efficient to build things in but but it was just kind of a clunky language and so so the attraction for us was just you know a more modern Java that had you know a lot of the more modern language features available that was what we were looking for it you know I think I think it mostly paid off the things that made easier the Scala code is more concise I think there's a fair amount of asynchronous action in any kind of high-end non-blocking server so where you you know you need to have tens of thousands of connections alive and you need to do you know asynchronous request fulfillment where a request comes in and it won't be satisfied until a bunch of other things happen in this distributed system and then you said the answer back so be able to represent that well I think some of the you know some of the functional aspects of language come into play nicely there there's been a ton of downsides the compatibility issues in Scala I think has been a huge problem for the client so the clients were finally just moving out into Java so that there's no dependencies and we don't have these issues of compiling four different versions and I think it has it has harder to get contributors i think is now it's like okay in addition to this like hardcore distributed systems engineering you also need to know this language but increasingly that's not been the case because now it's so prevalent that that there's really a good base of people who know it so so i think that's less of an issue now okay interestingly we've never actually run into significant performance issues which is unusual for a new language so we've had very little in the way of that you know few issues we're like yeah we had some weird implicit that we're making zillions of copies of it very very rarely do we do we hit anything like that which was nice that's good here so you know there is this metaphor of skull tribes where you know there is like a fe extreme yeah 3 scala as java and yeah that we were kind of middle there old how do you characterize your you approach you know your codebase like if you know somebody wants to pick up scholars Kafka can they use it as an example code base can they yeah yeah yeah so it's interesting I you know I don't know if it's good practices are not it these infrastructure projects end up being kind of their own world you don't yes and and most of the like we don't really depend on hardly any libraries because we're like the bottom of the stack so it's not like we have a ton of interaction with others khalaqa so so who knows if if scala people's call the people may think it's terrible are good we're definitely more conservative in our you know scala feature usage typically the the motto of any of these types of infrastructure systems is like no surprises so so we're relatively conservative there we were we're probably close ish to java although close so you know i don't i don't know it would probably be up to somebody more more proficient in the language to judge the alternate quality hundreds of reuse like do you do do consciously try to use as much responsible and kind of structural libraries this again they have there any Kafka libraries of components which other people can look at this example yeah sure I mean any of these systems is broken up into core subsystems and so the subsystems for Kafka are the network server which is just a standalone network server there's nothing really Kafka kafka about it so it's just an ni o network server there's the the log implementation which is actually what's writing out data to disk and check something it and dealing with yeah and you know some scheduler which runs background stuff and the whole replica management system which keeps track you have leadership and fell over and stuff like that so so there's a number of these kind of core systems that are there they each kind of stand alone for the most part with some amount of layering between them usually the lower level ones like the network of the log are kind of the easiest to grok independently you can just kind of look at it the log looks a lot like a commit log you can kind of see what it does so a lot of folks now implemented ribs and systems and I see like several new databases are coming up and startups right and I've seen a lot of implementations of raft or boxes yeah yeah so can somebody for instance take your leadership election algorithm and use it in their own database conceptual right is it is it something because we know that cop is one of the biggest disability systems working tremendous through post on suburban whereas Bruce's relational is proven in terms of fail over it in terms of throughput so can somebody kind of piggyback on that and you know yeah yeah so that's right that's right so so it turns out that a log is actually a kind of core abstraction distributed log is actually a core abstraction in distributed systems and so so this is an interesting areas not not very well explored it's something linkedin was doing a bunch of and a few other companies which is you can actually take Kafka and use it as a log to build a storage system or some kind of you know newfangled serving system you know what you would do is you would couple it with some kind of key value store like rocks DB or something like that so you have kind of serving nodes which have the rocks DB and when they write data to their stuff they log it out to this fault tolerant log and that basically takes all the distributed programming out of the system implementation so all that correctness and fault tolerance is punch it off to this log and your system now just has to deal with the how do I fetch stuff off off disk right so it's sort of related a little bit to this kind of what do you call it command query responsibility separation or sourcing yeah so this is maybe the kind of large-scale partitioned version of that so for and especially for data system so so you can absolutely use cough gun that way nothing internal to Kafka i think is being used you know as a library elsewhere I think making like a reusable you know I don't know scheduler or network layer is actually much harder than building a good you know network layer for Kafka you have to really do a lot to kind of file off all the edges so I think there's a big difference between like Kafka's network layer which is you know probably 600 lines of code and you know Yeti which is probably like 30,000 right for what Kafka does we don't need any of that other than the six hundred lines but to really make something reusable across every possible use case you actually need the rest of it right so so so yeah I try never to expose the internals of the system because it's hard enough to get the few api's you do expose right hmm ok mother man I think Scala folks are always asking for interesting systems to look at so no cough cough maybe and they're just an example for somebody to them to look at I understand that kind of see ya so for people who are interested in distributed systems I think it's a nice mid point where it's it's not so large that it's impossible to dive in you know there's probably probably bout fifty thousand lines of code and yeah yeah no it's not it's not huge um like a lot of that I think just comes from Scala like if you were to rewrite and job i'd probably be bigger they're not a showing more meaningful lines of code but you discover but so yeah it's small enough that you could kind of sit down and basically come to understand it if you understood the external way that the system works you know in a reasonable amount of time without you know devoting your life to write okay so i only kind of should give some didn't ask about confluent and we also know how this interest and evolution cannot follow the point of reference so we have lucy's like several related project so often Kafka is used to feed spark yep bridle spark now is a data bring back company in about 50 other the more funding yep am mesosphere you know used as a foundation some kind of scale right now so i think given the user base for Kafka a number of pieces I can easily project yeah into yours you're looking at the same scale yeah you better looking at journalism there you know San Francisco and you know hundreds maybe thousands of people venture down the road yo pense yeah hopefully so I mean give it a given so you know to be interesting just you know to see this you know couple years later right but i wonder how do you see what is your vision to basically take this open source and evolve it to the backbone of lots and lots of companies yeah oh yeah how we're going to grow the company what's your kind of strategy around open source sure you know what's your vision look up yeah okay that's that's a lot of questions that's all right let me let me start with the beginning so yeah I mean I think it's obviously like an exciting time for open source companies we're open source works really well i think is is lower down the stack so platform things you know programming languages data systems infrastructure stuff these are the things that i think just are hard to make into proprietary software companies these days it probably have always been hard to get off the ground in that way open source is a way that you can actually get to some level of usage to the point where you validated that this new idea works the interesting outcome of that is you're actually you know i think the first generation of open source work was largely commoditizing things existed so Linux is kind of a clone of you know the proprietary UNIX systems yes obviously it did a ton of innovative things internally but its external interface and what it does is totally understood it so you know product category that's well defined my sequel is I think in a similar category was taking relational databases it's kind of a commoditized version of that in those cases you know especially in the early days you say oh I'm not quite as good but much cheaper but what you're actually seeing that's really interesting now is all these open source companies are actually doing really innovative stuff like really innovative stuff like like mesosphere data bricks a lot of these companies I think are you know really kind of trying to define new categories of software so i think that's amazing it's an awesome time for it this is like one of these rare points in the cycle where the stack of things is getting rewritten you know i got this opportunity will be around in 10 years probably things will have calmed down a little bit more and we'll have a much more stable stack of technology but much less like new creation PAP or mature acknowledge yeah yeah so how does how does confluent fit into that you know basically are our belief in our vision is it this kind of you know real-time flow of data and representation of data as a stream is actually a really powerful thing and it's a really powerful thing that solves a really ugly important problem for companies which is around how they you know funnel data between systems how they process data there's a lot of kind of nasty batch processing and you know our sinking CSV files around and this kind of stuff is ugly it's expensive to have people doing it we think there's a better you know much cleaner way to do this at large scale and you know kind of thing that's new is the ability to really you know build a distributed system that can kind of have a single instance of the heart of a company and have everything plug into it that's that's kind of the vision that we're we're trying to make come true so that's our that's our belief about the world obviously to make that happen there's a ton of things that you have to do and that's that's what the company does right so so you know there's there's a kind of normal things you need with any software which is you know support and training and all that and we're doing that but there's also just this whole ecosystem of software that's grown up around Kafka that has to be matured and hardened and all the tools you need to run it and that's what we're really putting together you know we we my belief is when I looked when I started on this looking at what was in LinkedIn I felt you know really if you were to start from scratch about a third of the things that happened in a company like LinkedIn would probably be in this asynchronous you know stream processing domain right maybe uh maybe a you know a smaller percentage probably would be fully batch and the rest probably the largest percentage would be real so the like all the client interaction yeah yeah yeah yeah and yet yet when we were starting on this project if we looked at what we did very little was in that jahmiya the reason was because there's no infrastructure support for right and i think this is actually probably even more true of many other companies and it is on linkedin it's just that we started on it there earlier so so yeah that's that's our big bet that's our belief about the world and you know that's what we're to make come true right so it's it's interesting right like if you look at companies like spark so it has a lot of enthusiasm right yeah the open source community actually drive it got it yeah uh and it kind of you know it the the fuse was ahead of a business model right barrows it's undeniable that people want to use API yeah right i think personal that in my kind of simplified view of the world a spark is dissimilar skull he took scholar collections and president is really beautiful api to developers now they can do MapReduce in a small and they can do the same thing in the launch ever and that resonated yep yep and in a way streaming processing there's a nice yeah with data pipeline yeah I think that's kind of why a lot of startups are picking up Kafka and in kind of skull folks understand this earth it's very predominant among yeah let me so how do you kind of harness this enthusiasm do you for instance let it grow organically I mean obviously spark is you know spark meet up soon as one of the biggest get-ups in the world it was actually spun off of a subscale out two years ago now it's about you know twice as big about this you know 4,000 developers and I'm growing rabbi that works constantly supported NP and fills it in different cities do you have a strategy in an alt reach yeah banjul isn't like that you know what kind of your think in there yeah so far it's just been totally organic for Kafka and we're just starting to actually try and get involved and help that and it's amazing how many things are actually happening like different companies getting involved you know the people from different web companies are now you know committing code at all the time and just just watching that spin up is super exciting so we haven't done much yet to really kind of help amplify that but but yeah we absolutely intend to it i think the right way to you know if you're if you're trying to make some kind of platform change in the technology sector is a very very big change to make right right people don't like to replace the platform very much because that's what everybody stands on you don't want the platform to break so it only happens now again if you want to do it the way you have to do it is you know you have to make it open like nobody's moving towards proprietary platforms at this point you really have to make a kind of open sore standard thing that is a truly community-driven in the sense that if any one company including confluent were to die this thing would continue and it would not go away that's all that's what people want to adopt you know in second you have to actually express the idea that you have about the world you know how should you be doing it to everybody else right and and the way to do that is again I think through through a really rich community I think what this recognizes is how much of software is actually in people's heads it's actually human capital not intellectual property it's actually human capital right it's things that are in people says it's what people know how to do it's how they conceptualize the problems that they see and so so yeah I don't know the right answer to your question but I think the result you want to see is you want to see a diverse community with a bunch of people talking about it people at different companies people you know using it all those kinds of people and yeah the role for these open source companies are centered around a technology is absolutely just to help help amplify that that voice and let people let people speak okay so you know since there is a spark samantha mathis connection the owner what's the prediction when we're going to see the first quarter conference oh man it's just we can get it together there's a lot right now we're just trying to get office furniture okay okay yeah i'm looking forward to it too were able to come in together okay and I guess I mean the last mission Alaska uh it's so this is fine balance right between open source and proprietary anything in case of a lot of companies are doing support and training yeah right and some companies are picking pieces we should I keep closer such as you know the divers cloud which is which as a platform right again and and I wonder what is your plan here are there gonna be pieces of confluent kafka we should be closed source and if they are like how are they can interact with the open source parts sure what's the visually yeah that's a good question so so yeah like the balance with any of these companies is you need to have a working business in order to fund the company and so so there's lots of ways to go about it you know what I like the model I like the most is companies that it takes some piece of infrastructure like like Kafka and they take a use case and they helped to build it out as a product right and so we're a couple of the common use cases around data integration and processing and really trying to productize those what do you need to run it what we are building and selling to people in supporting is really an open source platform that's what you build around that's what you plug into but there are areas I think on the side for applications that help you run things and so on that can be proprietary it has to be optional it can't be your your core commitment otherwise you don't want to adopt it right no one wants to be on like a forked version of an open source product they don't want like some weird you know proprietary kafka that they're getting but but there is a there is a i think a lot of opportunity for you know a company that wants to adopt it faster and get going and run it easier to basically you know pay money and get some piece of software that helps them do that and that's that's a totally legitimate and good model and I think you know you can look at these open source companies and see a number of done that successfully and we will be as well great very great so do you think that it's safe to say that in open source people picking up a source kafka github will always be basically ever able to to work with let's say they take it first on the test drive at election on production will they always be able to kind of upgrade to confluent offering without kind of any measure like it will be like a kind of all benefit and no problems will it be getting always the full version of data on the platform and like ooh the closed source pieces would be monitoring and kind of debugging or else taken out like do you think the some functionality itself or in terms of sly performs can change in the vapor room mmm yeah I mean I think one of the important things you can kind of look at these different open-source products and companies around them some of them are very open source and some of them are kind of not like there's no real open source community like they publish the source somewhere but it's under some a license where you can never use it and the usage of those systems ended up being really different right and there's nothing wrong with closed-source software there's nothing wrong with open source software that has no communities right but for what we're trying to do where we really want to be kind of at the heart of the company and really be this kind of interchange or hub for data that's a plat form thing you're not going to want to be stuck on a proprietary version of that one of the commitments for that to work has to be that yeah not only can you easily upgrade from open source to the closed source but but even more importantly the other way you need to be able to downgrade from the gun glue it operating to to just you know Apache kafka that i downloaded off the apache web site and what we're hoping to do is actually just amplify the work of the community like a lot of the gaps that you know constrain things today are really you know the set of clients for cough gathers there's a dozen open source projects that have different clients in different states of maturity none of them are tested against the main you know server when it's released not all of them are even feature complete so just being able to kind of package these up and make sure that they're robust and they work with each version and extending that development getting people to help out on those projects and advance them along like that doesn't need to be proprietary I think that adds a lot of value and kind of helps amplify the work that's already happening in the community is totally the right approach for this stuff right so I guess you know lost the kind of question will have you know if folks heard about Kafka they think it may be useful to business what advice do you have for all the startups on health at how can they start you know what should they do to check out if this is something which will kind of help them react attack their systems phone you sure hmm yeah that's a great question so um so the documentation on the kafka site is actually pretty extensive it'll walk through a little bit of the use cases we're putting together more material to help walk people through more specifics of like hey how do I build X so we're going to start releasing these things that's blog posts and stuff as we start to have that more built out I did a really long blog post on kind of logs and the theory of it that people can pick up it's even in a Riley book now if you're interested called I heart logs that's right so all of those are great sources and the the mailing list for Kafka is also very good so if you if you feel like you have a really weird use case that's not discussed anywhere you know just just send an email to that mailing list was usually a dozen people will you know spring up and give you a variety of ideas okay and also mention that we J give a talk call I hard dogs at escala earlier in 2014 and that's also available in the same TV channel so you can look it up we'll put a link to it from this one and you know you can use to get educated on cough can get it going thanka wrong g sorry oh yeah yeah thank you thanks