sfspark.org: Q&A with Robert Metzger, Kostas Tzoumas, Stefan Ewen, by Alexy Khrabrov
Recording: sfspark.org: Q&A with Robert Metzger, Kostas Tzoumas, Stefan Ewen, by Alexy Khrabrov
hello everybody I'm Alexa crabber of the organizer of the new SF spark and France meet up and today we have our inaugural meet up at galvanize a great data science place and with us we have the three co-founders of the open source project Apache fling a lot of folks interested in data streaming big data space heard about it we have not seen these guys here yet so this is a momentous occasion when I look I have all them Buddhist here and I'll just let them introduce themselves and maybe say a few words you know about themselves welcome sirs thanks thanks Alex so I'm Costas I am together with Stefan Robert part of the original team that created fling initially out of the university research project took it out to the apache software foundation about the incubator appetito global project I think for the invitation my name is Robert I'm a software engineer at data artisans I'm a committee at Apache fling project yeah and I've I'm working on various components of the filling system for example the yarn in decoration the api's and a lot of things that users in touch with every day when they using link yeah hi I'm Stefan I'm like Kostas one of the people from the university research project link was or what became flink was I was a big part of of what i wrote my PhD about and yeah and after that um we took it open source into Apache and uh now and I'm actually happy to see that people that people like like the stuff and I'm picking it up thank you so basically I will just ask you know a few general questions right so the the spark and France audience mostly now knows about spark right so the response i meant happening in town a lot of folks are walking from there I think spark streaming is one of the very interesting pieces of spark we've seen deployed more and more in production there is a lot of different other steering systems notable of course storm which started kind of the streaming revolution kafka now has Sansa so I wonder you know what is kind of the evolution of your system how did you guys think about it in a few words and I know that streaming is very important piece of this in kind of what made you since three me how do you see you like spotting the sum compared to all of these guys sure so the basic premise of link is to see data analytics as streaming first so the basic premise of the system is to do both baths and streaming analytics on top of streaming engine which is yeah the main differentiator you had a few other projects in particular it is pure swimming dancing it can do pure striving but it can all support buzz processing of the web streaming so for fleeing but is really a special case for streaming mmhmm yeah and in the in the pew streaming space we see actually that that fling feels feels a bit of a sweet spot where where you can combine low latency with with real strong guarantees exactly once including a user-defined stage and answer you get good very good throughput with that so this is sort of a sweet spot combination of features that that other systems have um I sort of mapped out nice so basically where I mean it's very interesting because you know not every day it seemed like these kind of suddenly peers right and obviously a lot of you know Adam for instance knows about link and some other folks but majority of people for Susan spark another way of filling it and but if you look at the sack it's it's it's really full like there is a lot of we have a machine learning library have a graph library right if you have a lot of different components so I mean we're having has been like how did all this happen was it mostly European project Tino tu Berlin what is kind of the how do you like how do kind of move it from Europe to the Bay Area awareness space what is kind of what the history there give me like a few facts sure yeah so it's a it's true so the the origins of the project are in Europe original in Berlin other European institutions getting involved later but I really like to think of the project as an international project so have you know a lot of commuters and contributors both in the US Europe and Asia so it would be wrong for them to sort of label this is European project so I'm sort of against it and so sorry what was the second part of your question yeah and so what what is the kind of right right life like how would it how evolved like how abortion to the stage yeah yeah yeah so it's two drivers I would say the first is that we had a lens in that was powerful enough to power a bunch of different applications and power them with very good performance including club including graphs passive learning streaming and baths and so on so that was one driver the second driver is that this is really organic growth with the community growing and diversifying lots of contributors coming in with different backgrounds interested in different kinds of analytics and contributing back to the project okay it's may be interesting to add that like the origins of the project that became fling it was called stratosphere before more actually quite a few years back so this this project started in late two thousand nine actually so we've been when working on this at the University for quite a few years we it was always open source but because we never sort of went out and spread the word basically not nobody knew about it but we spent quite a bit of time lent laying I think the foundations of the stream processors so after after we decided to put it out open source and grow it we had a yeah we had a very very strong foundation of Rose Stagg fast these very interest because it reminds me of another University UC Berkeley and I'm well I guess kind of its resembles the trajectory and I think they started around the same time so that's uncanny that's that's very interesting so if you know if probably I would do the meetups in 2012 in der Lyn maybe I would have come across you but that was good here so i meant a right and we started you know talking about spark them so don't tell me about your program stack you chose the program languages how you know what was the favor technologies for fling oof yeah there's there's a lot I'm let me try and then sum it up okay I'm a bit of a difference between between flank and spark is that the court of link is java and the api's on top of scala where is in ensberg's i think the other way around hmm in in South language a combination of techniques we use um we've cooked up quite a bit ourself for example memory management utilization framework and so on because we had very very special requirements for that we're we're using also other other popular like libraries like a cough or distributed coordination then a lot of the Hadoop stuff for deployment and yeah yeah I'll go with Java are we using party with Java and partly with its not gonna version let's go inside and outside and then there's Java inside the skull that's what there's all kinds of accommodation yeah yeah it's actually right there the points we're using are actually interweave scala and java very seamlessly yeah you know business the errands of skol i want to kind of see the bigger ratios coincide but i would take to cal ipi is any day right like as long as it works fine all right like it's a java inside my power to it right like if i can use it from the rebel i can play with this yeah you told you totally can so you you in touch with the program language he like yes it's um well this is this is great yeah so below this like stream pipeline engine air pipe and streaming engine we have also different deployment options for example yarn deployment BAM insulation on the cluster but you can also run flink immediately inside your ided bucket locally we have an option for running it on test as well for fine-grained and resource isolations or you get elasticity in yarn so there's a wide array of opportunities to try our fling and we also have contributions external contributions for libraries on top of this API so we have for example machine learning library and the graph processing library and on top of the AP is and but also external program projects like a skating and google data flow which you can use with drink you see so i would kind of end with two pressures so first is euro open source project and we're meeting a bunch of developers right we have hundreds of developers already you know 400 of developers in the spark met up you know 27 front of developers and Scala meet up and some of the biggest data communities in the world here what do you want for the open source developers interesting link to help you with you know where do you need them to jump in or and contribute to this open source project yeah i think the the the nicest part to start contributing is probably with the libraries they they are the let's say the easiest to get into and they're also the ones where where you need the most user feedback because ultimately libraries as good as it meets exactly the requirements at the beautiful yeah the use cases and i mean their feedback on the use case is one of those what's really used for like give an example what can we try it like to do it should be like can we processed we threw them in common day the sensor data yeah totally so there's there's a there's a bunch of connectors for example two different two different sources kafka RabbitMQ twitter host bird client and and so on the HBase on yeah try those out see how they how they work where were you and and help us at the tooling around those two to make them work seamless word for a use case I think that is the that's the most valuable part okay ah and the final question will be you know where you're at the begin of the wild right right like if you look what happens to spark you know I think we have a lot of ingredients similar to this right then kind of my feeling is that by Reggie is ready for this and you know there is something interested is not happen here so let's make a prediction at one year way from now we are data artists on as well flank is what's happening give me some predictions some ideas yeah so so people think you're right I think you're we're exactly at the point where lots of people who are trying it out lots of people are sending feedback lots of you know people are starting to use it in production stress out the system so i think we're going to see a lot more of that mm-hmm I'm personally very excited about the data streaming space I think this is going to become increasingly important in the future and we're going to see more and more use case from that space hmm and and I hope that this will sort of validate architectural services to provide a good stream processor for people to use right so I hope you know it really takes on and we see a lot of interesting coming from wash blink and certainly will be happy to have you guys over at the sub spark and friends and we have your own meet up a patch of link tomorrow and definitely will be following them as well so you know welcome and we're looking forward to a talk yeah thanks a lot