Transcript: sfspark.org: Alexy Khrabrov interviews Sandy Ryza

hello everybody I'm electric robber of the organizer of SF Sparkle friends give it up about spark in its ecosystem such as Sakha Kafka Cassandra Pasha blink and you see simple men date that will hear about later today and we're very happy to have here without send ariza who is the senior data scientist at caldera and the author of the new book from o'reilly called advanced analytical spark which we have multiple copies here in the office and they're very much enjoyment great to hear and the topic of Senna's talk about semantic analysis of formula documents the patches park is highly relevant nitra which is a document productivity company we work with hundreds of millions of pdfs want to understand what they are evolved so this is the technology we chose to use as well so welcome to india is great to have you here thanks Evan here so I did some research on you and the cold air has a nice blog post type of meeting the data scientists and and we both know Josh real pretty well so it's kind of expanding on Josh reals theory of what a data scientist is a little bit and that but that was a thing a year ago so I'm wondering so it basically can appreciate that here the data scientist is a little janitor and and has also some interesting points but I wondering you know after that year hope they're wrong and I'm special after writing this book your initial vision change or was there some evolution of this and what what it is that's a great question yeah I would say that probably if anything my vision then reinforced in that a lot of my job is doing engagements at different customers oh yeah help you fellow come something you get it set up for whatever days whatever people want um and very rarely does it come down to actually i would say applying machine learning algorithm um often people turn to people were called data scientists to have someone to think about the data reason about it and try that sort through do the same common sense stuff that you know you might use another spheres but i don't know some people i'm confident in doing that in relation to Lord data sets um so yeah if anything I would say that uh that a Chandler Bing is still some huge percent of it and then another percent of it before you even get to doing math or statistics it's just sort of what makes sense here gorilla right over there I think there is a number falls in around that eighty percent of identify his job spam Monching the data do you think that's roughly the right ratio I think that's perf little better ratio and so I mean in my theory I I kind of also have some kind of simple way sore eyes were the scientists are so I wonder I know or they kind of exaggerated but I will run the Bayou so one theory runs that no it's also a data scientist a small insight is a Python programmer cook and you know sort and ready right and and and because typically i wonder if you agree that most algorithms most results can be obtained with the elder a delicious sort uniq take top tab right which is super easy to do a new leak stories because you can fight these things right and you can but you do explore it at once you can very easily as education whether you're sorting right so once you have something to soar it then it's probably was going to be here just trying to get things you know in the state that they can be ordered that's right you some shorts and transformed into pulse is sorting some field and then you get the red thread so so this code usually starts in spalding so the big date as I this is somebody who can sort the array loop or spark or something which spans more than one machine and then of course it becomes much much harder because even getting air in the air and certain it and it's become so so I wonder you know if you do this and kind of in their kind of the question is do we have two kinds of data scientists because machine learning is harder and very different thing and usually it's explained through our presence and statistics right so so in reality I'm not just that a lot of people who can really properly sort distributed array are different from the people who can reason about you know gradient boosting and various optimizers and very special algorithms and the the letter kind of people use the development matlab and the top for instance vision sciences program in matlab and and so they still have to integrate in some way so I wonder what you're taking this what do you see it caldera what you seen these are distinct merging they still kind of develop in a parallel what's your views um yeah certainly I mean I think in general people columns of data scientists or a great extent people that come from or math or physics background and people who come from more programming background um I think the whole pieces of having like data science as a term at all is like the idea that is useful for these people to get together and for single people to try to gain a little bit of both of these but these skills um you know in many situations you call the latter group of computer science group data engineers mm-hmm that's the breakdown that happens to be a lot of companies but I don't really see that division going away and so but grab this kind of unicorns who do both right i think you're closer to uniform right i mean the dream is that you are able to build frameworks like spark for example that makes it easy to dupe up right to work with you data sets to put things in production but also to easily let River knowledge you have about an asset istics hmm I'm in a way it doesn't require putting up in Java or a MapReduce yes so you're also a contributor to purchase park so a year ago you described the job is spending half the time with customers and how the time contribute into tools such as much respond uses steal the same ratio it's a good question yeah I think that in the very recent past I've moved a little bit away from sparking a little bit more towards the data science side of the job um yeah I've been on a lot of customer engagements not doing a lot of Education for Clara talking and speaking engagements this kind of stuff careful with yeah not enough to do it and I think I might be coming back a bit more towards Barca and near future like get tired of flying around them enough and when you are can you just park what a race are going to get into um so the two main areas are darn and then statistics functionality and ml is um we're probably a much more of an emphasis on yarn so I come from a Hadoop background huh committed on yarn mm-hmm so I've been doing a lot of yarn scheduling stuff right now one of the big things I'm working on is reviewing a patch that will bring in the ability to do executor replacement based on data locality preferences mm-hmm and this has been something that sort of it missing its park in a while maybe a place where MapReduce they'll have differentiation but this guy sigh sigh Xiao at Intel eyes thankfully taking this up and we're trying to move that forward oh nice that's something that's great to hear because I think little college is really important for performance so yeah I think it's very interesting that the other yarn so it's I think it's really kind of shocked eating the can kind of folks who can actually make real progress as far because you know yarn as a divorcee things right like how the actual raleigh's doesn't it's not obvious thing in you cover because usual star on amazon but then he got a production environment like natural once the question is how the actual wrong with the 47 really lightly have a liability of them is on what yarn or or mrs. right then the historically spark was developed on VESA sexually messes was the project but they was working on that his bond was a cute ab demo right then suddenly kind of it reversed and so messes with motive is the twitter but now there is Mississippi here and a lot of folks actually are running sparking messes so and I'm probably more familiar with companies running messages bitter and sigmoid only takes guys their bicycles very easy to come into a shop and install messes alright if you don't have the full cdh there already so I'm wondering kind of kind of emphasize or highlight the differences are in between erm SSN you know why they obviously by contribute to yarn then you you bet on the yarn or you're like you're so how would somebody new peak between yarn messes yeah it's a great question um I mean in theory they did the same thing I personally wish that we could just standardized on one of them um may supplement fine fine um but as it ended up handing out we have two of these I think what is sort of ended up happening is that people who are trying to use clusters for scheduling much of services I've got 10 more tomatoes and yarns become more of a framework or that high throughput data processing mm-hmm um so most people i talked to run spark on yarn it could be a fluid in some sort of insulated Cloudera situation but my hunch is that sport gets run a little bit more on your it does on missus I can talk about sort of specific you know sells for yarn ourselves for missus but in theory is you know their systems that do the same thing okay I hope you know I would like for us I'll be able to pick one and move forward with it but maybe to stuff with a flipper interpretation Oh couldn't do a bit of altaf achieve myriad which I think that's some kind of preparation of this ah I actually not her that is that something that everyone you're not Messrs that's what I you know what it does mommy know we don't have talked about it um yeah I mean there's been quite a few of my proposal to run yarns and mrs. and mrs. inside of yarn um you know it seems a little weird to me like within yarn you're already doing to level scheduling or are you already have you are doing scheduling and then sparking scheduling that's right we can hear on scheduling so why another layer is needed a nut sure um you know at least in Malaga maybe is a temporary solution yeah make sense yes yeah that was my question as well right because obviously it's already the way to do performances to basically make the schedule Kevin transparent like over assembly right between different frameworks so that kind of what my personal was ah interesting so so months definitely the kind of the Golden Age of spark right there is tremendous interest we see IBM announcing spark technology center right here on market street and get behind spark before us so this is something this is this is a fantastic time sourcing way some folks are saying maybe couple years ago that Hadoop is going to go with it is actually going to be sparked a dupe name is going to be there as a brand but in fact is going to be spark is going to be big data now do you agree with this um so I think it partly comes from the original completion with Hadoop and hadoop mapreduce hmm so yeah I wonder doesn't believe that produce is going to go away nobody knows how soon Burke has a lot of you to catch up but ultimately matters can be totally gone I think spark and satisfy all the use cases I'm not convinced with the other parts of Hadoop are going to go away not convinced that at HDFS is going to go away or or you know even if it gets replaced by some other file system those Hadoop api's I've sold away mm-hmm I'm interacting you know of the processing layer to interact with the storage layer and then the resource management aspect yarn on top of that um and then of course there's the whole broader ecosystem right boom century every sort of sort of project so of course those aren't technically had you but I wouldn't say that spark spark has extremely important role but I wouldn't say that everything revolves around it yes and so we see I think you see definitely everybody with also led to use in HDFS so I think there is not really any other simple alternative to just break into in stores and um txt files into it forever right or up arcade or doing that so what we see a lot of places running down some somebody came up with the term smack which is spark mrs. acha Cassandra Kafka so so it's kind of a new stack yes I think that's it folks came up with this and I told you know this should be an extensional it's my card which is how available resilient distributed is not in it all right you just by default it is so I'm not like a pirate that's right that's right that's right about so um so do you think that running spark on top of Cassandra and the rest of small suitcases ah you know people doing industry or it's still necessary in most cases to set up a HDFS and remember the full system I think a question I mean it depends what you're trying to optimize for um what cassandra is great for is if you really want to have like random writes and reads book both Cassandra in space no sequel store optimized for a huge data and then getting doing point lookups that kind of stuff if what you really care about is top performance for scans and doing batch batch data flip of stuff then I think issue if that's going to be what you want for a while mm-hmm oh yeah I mean it's also a little bit less operationally complex to deal with a no sequel database for just until with the file system right so see i don't i don't see i cain't are taking over but they're certainly valid right and the kind of thing that relates to another development so we recently CA a passive link and I am trying the space of streaming systems and today the second talk which would have is on min data which is a storm like very efficient streaming system so these systems put streaming first on patch second I flink make this point that there's a continuum and batch is just a limit version of a stream right so so where are you know a third of storm and then spark was any initial page but responded with mini patching now we have language basically do instrument first and beige being eliminated right so I wonder right you guys because they're essentially serve your customers right how do you see the space of all being do you see spark and flame kind of and similar systems co-evolving woofers is this kind of steering first base second base first in a second dichotomy is it you know kind of reduced oh this it's hard to change for smaller essence or fall fling to achieve the same best performance right i don't think it kind of relates to this edge defensive that because the normal setup right now she's done gigantic amount of stuff into the APIs then read some subset of it the membrane spark to beijing ready and ghetto streaming comes out of that one in production nice you put it into a stream in context but essentially assume this batch model so i think that's kind of why is it rather than that right yeah i mean so i think first of all you know giving this to the continuum is great primarily because it allows you to have you know ap is for these two things yeah what realize maybe even a while ago if you can create platforms that make you able to use the same code or fashion pre-processing um in terms of support for this fling so I would you know always say if you can one system is better than two so if it turns out that spark into everything definitely can do um having that in the mix you know additional overhead on the other hand if it turns out that flink sort of you know this business a long way to go i believe have both spark and flank to be a full training system whatever that means so if link is somehow way faster at getting there then the spark but i don't see any reason why we wouldn't you know in cdh let's say include that in our platform mm-hmm okay and I guess my wife ambition will be force on Scola because in a way of weeks Kyle shop we there's a real difference of Scala and and sabari can and so natural use a scholar of the spark and almost everything I will simply play app and akka so ah sure that's correct so and we have this big dance call conference nagas where we're emphasizing end-to-end data pipeline scholar right so basically we say noisy yeah we don't want export and report and low the rule to keep it everything in very type of manner and move around so I wonder if you see this is a feasible vision in the industry which is experiences with Scala you know can we move enough data science from Python to scala what would it take where should we you know ask our community focus on making data science in Scala when I kind of the cubicle um yeah great question so at first used a la maybe a year year and a half ago when i started working on this part i hadn't touched it before um since then I'm a total convert think it's you know the bee's knees best thing ever I you know I would love for everyone to use skull for everything whether that's a realistic goal of course is a whole a whole other question um so I've started developing on top of spark this time series library a lot of statistics functionality and stuff to make it easy to do concerts analysis on top of spark and so that I start building in Scala and you know the hope there is that we can add a lot of functionality the currently exists and let's say MATLAB or pine one package it looks that's models on top of Scala so yeah I would say that I'm a low-grade Scala evangelist so obviously Python an R have a huge amounts of momentum for data science I mean Java has a huge amount of momentum just for Kate engineering stuff yes um so we'll see how things pan out in the next five to ten years all right well thank you much we're looking forward you're talkin the is great to go again thanks very much less a