scala Neville Li interview 3 23 16

scala Neville Li interview 3 23 16

Recording: scala Neville Li interview 3 23 16

hello everybody I'm Alexa crop the organizer of SF scholar and here we are at nitro with a very interesting meet up I have to talk today one is from Spotify on SIA which is a new API for Scala on the Google Cloud and have Nevel Ali here does he is the software engineer at spotify working on the infrastructure for machine learning yeah welcome never it'll be here is good to have you again you spoke at the last year scholar by the baby gate installer conference yeah I which is really cool talk so maybe you can start by just like tell us what you do is specify and what's happened recently while Google Cloud and what's going on okay so i've been working at qualify for four and a half years now mostly you i right now i work with the music recommendation team mmm and that's the team that built discover weekly radio stations all sorts of music reclamation and related features I mostly work on infrastructure tools for doing machine learning like scouting spark mm-hmm and we use a lot of scholar of course and over the past year and a half we are investigating moving to cloud for many different reasons cost operation experience and all sorts of different reasons Google Cloud is very attractive because they have 10 or more than 10 years of experience building big data infrastructure like one of the killer app is big query mm-hmm so they invented big table and among other things so those are the interesting things that we want to investigate and it turns out and they worked really well for us mm-hmm they also have this new framework or data flow which is based on Google's paper first paper Camille well two years ago and then last year was a paper called data flow basically describing this unified model for doing in post patch and streaming in same application and they release the Java SDK and of course we want to to have a scholar API so I have to run it and then build something all right so it's super exciting so I mean we heard the names but you know don't really know much about it so like why is it useful for coming like Spotify do do data flow what are the typical use cases well first off data flow can run on gcp is a hosted managed service which means you don't have to set up your own Hadoop cluster spark cluster you just run a job from your local chameleon and then make requests to the to the Google cloud service and then allocates resources creates new VMs bring your job it handles like auto scaling and scales up and down mm-hmm it also integrates with all sorts of google data products like hey it can read and write back or a big table and GCS so these are the very nice interesting features for us of course bigger is really fast and highly scalable to be able to do a lot of its in this sense it's very similar to what people are doing in spark likes doing sparks eco inside the Scala code mmm for your typical etl slice and dice mm-hmm tasks and then euskara to eat to your like custom machine learning functions you can have both best of both words inside the same application do you get the machine learning library or do you have to do your own right now the API is a more closer to sparkle like the standard LED API hmm so not much machine learning but that historically has not been a problem for us because we mostly use our own custom our wisdom so we have to write from scratch anyway mean it's nice that spark has all the MLA libraries that mean it took years of work to build all these different libraries but we use I feel like a home rule or wisdom so that's not a problem for us well Google is also working on tensorflow which they open source like a while ago and they are working on a cloud version of that we are just adjusting merchants today yeah okay well there will be integrated into data flow in the future so I mean basically you have machinery inc kappa inside data flow so you guys before moving to Google where are you on Prem so he'll your own hardware we had a few pieces on a diverse mm-hmm we we started a Hadoop cluster on preamp which is basically a rack of service in some office mm-hmm and then we moved to you TMR and it was too slow that we that lasted like a couple of years and then we move to our own on-prem loop cluster right now as of today it's maybe twenty three hundred machines largest in Europe we also up before the Google cloud migration we have our service infrastructure guinea I'm pram okay yeah i think i met your Hadoop guys at the strata last year a basin in Sweden yeah right and so so the cluster was actually eager in Europe interview interesting and so you're gonna basically like it's the plan to replace that with Google Cloud yeah but also because Google has many data centers we are storing our data Europe as well right there are some legal reasons for that i right don't I won't go into detail Tommy there is a law like we need to keep up your data exactly European customers yeah for your customers yes yes so so I'm just gonna pee I wonder if you if you can elaborate because folks mean maybe not dunno what bigquery is all right so like you know for people who familiar with spark right like what is the typical use case like if you can give a hypothetical example what is the data set like what kind of query would you use you know for you know in bigquery for right and how you would pump date a kind of like outlining very general like what's happening so I mean at the beginning of big data there's MapReduce and then people introduce Seco engines query engines on top of Hadoop which translates to MapReduce it's like hive yeah but most of them still like performs in a very old-school MapReduce fashion which reads all the data like row by row it's inefficient and doesn't have any the indexes or advanced data structure that traditional sequel databases have yessir Curie has introduced a columnist storage it's a you can read out from the Dremel paper like a couple of years ago mm-hmm and basically storing data in a different format with a lot of tricks to do like compression and blog index is look up mm-hmm so that enables you know C code to run like 10 or 100 times faster then then your average big data seco Angie hmm and also bakery has its own hardware infrastructure so you don't have to set up a Hadoop cluster and you'll manage your own own you know hardware and software stack so that's a huge advantage over the traditional big data on a sequel on big data solutions yes yes yeah about I mean I'm just curious like given the kind of that can you give an example of an actual query if you can well I mean today at the demo the bakery Google Beck really was demoing basically performing like petabytes of see whole scans through petabytes of data in a couple of minutes hmm so that's a huge speed benefit okay yeah okay and so I remember you were doing operon parquet mm-hmm your self yeah right so obviously this is the same set of approaches right there's cold storage in the direction so when you do this in in google cloud how do this skills translate like do you give this to you essentially implemented under basically parque was inspired by the Dremel paper it basically implements the same thing like s columnist storage on Hadoop mm-hmm but RK is a pretty low-level API so you still rely on the higher-level libraries to leverage parque in our case we used to do scouting everyone okay which means the programmer the users will have to code in a parka API which is kind of tedious yes not very nice I mean there's also a spark Seco which is very close to what your creators quickly write see coco de translates to park a api that does the slice and dice under the hood here's the downside of spark of courses you have to manage your own cluster and the cluster size determines your your throughput and the scalability and Google obviously has puts like years of work into bakery so that they typically can realize thousands of workers in parallel and processed petabytes of data in practice we have an scalp spark to that size yet mmhmm yeah so what's going to happen to your spark or fall so they can be replaced by the Google we are still going to have there are certain things that spark offers that's not available in any other options tools like interactive mode or MLM and the graph X our wisdoms so I see spark has hits a sweet spot of the medium-sized in memory data like interactive data modeling exploration so for data scientists like developing your algorithms it's a hugely useful tool yeah yeah yeah I like I think interactive prompt is really important for scholar developer because we are used to that like one yeah cuz it sounds like spark as to me it's basically a ripple back by cluster right yeah you know as a possible Mac and architecture wise it's also very different spark as the master node and the master node drives the workers mhm I mean if you compared to like scouting and indeed a flow they are both much less basically you have the job execution graph is determine ecstatically it has to be determined before submitting the job you can do the the interactive kind of process processing liking spark okay yeah and so obviously you know you did the skull API so what is the kind of scala adoption that spotify who are the kind of people you see as using this skull a guy how do you kind of this knowledge spread through through the company right now we spotify a little history history as spotify you should be a mostly python shop mm-hmm and then we moved to most each other because we have a lot of job expertise from X Oracle engineers etc but Scala is really picking up as a data processing language mm-hmm most of the ML people using scholar Scotty and Spock all right a lot of ever data reporting engineers you know engineers to etl simple pipelines they are also picking up these tools mm-hmm and also head like for this reason having a dataflow API in Scala is a office a really easy transition for those people mm-hmm as I I or demone a bit later the data flow the shield API it looks probably eighty percent identical to to spark or interesting yeah so called shoe yeah SC I oh cool yeah it's like Italian spelling yeah well the name was suggested by Italian die Oh interesting yeah she all this awesome yeah cool so that's that's very exciting actually yeah and then you know situation et al I got to ask you we have this kind of semi joking website called motel before right and so they think the iid like if you look what happens in in spark data flows and Kafka you don't really have it here because you can if you typed objects at the inception level API and they're falling through the system so you you kind of never hope to extract them and right there just transform ya so do you see the same flow happening in Google yes I mean that's exactly the point of data flow if you watch live streaming I think you can watch it live tomorrow we'll have a talk it's a by ego this what if I engineer me like we share the same session but he will cover the streaming and etl sort of part of data flow where we use pops up which is Google's insert to Kafka basically hmm and and data flow to to process or aggregate layer log data in real time mm-hmm so yeah I mean in a sense data flow can answer to answer some of the ETR and log connection yeah so that's like another way to get rid of ETL bless you i flow yeah also so thank you so much we're looking forward to your talk and kind of see dimensional bigquery one thing I'd like to mention as well is that you know we have this conference call data by the pain may we have one of the senior engineers from the query team and Seattle carbon and explain how this works I think it's very exciting that we have now in distance color community right because we're kind of through your API where getting like all this goodness from google so so what can we do right now we're not at spotify like your open sourced the shower like how can you play like if I shouldn't get a google called account can I start playing immediately to see well the nice thing about the data flow SDKs of course by default a random google cloud but it also has a local runner you can even run us back or flink moons our work in progress but it's possible so you can totally ruin those on a laptop with without even paying for Google account uh-huh and Google is sta they mentioned today they have public data sets are bakery which are free to query so that's another way to try it out for free alright so they have like public domain data from nonprofit agencies or whatever uh-huh now you can try out cool so as soon as we learn about Shire we're going to play this on google this is fantastic yeah awesome thanks no problem thanks