scala.byethebay.io: Xavier Tordoir, Scala: The unpredicted lingua franca for data science
Recording: scala.byethebay.io: Xavier Tordoir, Scala: The unpredicted lingua franca for data science
you [Music] thank you good afternoon everyone it's a pleasure for me to to be here and speak at sky by the way about I wats Cala and how we we are starting realizing it's a very valid language to undergo serious data science work in the enterprise so just a few words about about me so my name is Xavier and co-founder of Kensu formerly known as data fellows when I had one partner the other co-founders and the petrol out who you might know some of you and when we were two we were the fellows of data a mostly doing you know he Reverend speech talks at conferences now we are growing into a more respectable company and we changed it into something that is more how to say let's gender specific let's say so we're quite kind of proud to be open to to the world when we have iron people and so on thee is quite well known in the spark community and skyla community for its compute contributions and specifically for the spark notebook so a web UI an interface to work pretty efficiently with with spark that is leveraging the scale a language [Music] specifically and so with Andy we've been doing a lot of work either independently or where we joined our forces as a consultant in in data so we've seen a lot of things happening related to data science and how that work was conducted in different projects reads in academics where I spent a part of my time doing physics bioinformatics quantitative finance or in geospatial projects IOT and the conclusion we had is distributed computing is very important it's really the way to go and the industry is proving it those days but the way you do the traditional statistical and data science work is really hard today still to match this new environment and that's why we started working on a product that is really aiming at having data science sustainable in the modern world of data just a few two things I want to mention we are working with Alexei on a book skala for spark in production why in production because we believe that it is really the valid choice if you want to go in production in your enterprise to be on the JVM and Sky's imposing itself as the right way to go with data science we also so this this book should be in early release program already access program within a couple of weeks if we don't accumulate too much if we don't get too late then we organize that you know so we also have with already a collaboration so this this workshop this training on building distributed pipelines for for data science which are I think pretty cool because you get to see working really easily with docker docker containers kafka sprockets on drives on machine learning generating service to create the stuff so truly amazing stuff but I'm bragging too much here let's go to really the topic of this talk which we really want to talk about data science today and when we look at happens in enterprise today it's just the same work usual that means it's collected data exploring your data set preparing it correctly than doing the analytics work that is you know traditional statistics or more advanced machine learning but the environment and the nature and the kind of data you're working with is getting very different and that's why the traditional BI tools that are pretty efficient to get you know business where people are very efficient at giving answers to their managers is not enough today we need some other form of talents people like you who know how to code who know advanced mathematics and modeling of data to get more from the data okay and they have to be able to do that but from an excel sheet locally on your computer that from huge data sets that are you know in a distributed infrastructure so that's the first thing today we need scalability we need to be able to work on big data set and we want to be able to do that in production but as well in development you don't want to discover an interesting feature on the sample of your data because it might not be valid on the whole shebang okay and also because more and more people get you no knowledge of these advanced machine learning methods we want to be able to apply these models in the distributed environment so there's a need for exploring the data much more than than before and of course your manager wants to go in production that means the data science work that that isn't the end of you know let's say statistic 36 people so mathematicians that may not be the most proficient in producing production carroty software they have to get along with themes developing those those applications with the right level of quality and also in collaboration with the the system guy so engineers DevOps and that is the hard part so going from having a mother to having it in production can be several weeks process because everything has to be redeveloped the library that the data sales guy used in his favorite language is not available in the production platform so you have to find ways to to deal with this and then there is another aspect is that there's a shortage of talent in that era there's a lot of work that is being expected to be done in the future and now it means you need people to do that there are not so many of them there are aware of that so they ask for a lot of money they change job often and that is very very painful because you know managing code is quite easy you have to get up for that whatever you do in code it starts somewhere it's possible to read but with the data is different whatever you run on the data is not necessarily stored so you don't know what is the history of the exploration of the data how did the guy discover that this feature is interesting you don't know that and when it goes away because with that and so there's really a program of retaining the knowledge on the data there is no brain of these guys the consulate and move around and so we need to find a way to share that work to be restated or enterprise okay so let's first focus on more technical aspects that really relate to to scale a so how do we do that work with scanner what is the way to get scalability and do we have the right machine learning methods to to do that so first a few words about wise guy is a choice for us and that leads to the tvm the functional prodigal the simplicity of the language and actually its interactivity so first the JVM it's very obvious that if you work in an enterprise world you need a setup that is enterprise ready and with more than 20 years of kind of approval from the industry of this setup it's obviously a very it's the platform of choice for your data science to run do you really want to introduce a new technology not that often in big big industries split and especially if it's you know for running the lower level that is almost the runtime level of your applications and because of this approval and the number of developers applications and libraries that are being developed into the GTM there's actually a lot of things to leverage and a lot of people do to add so the JVM is certainly the excellent choice for running your data science in the enterprise then it's the functional pardon today data is distributed more than ever gone is the day where you take the data of a sample of it you model it on your laptop at home and then come back with your report claiming I get the clue to get money out of our data because you cannot get the entire data like on your laptop data is distributed you have to work with it there that means any processing of the data you you conceptualize as to be shipped on the data and more on each chunk of data distributed in your infrastructure so that means we are to think in terms of functions on the data and not the data being processed close to you and of course you want to express that code in a way that is completely independent but that execution model or orchestration that as a data scientist you don't want to hear or know anything about right while in my opinion is really good that you know that was happening in two partitions maybe you don't need to know the really intricate details I lose time on that but the code shouldn't reflect what happens in those chunks of data if was the simplicity of course the syntax shouldn't be a barrier and when you look at skala syntax well I really like the skylights intact because it's reclose to those scripting languages and python for example it's as close to top item is as close to Java than Skala is close to to Java they're just on the right side of the of the edge i would say and python and skaara are really close to each other if you have a data scientist doing Python and is you know completely against Java or the JVM you show him skala code and is really fast into it so that's a good way to create into this and sky is very powerful levels as a language as we know there are so many crazy features and and it's easy to work with interactively so now i will i will get through some notebooks so most of the presentation and the thing that we show are actually available on on github so there is this repository where we have a couple of notebooks that you can play with at home you can make put request add some stuff it's very open and we we are we will be grateful some some contributions so i will get to the demo and have some help for a microphone so who is familiar with the use of notebooks yeah quite a few the spark notebook specifically a little less ok so the spark notebook actually was developed first by Andy who using spark got a bit frustrated you know using the wrapper for several reasons including one that is pretty obvious right you are working interactively with your data when you're in the command line but how do you get access to the history of your work what has been in the end really executed to get where you are that is something that is frustrating so getting in your end something that is solid and that you can save for reuse is not easy also it's always good to annotate your work when you're doing data sense you're being your are very into exploration is good to be able to annotate your work and working in a in a web page like like in a notebook is really the way to go and that's also a reason why this ipython and Jupiter notebooks are also popular so this version of the of the notebook actually runs with a scatter rapper behind the scene instead of for example a Python Ripper with a spark context that is initialized for you so you have direct access to to spark from your from your browser so it works with a server and in from that server you have a browser like a Phi browser like you have here with different notebooks each notebook containing code organized itself that you can ask to be executed in the in the wrapper running behind the scene so we start with the while why scallop notebook so you see structured here as cells market or code I will get over this scale a notebook so the audience is probably very familiar with skyler concept so i don't need to spend 20 minutes on that just getting Trudy's in a few minutes showing the D features that that are really the important ones in skala and all day they relate to data science work and other platforms of interest for data scientists and here I'm naming of course Python and are so a lot about these these notebooks the first version of these notebooks were actually developed by Dean one pro who most of you must know from these books on on Scala and an Andy and I contributed lately as well so let's get started so why Skala for data sense and here I'm really pointing directly to spark and the reason i pointing to spark is because this API as an API for processing data is really good it's really matching the collection API which itself is you know taking its concept from from the functional / adding on on collection on collections and so it's really to to work with and the execution model management of data in memory is really efficient as well so it's one of our favorite choice for processing data for pressing distributed data and when we think of spark spark is is actually written in Scala and it has interfaces in our and Python so why would you choose skaara over this interface that's a first of these reason to that is that if you're closer to the language if you want to implement something new a new model it always has to be implemented first in skala because then you need an interface from the other language on that new method so you just closer to the system and also you don't usually in the Indian surprised if you want to be efficient with going into production you want to avoid mixing up too many languages and you have less feature coming our features in spa in Python or are are coming later always and and you have less features so skala gives you much more power and it allows to be closer to the the production environment and the key here is to say yes skala is valid for the data scientists as well maybe they have a little learning curve to take but they will just get better and they want a lot of money yeah they'll be on top of the of the stack and that cannot be bad for them right okay so first functional programming and objects if you look at functional programming it's really about you know immutability of values functions with no side effect why do you want that because you're working with data that is distributed you don't want States to be managed you know you have to process things all over the place you don't want to have a state that is you know a nightmare to deal with you want to think in terms of small things that are well contained in a context these functions oh I give you parameters you give me a value that's it okay and and I are order functions will see the collection a pilot that you probably know it's really good to say I create a function and I send it to the data and that's it it's really a way to think that is really close to the way it should be executed so you extra your code in a way that is very very close to to what you want actually I I took I took in skala after having worked several years in Java deciding that the JVM was kind of game over for me went into into trying scripting language like Ruby then when I discovered Skala I could reimplement many things in a pretty efficient way and with a way of thinking that is much more efficient I think so other languages Python has a mixture of functional programming and object-oriented the reason you want object-oriented is because you may have larger projects manipulating complex things with different teams it's good to have libraries and you know domain objects to manipulate your encapsulation of the behavior of your of your object so it's important it's not only functional and so Python supports both but not as efficiently as Skala are is really much more functional than object oriented and Java has started some support for lambda so like a functional style but it remains a bit cumbersome in some some aspects another thing with Calla versus java that was mentioned by by some some people is of course the inheritance model using trades so mixing behaviors into into a new class is really nice in skala you know having this this these fields that are initialized in trades implementations of functions of methods in trades is is very good and when you compare to to java and the collection api you have always disk inversions between the notion of a stream and the collection it can be really cumbersome to go from one to the other bit just because you want to use a functional parody okay so this is a first little example it's a bit silly it's just a method to compute whether a number is a prime number so here is the first execution of some code in the in the spark notebook i define a function and I hit ctrl enter and now in the repair behind the scene it has been executed and I have access to this dysfunction of an integer returning a boolean and you know I do simple things with a collection so I take the first hundred integer and I applied this function on each ayman so creating a tupperware I have the number and its finality then I will group by the primary key it makes you know an iterator containing all the numbers that are prime numbers and another one with the numbers that are not prime and then i will count actually the number of such elements being prime or not try and you know it's a simple process but you know it shows how it's simple to express things in a pretty interactive way and when you develop on data working with a notebook like that with the rapper you take your data you make a transformation you get one step you make another transformation next step and you just think in terms of small functions to apply to your data to get exactly transformations you want and in the end you get something that is maybe not really a trivial but that was quite easy to develop so here you see we have one quarter and not one third of the numbers between 0 and 1 data prime numbers if you want to do that in spark we need to work with the spark context is the gateway to your cluster so you can define operations on on your cluster and what I say here is that I want to take those 100 numbers I want to paralyze them so ship them in the cluster distribute them there and then the operation at are just the same so it's exactly the same API even though we're working with a different concept one is you know a collection that is local on my computer the other is something that is distributed and it's a lazy a lazy collection that means that up to that stage nothing is executed I'm just defining its some transformation on the data so I'm just saying on these data you apply this function that dysfunction that dysfunction they wrapped together in an object that is shipped on the different nodes of your cluster and they are only executed when I asked for results so here when I do collect I say okay the result you give me back here on the driver and I want to see it as a local array of course you have the same result but you see that it's pretty easy to get into distributed computing there is nothing in that code that points to something specific about the cluster about the nodes about the way data is split in chunks it's just the collection API and that's the power of course a functional programming so what about the other languages well the Python API is very very very close to the scallop API you write Python code in spark it will looks like skala cot r is a bit more tricky and and cumbersome and Java looks similar when you use lambdas but some features are are missing and it's not that concise as in skeletal remains on the top so the wrapper java we don't need to talk about the wrapper it's not yet there someday it will come are as a nice ripple and Python have a nice trip also there are you know good competition i would say in terms of of interface the topper syntax it's something that is very valuable when dealing with data because usually you have a set of free cards and these records you know they're just columns so as a number of things you want to manipulate with the type and one color may be integral and other floats and other string whatever so it's good to be able to manipulate a bag of objects of different types that's exactly what tuples are and so yeah that code we've seen before we use the toppers always it's really common use in big data to work with key value pairs do we have these features in other language yes we have them in Python although of course dynamically typed and not static types are has to / but it's really much less convenient syntax to work with them and Java we don't really need to talk about that it's it's really hard to work with the notion of tuples of mixing in different types you know without creating a class is really hard okay now pattern matching of course so in that case we rewrite the same execution but we leverage the fact that we know the structure of the results to decompose it instantiates and variable and extract the information of interest that is something that you don't find in other languages easily and it's actually very powerful as a developer to be able to you know whatever the depth of the nested structure you just get into your object take the element you want and reconstruct a new a new structure just in in a single small simple wild one liner that is actually very easy to read that is incredibly powerful of course and then when when we think of tuples and and pattern matching you take the two and you you have case classes that are another way to group different different types together within a stroke unnamed the structure and you can pattern match on case class of course so this is an example where we actually take I didn't execute that one so I'm defining hear a case class so I want to group together a name last name and an edge in the person a class then i instantiate some of these objects and i can pattern match so instantiate a variable here pointing to to this person and get access to the fields and then do some transformation here so here I take Dean and Andy and I add one to their age ok so do we have these regular of course regular expression as i said we we don't have we don't have in other languages and case classes so being able to look into the structure of a class we don't have in in other languages only skala provides it then there is type inference and i can tell you that developing a data processing work flow so transformations on data having type inference is incredibly powerful because you don't have to know in advance what type of result you will have when you're working with one stage of your transformation but the system will tell you what you got at this stage so if you would develop that with python you would have at each stage to ask for output for for example the first five elements look into the values to understand what you're dealing with we scale i if the system is give you simply back here in the end i have a tuple with boolean and in integral so it tells you what you have in your hands and maybe you don't even have to think about whether well you know whether your transformation is right on it because you have the right type and you know where you are always so that is something that is very powerful of course you never have that feedback in a language like Python and our and Java is even more frustrating because yes it is technically tight but you have to write explicitly the type of everything so and because you have no rapper you never know what you have in your hands you have to to figure out on paper i would say okay and then you have other things that are very powerful with scala you can do you know list of integer and the system for you will use the underlying primitive integer type so it's optimizations that are very very valuable to do to get the work done and then you know you have this capacity to create dsl's like you have here with the the in spark the data frame API so here I'm reading some data so it's a list of airports I read this the schema of the data is inferred automatically from the front data itself so I've the name of the airport after the city and I have more fields you see that in a minute that I can ask to cash it so it's a sparking to have things stored in memory so that I don't have to read the data on disk every time it's just there in memory so it's very optimal and I show the data I should have the data available you know not sure why it doesn't display anyway and then you can use this DSR so dsl for doing fantasy fan settings like I don't see data why okay for some reason there is no output showing but what's important is the code here so so I have a list of airports and I can do really just like sequel I mean oh I want to group by state and country and count the number of airport I've in each state and country and order by the count and and get these data and so you have this this dsl here that allows two point two columns when you're grouping making your account and and making the order descending pointing again to the common and that is the kind of powerful things you get because it's its scale okay and with dynamically typed language you I've actually the capacity to to create some some dsl's and actually the the spark data frame API is directly inspired from the pandas API which is a Python API on on data so you know all these experience of Python data processing as as its value that's for sure and it's good to take those features into into scanner it's possible so such a good thing there are fewer things that I invite you to to get through let's go into some some of the negative aspects of scalar quickly one of them is the fact that if you're working with our or Python there is one good reason usually is the tools you have in your hands the number of libraries is just enormous so there's a catch up to do on that the JVM also has some issues that might appear in the other language as well but you know for example arrays are indexed with integer so we're limited to two billion and in a world where you can have terabytes of ram it can be a bit frustrating and there are efficiencies in the JVM memory model we all know that it can be a painful to get garbage collection don't properly fast when you have many records but there are projects liked understand that are trying to overcome this problem by representing data internally memory in a much more efficient way and also when you work with the scatter rapper you can have some really weird things like memory leaks because you're it's a rapper but each time you eat you know execution of a line or of a cell you actually have to compile what has been executed in the two new objects and get it down that means that previous lines where you may have instantiated variable with a lot of May more data in if you re instantiate those variable with the same name actually you lose the access to that data but it's still there so they are sometimes some some bad things to do to overcome so that is for the language scalar itself no yeah the next thing that is that I want to it's going to be fast the next thing that I want to mention is that we scatter and JVM I mentioned that already there is this this tooling do we have the right tools to work efficiently with data science where the answer is is not that good so for example plotting is one of the great strengths of our or Python you have this JJ plot API so you have a data set you can plot it whatever you want in an easy way and get really nice floods easily if you want to do that in skala you kind of in a trouble and also because so many things were implemented in Python and are it can be frustrating to be in skala or even to be on distributed computing because all of this work has to be ported to scala and distributed technologies and because we actually also have access to so much computing power it's also time to you know implement new mothers for four scalar and and distributed computing and now we're going to to the spark notebook to look at you know what are the features of interesting to that that platform so why spark notebook so the spark notebook is open source and it's refocusing on on productivity in enterprise environment it has many many stars 1000 and 500 to date or more it has quite an interesting number of participants on a guitar channel so you will always have support on it we produced many many log books notebooks of example so you can get started and see many things so the interesting point here is that you're in a web in a web page and you can interact with that page the way you want so here it's it's a silly example but you see I'm just executing a cell in which I just say okay I want this HTML chunk to be incorporated in the page and I changed something I change the style of the page here so so you can create really nice visual if you want second thing is multiple spark context that's one of the specifics of the spark notebook so you have a server but each individual notebook you instantiate will have its own spark context it's it's own new JVM that means you can work with your own set of dependencies in a notebook another set of dependencies in another notebook you never f clashes you can have several people working on the same server they will never interfere with each other they can connect different clusters one working local one working in the cluster so that's really efficient and to show this one minute left okay so this this shows you that you actually have several notebooks or running and there are each independent so we'll get through this feature at one minute that's a that's an so configuration that you can do interactively as mentioned you can define dependencies that means you can import anything get it from maven repository and get it running on your notebook one other interesting feature is side pane here so any variable you instantiate is referenced here and you can access to this its definition so we're really trying to get something close to the our interface that is our studio so having code what is executed and a view of your data okay and then the important thing which as I said is really plotting and here i will just run an example so i have a case class and I generate some some data here so I have a list of data and because it's kayla and it's evaluated the system knows the type and can offer you a different kind of representation of that data because if i have a list of integer i know i can make a proud of it if i have a list of strings then i can show you a list right and so here the options because i have a list of case class that contains a number of elements i can have a pilot or a list ok so that is how it works and when you go further along you have graphs that are implemented and all these features just to make sure I anguish that all these features actually are implemented in skala under the hood you have of course some JavaScript running and some some HTML but the plotting API is just Scala and you can actually extend these and create a new type of graph of plotting just by implementing the JavaScript and the sky right guy so I have no time I could speak during two hours just to to finish [Music] there are some many libraries that are being developed today to do data science that is machine running it can be Emily you have h2o you have deeper link for J all these are compatible with spark with scallop interface API so you're very powerful and you're in your language and you can do many many things so one last thing about going to prediction and retaining and sharing work against kayla is a leverage there because when you have a notebook written in skala you can know from the compiler what data is being read what data is being saved what is the schema how it is used and if you collect that information you know exactly what you want to know who what is being done on data and you can create a knowledge graph of all these information that's exactly what we do at consumer thank you for your attention and your your passions I would like to mention and D&D in once again who made a good part of this of this world you