Cognifest NYC 2017: Jerome Nilmeier, Getting Started With Data Science on Spark with Scala Pt. 2
I just wanted to show you the Scala 99 problems these are pretty cool so these are the kinds of things that they'll ask you to do in Scala 99 Oh you know it's a fight fine the last element of a list fine find the K element of a list pretty basic stuff but they get they get pretty hard pretty quick run length encoding duplicating elements and I think down here determine whether a number is prime so it's got its got some recursion challenges so it's pretty you know I haven't gotten through all of them that's for sure but you know you get through the first the first 20 or so you get a really good feel for just the basics I mean most of the time when you're writing spark code you're not really doing anything super fancy at least at first a lot of times you're just calling libraries and things like that but they're but there's this there's this the ETL phases when you're you know you're extracting data from if you're extracting data and you're trying to process it into a you know a data frame or something like that these kinds of tricks will come in handy that's for sure so so it's really great if you if you have some time it they're not too bad and they have they have solutions so you'll be able to take a look at the solutions notice that the solutions are in singleton objects so you don't necessarily have to put them in an object you can just you know define them and run them as a script from the shell so you can get surprisingly far in Scala without having to compile and I'll even show you some tricks in spark where you know you can define functions in a separate script and then load those functions as part of your initialization and that'll that's sort of instead of instead or you can compile up in an important have access to the methods that way but but it's nice because you can you know if you're if you're a customer using an IDE you can you can get on the you can get on your cluster and just kind of do these ad-hoc edits on the cluster rather than going through the whole dev cycle of compiling and deploying and all of all of that so it can make for a fast death cycle in certain cases but let's see so we'll go into components of components of components of spark will so so the the the spark application has basically three three components one is a data storage layer the other one is an API which is the interface which is typically Scala or I'd like to encourage you to use Scala but we've got Java in Python and also our and and then we have something called a cluster management framework so now we're all running spark on our laptops today but this is really designed to run on a cluster so everything that we're doing is just kind of you know a sandbox for for the time when we actually you know get our code working on a on a cluster so the the frameworks and we'll talk about are there three frameworks one is called standalone mesos and yarn and I think kubernetes may also become a cluster management framework but I'm not too sure about that he'll hear more about that shortly but Stan these are the these are the historically these are the ones the three main ones so me so this is not used as much so look yarn is by far the most used if you're on a multi-tenant cluster where you have lots of different applications running on the same cluster because yarn is like the default resource manager for most hadoop clusters so yarn stands for yet another resource negotiator and it basically just manages all of the jobs that are running on the cluster the other one that you'll see a lot is if you just have your own let's say you have a data Lake type model where you've got a bunch of unstructured data sitting on a Hadoop cluster in you and a small team of data scientists have exclusive access to that cluster and that and you're not running anything else but SPARC you can use a standalone mode which was would just spark well create its own cluster management framework so so it's a lot of times it's a lot easier to install because and you don't have to install yarn so standalone does not mean it's running on a single node stand-alone means it's it's a it's a standalone framework where it's SPARC is the only thing that's running on the cluster but it's still running on a cluster so the nomenclature always was a little confusing for me but but those are the three cluster management frameworks but the one you'll see a lot is yarn for sure so this is a little bit of what the SPARC ecosystem is it's got it's got spark core which is really all of the cluster management machinery built into it and then it's got some other libraries like spark SQL spark streaming machine learning library graphics and then there are other connectors here this is just one instance of a connector but there are lots and lots of database connectors and also things like spark our integration so so all of this sits on top of something called spark core which manages all of the all the parallelism so it's it's the base engine for all the the parallel computing it does all the task scheduling it does all the memory management it handles all of the fault recovery so we mentioned briefly about when what an RDD is is when you when you when you load the data and you you put a you put it through a series of operations and that node that you were doing the calculations on crashes how does it rebuild that to not only to the place where it loaded but to the state that it was at when it when it crashed and that's all done by by spark core it interacts with the file systems so all of the all of the sort of low-level stuff that that you would expect kind of a framework to do it also holds this API called the RDD which we will talk talk about so so these are the three cluster manager managers as I mentioned yarn is the is the one that that you see the most it was actually originally written for for me sews and but in general at least for our use cases we don't see it used as much but but I believe it's still used in in practice so this is kind of what a SPARC program looks like it's got a it's got a driver program that's where you were you where you open up your spark if you were to open spark shell you would be sitting on the driver program if you're running a spark if you're just running an application you would also be running it from the driver program that has this creature called the spark context which is the window to the entire machinery so all things spark happened through the spark context spark context talks to a cluster manager and then the cluster manager sends out job requests to worker nodes so these worker nodes have this notion of an executor and so an executor is this persistent container that sits on a worker node so you can have a worker node that it can have multiple executor zhh and all of the you know all of those executives can receive a particular job so it's a persistent container that gets job requests from the resource negotiator which is yarn usually or standalone so you can have all of these distributed throughout the help the codes now the executor within the executor it's processing a series of tasks those tasks have access to a cache and they have access to shared memory they also talk to each other during the during the process as well now if it needs to return results it doesn't it doesn't go through the cluster management or just since the results directly back to directly back to the spark context so so this is the you know I would say the basic the basic idea behind it one other thing that's kind of neat about the executives is once you create the executor it persists for the lifetime of the job in general because in previous cases they had this thing called a mapper which would only exist for the duration of a particular calculation and then it would be and then so you have all these containers that had to be spun up and spun down in this case the containers persist throughout if it's not dynamically allocated it persists throughout the lifetime of the of the job and they can receive job requests and so it can have this is the this is that note it can have some persistence and make things a little bit more efficient you can also store things in memory throughout the lifetime of the job and all of that so that was a big that was a big change in in the way MapReduce operations were done versus spark jobs I think we've covered a fair amount of these now so application is the program that you're writing the driver program is is the process that's running it your cluster management you've got your three flavors the worker node houses the executor --zz the executor czar receiving the jobs tasks are units of work that land in the executor x' jobs and then a job is something that has got a series of tasks and a stage is also I can't remember which one is which I think yeah stop give it that's right so a stage has got a sequential nature to it so the SPARC context is again the main the main creature that that sort of lives on the on the driver and directs all of the the jobs it you know you use it to create all of the data types that that that there you're going to be doing SPARC operations on and for SPARC shell the SPARC Texas automatically initialized so usually when you spin up spark shell the first thing you do is type SC that's a short for spark context just to make sure it if it gets in if that is able if that's initialized you'll get a value a return for that if it's not then maybe something didn't can didn't get initialized properly so if you're having trouble getting the spark context let me know and Lowell will troubleshoot it most of time it's you have to do to be delete Derby log and meta store DB directory for our examples at least so we spent a lot of time in the repple from the Scala session so the fact that spark shell has this repple is is we've already sort of covered that PI spark also will open up a Python shell or a Python repple and you can do the same thing so if you wonder we've been we've been starting with spark shell if you want to start a Python instance you just do the same thing spark home slash bin PI spark and then that'll open up a pipe ice bark shell and it will pick up the version of Python that is natively installed on your machine so if you have installed Python 3 some of these examples won't work exactly because Python Python 3 and Python 2 are a little different and so just to be just to be aware of that one but also the spark context should be available when you when you bring that when you bring the PI spark shell up so you just type SC and it'll it'll return a little object pointer so just want to talk a little bit about the basic abstraction in spark which is the resilient distributed data set and we've been we've been covering this idea a little bit throughout the talk but what it is really is it's a fault tolerant collection so a collection is a big list of items an RDD is a parallel parallelized form of that so you just you know if you you have a terabyte terabytes worth of numbers you're gonna do something called SC parallel eyes you can think of it that way create a very large list maybe an iterator or something like that so you don't use up too much memory when you do it and then you say SC paralyze that list and then it'll take that list and it'll put it on the put the cluster it can be operated on in parallel so that means you can have each node work on a piece of it it'll divide it up into partitions and those partitions can you two be operated on by by the executor and it's immutable once you construct it so in scala things are already immutable so you're already kind of used to the idea of immutability but in an RDD is is is that you don't have the option of defining it as a variable it's always immutable and it has to be because now you're always operating in parallel so the three ways you can create an RDD is as i said you take an existing collection and then just say I see parallel eyes and then that will create a a parallel collection and or you can load a file that's sitting on Hadoop it's a parallel file already so you just load it and it goes directly into to an R DB or you can transform an existing r DD so we'll talk a little bit about transformations in in a moment but a transformation on an RDD always returns an RDD so one example of a transformation is a map so you add one to every element of that of that RDD you say map X forward arrow equals x plus one and that will that will create another RDD with not the values but the instructions to carry out that operation and we'll talk a little bit about why that's important but but the main thing to to remember is that is that when you do these kinds of operations on RDD it'll create another RDD so it's just like parallel abstraction and these are all the files that are supported currently and there are many other ones or c is one that gets used a lot too but there are lots and lots of file formats for these so I need to grab a glass of water again with it sorry excuse me okay excuse me that so so the idea of a transformation is really is is is not to actually carry out the calculation but to to issue an instruction to carry out that calculation so you say I've got a very large list of numbers I want to add one to each of those lists each each element of that list but then I want to filter that list based on some criterion so let's say I want to add one to each one but then I want to take the ones that are only greater than 10 or something like that that you can imagine that there is some clever way of understanding the best way to carry out that operation if you know both are happening right you know that you're adding one and then you're filtering because you don't need to add one to the ones that are going to be filtered right so there's this notion of lazy evaluation that's built into transformations this is why when you do a transformation you return another RDD because it's this it's as parallel it's this parallel abstraction that you're not necessarily carrying actions out on so you can instantly define you know your your RTD and then add one to it and then do some other operations you get all of those happen very very fast and then you do something like an action which hit which really tells it oh now I got to do the calculation right now I okay now count count the elements in the array how many how many elements actually met the filter criteria and that will initiate the actual calculation that will actually send me the job requests out and it'll start to carry out the operations and those operations will be a whole series of transformations that were applied to to the RTD so so you think your code is running really fast and then you hit an action and then you're like oh okay now I you know so so so when you're when you're coding this stuff always keep keep in mind so in action the way to remember it too is an action always returns a local object because then what it's doing is its carrying out some kind of calculation and returning returning the object to the driver so that you can actually you know see what the value is and things like that and so this whole RDD really is has this whole series of operations and it's called the dag when you see the debug logs and everything like that it'll have a little it'll it'll it'll you'll see dag show up a lot in the debug and so so this dag is really the key to the resilience right so what happens is you know if you lose your node it actually it still has that dag which said oh I was supposed to load it then add one to it then filter then do this and then it and then count and all that and and that's how I can have this fault tolerance because it has a whole series of instructions stored already so it can it can recover in that way so that's that's the RDD that's the resilient distributed data set so that's the that is one of the really fundamental ideas in in spark and so here's some of the basic transformations map filter this thing called flat map which is like takes an array of arrays and treats it as a single long list so for example if you're reading a line of a line of words the first line has a list of words the second line has another list of words so that's a list of lists but you actually want to just treat it like one big list so that's what you use flat map for actions so reduce is an action because it returns it takes very large list numbers and returns returns a sum right if you're if you're adding numbers take will take the first few elements of the list collect is the one that you have to be careful with you can it works great when you know you're developing on laptop scale but then if you do collect on a few terabytes of data sitting out on your cluster then all that data goes to your driver node and will crash the driver node so it's so so take is usually what you use to sort of a debug so you can just say take a peak at the first few elements of the array then take order there's some other ones that you know various flavors of of take so we'll go into the first demo here yeah I'll see if we get yeah I spark working okay yeah I just wanted to well we'll do the first one in in PI spark I think this is my one and only Python demo part part part of the motivation for putting the Python demo is to show how very similar Python and Scala are so they're pretty it pretty much the same things with just a few minor changes and so-so and in fact what's really nice about the docs if you get a chance to see what's really nice about the docs is it'll have tabs of a Scala a Python and a Java version of the same code so I verified that my spark context was created what this is saying here is that the spark context is the master is local which means it's not running in cluster mode it's running in local mode and the star means it's running with all the threads that it can get so however many threads are on max of 16 threads or whatever and and then it'll assign a PI spark shell name to that to the app and the other thing is kind of han't handy how many people have Python 3 on their machine Python 3 Python 3 has Auto tab or tab complete built into the shell Python 2 doesn't so you know you always do this red line RL completer trick so you can see what all the methods are it's pretty handy because when you're creating these rdd's sometimes you don't know if there's a lot of there's a lot of methods to keep track of so it's nice to have them available to you so and then I don't remember all this stuff in in Python 3 there's something like this to get to to run so what we did there is we created data which is a and I think you can actually yeah you can so you can actually get the type of the variable in Python by doing type which can be pretty handy because the thing the thing about spark is that you have an array which is a local object or any collection which is a local object and then you have something that's pretty much identical to it that it's a parallel object which is this RDD so knowing knowing whether it's an RDD or an array is can be a challenge sometimes if you're if you're if you're not keeping track of the types and especially in Python because it's not statically typed so so all's we did here is we created this RDD and notice when you're in Python you all I've seen this in other cases too you're almost doing the typing in your variable naming right now we're saying we're saying oh here's data and then one and then but but we're gonna create an RDD so maybe i'll put our DD in name because I don't I might lose track of the type so you're actually kind of typing it while you're while you're while you're writing and so now you've got x-range RDD and and i think you can yeah yeah so now it says OS is a this is this are DD type here if you wanted to figure out what the type was now the only way you can get to that variable to the contents of that is either by doing is by doing you can't you can't see what's happening in it as a our DD because it's this cluster abstraction so you have to do collect or count or take or one of those actions to actually get to the contents of it and so that's that's what that's what was sitting on the cluster that that list of numbers then you can do a you can do a lambda expression just like you could you can in scala except for it has a slightly different syntax this is lambda X calling X x equals X minus 1 so and then it's got filter and and all the the same type of lambda operations that you would see in Scala so the and so this is what it looks like in Scala right and now that sometimes when I'm doing the spark class I don't always do the Scala primer at the beginning so but but now we're all Scala pros so we can see we were we can see what you know we can see that this is all this all looks pretty much like the collection operations right I mean there's not much difference but the thing to remember is that this is these are these are operations on rdd's so these are SPARC these these these are collections that are created by the SPARC framework that do pretty much the same thing but if you look so filtered RDD and now let's see so these are the things that you can do with test which is list here right and these are the things that you can do with our DD our filtered RTE was what I collected right so it's got all this other stuff all this all this spark stuff first distained count collect and some other some other some other stuff so so yeah definitely definitely the the autocomplete will will help help you get a feel for all this stuff how we doing for time 350 okay so Friday first demo and Scala okay so let's talk a little bit about what we just did with with those rdd's when we create when we when we when we said hey this is a parallel object it broke up that data set into partitions and you can assign the number of partitions or you let you let's Park figure out what the best way to partition it is but then it'll take those you know and then then each each executor gets to get to gets to work on a piece of that then what we did is we did a map operation we said subtract one and so each executor crunched through that you know in parallel and stored the result on the executor so so this is all happening pretty much at the same time there's a little bit of a synchrony but but but they're all completely independent operation so so I can it's not a concern so now then the filter operation just says oh just bring me the ones that that that that satisfy that result there are other things that you can do collect wood wood and we mentioned this before I would take all the results and send it back to the driver and of course that can possibly cause cause issues and this one is a count and so this starts to speak to how spark handles these counting operations reduce operations there's other clever ways all this stuff is happening so this is what I had mentioned about these sort of the notion of an iterable you say you just you send it a method and you let it figure out how to how to how the best how to do it in a way that's most efficient and and so what it does is it just goes to each executor says oh add those numbers up and then store those results and then and then I'll just take those sums of those and it does that for a lot of these types of operations where it's sort of does these node local operations and then sends the summarized result as needed so there are there's a notion of persistence and I think I might actually show you that an example of that one so what you can do is you can persist your data on the cluster mostly in memory so you've got your you've got your data you load it and you you want to access that same data set multiple times very very rapidly and so what you do is you say ok store that in memory and then there's other options here if it spills over it can go into disk you can go in a serialized form or deserialized form and various things and so what this a lot of times you don't necessarily mean it may come up but I mean let's say you're just calling a machine learning algorithm all the caching and stuff like that is handled already by the machine learning algorithm if you're doing a data frames operation a lot of the caching is already handled but let's say you're cooking up your own RDD operation that that really you have to write at a low level then you may be doing memory management on it the other thing to note is that there really is a lot of time you know the ratio between memory on your cluster and memory on disk is usually pretty small so if you were if you're going into an application thinking that you're going to get leverage from me let's say you have a data set that you're querying all day long and you want to just cache it and just have rapid access to it you need to have enough memory on your cluster to store it in memory otherwise you're swapping out memory and spilling the disk and things like that and you're losing all of the optimizations that you you think you may be getting so make sure you spec out your cluster so that your data set will fit in memory if that is your intent so and there's there's a quick way I can show you a quick way to do that so let's simple cash and and then I'll show you a little bit of the I'll show you a little bit of the spark UI - and I will do spark show so I'm not going to get that much of a bump by by caching the data for this example we'll get a little bit of one but-but-but-but not really but you'll see a little bit of it in action so what are we doing here we're we're create we're parallelizing this this range into 50 partitions and then we're caching it now the thing to remember is that caching is a is not an action it's a transformation so you have to actually the XF have to actually carry out an action to initialize that whole load process so load is even not an action load as a transformation so when you load the data or create the RDD it still it still hasn't done the calculation that is associated with that so you have to initially you have to do something like count in order to get it to go through that whole series of operations so when you do count then it will cache the data and and then the second time you can't count it it should count faster after it's already been put on the cluster right so let's see if that if that works out so again to to to run scripts I don't know if we covered this in the Scala examples cuz we were doing cut and paste you do : load and then simple cache okay so the first time the first time we counted it it took point eight eight seconds the second time it took one-fourth the time and so that can be up to 10x that can be up to a 10x difference maybe even larger in some cases depending on how you design your system but you got it you have to design make sure you have you know thoughtful design in in place when you're doing it but let me just open up the spark UI cuz it's fun to look at and so let's see I think it's here and then so it's localhost 4040 maybe let's see maybe it's forty forty-one 4043 so the default part for the spark UI is 4040 if you got a spark instance running it'll look for 40 41 and they just keep keep going till you get so I've got a bunch of I've got a bunch of latent so here's so first let's look at the job there was a first count the first one took point six seconds and then it says that the second one took 45 milliseconds so I'm not sure exactly we're the extra latency in our timing comes from but but but even here you can see that there's a difference I mean this is significant difference in in the count time the other thing is you can take a look at the stages each count was the stage the tasks we're broken up into 50 tasks because there were 50 partitions we cashed it so now it'll tell you this is the part where you can really trouble make sure your cluster is up to the job of the caching that you're that you need to do is it's got a storage thing and it says oh I've got 50 partitions it's 100 percent cached that way that and that's that's what you want and this will tell you how much size is in memory then you can start to to to to size out your cluster this is your environment this tells you all of the settings for example the spark master was local star and then it has you know the Java a bunch of other stuff that that will definitely come in handy for you the scheduler mode for example the executor is here in this case we only have one executor which happens to also be the driver so that's the whole because we're running on our laptop but here this will list out all of those executives that I was telling about the worker nodes and I'll tell you the IP address they'll tell you you know how they're performing all the tasks that they're carrying out and things like that and then if you I think I actually do have a data frames example so we can show you how the SQL is working so so that's just a quick quick overview of the spark UI okay so we've got we're looking at about 40 minutes to cover some of the libraries in general I don't get through all of them so you guys can kind of you know the ones I really like to cover are our data frames and ml Lib and and then after that there's graphics spark streaming and system ml are from L and some of the other ones so as I start to get closer to the end maybe we can discuss about which ones you guys want to hear hear about the most you all of these work pretty much out of the box so if we don't get through all of them you just you just do that load file command and and just run through the examples and everything everything should work as expected if they don't please feel free to give me a call so talk a little bit about spark SQL this is this is the module that takes your RDD and turns it into something called the data frame that data frame looks like a table in SQL so then you can do you can run SQL commands on those tables you can join those tables with other tables do all the things that you can do with database a regular database it is almost completely ANSI compliant which which means you can do you know nested joins and windowing functions and all the fancy stuff but it you know you can also just do your basic SQL workloads which is sort of what I prefer just to make it legible but but but it's it's a pretty robust sequel engine at this point so and let's see wait I'll make sure I covered that yeah so so yeah the main the main abstraction here is this thing called a data frame which is just a generalized version of an RDD it's an RDD with some additional metadata that allows you to treat it like a table and it's a collection of rows and those rows each have you know column elements and things like that so we'll show you what it looks like in practice that it'll be a little bit more helpful okay so we're here okay so we're gonna we're gonna load this very large data file with three records in it and we're gonna do some SQL on that so I'll just go ahead and say let's cut and paste it why not what's so the first thing it's doing is it's got this it's got this JSON reader so it says spark read JSON which just allows it to read JSON form again pretty much all file formats are supported I'm not sure if CSV is it didn't use to be there you had to load a library for CSV readers but for the most part you know most formats are supported you know when you're working with larger data sets it's preferred to use the compressed formats but for this debug we can do the JSON format and so now you've got a data frame you know you can do things that you're sort of used to doing with data you can show and print print schema and there's a bunch of other standard operations and then the trick is to do this thing where you create this temp you just so how many people are SQL like writing in SQL prefer writing in SQL okay so III do the first certain workloads it's really great it's you can you can you can really do a lot with it and the so so you create this Tim view and then you name your table and now if you just want to run so there's a few ways to do it this is sort of the more you know the the more method based way of doing it you're just saying DF dot select you know it's got a DF group by it's got all these kind of SQL like operations that you can call as a method but then you can just write SQL too so once you've assigned your table here this people table then you just say select name from people and then it will and then I will do your SQL operations so let's take a look at what that this one does so it created a it created it let's so oh oh I got to do this two dots show okay and so so just what it does is it I defined it here just so you can see that it creates another data frame from this operation so this operation resulted in creating a new a new data frame object so you these the output of a query is is another data frame another data frame that Mayor hat may have different columns may have different data types you know it's not completely strongly typed but it it has it has the data frame type so you can do other things you can do filter there's a few ways I mean if you're like an AR program or this dollar sign notation I think is you see this a lot so it supports a few different ways of doing SQL I still prefer the actual SQL seems to be a little more readable but but you can do you know counts and group buys and that this can be this can be very handy for for lots of workloads it's also you do see a lot of cases where you actually have a workload that's already written in SQL using some other database and you can just translate those SQL statements directly into spark so it does make some migration issues very very easy but it's great and the it if you can do stuff in data frames it's really strongly urged that you do because it leverages something called the catalyst optimization engine so there's all kinds of these optimizations that are built into these these SQL queries because so much is known about how to optimize SQL queries and a lot of that expertise is built-in to this it's also if you're writing in Python it's also much much faster to use the data frames operations because what it does is it compiles this instruction set into Java byte code that's identical to the Java byte code that would result from the Scala code so it is the one library where you get identical virtually identical performance in both api's so so so it's it's it's definitely worthwhile if if you're committed to PI SPARC workflow to try to try to write it in data frames if you can so okay so any questions on data frames okay the ML Lib is is really the whole reason spark came into being it was a machine learning library and the goal was to make make machine Larry and in performant frankly there were at the time it was really difficult to get machine learning to perform well at scale and so it has lots of optimization primitives and higher-level api's it's got two packages which can be very confusing and I am still a bit puzzled by a lot of this but the original one was ml Lib and then later on they came up with another library called dot ml machine machine learning level and what really what the big difference is is that ml leverages the data sets API which is a newer a newer API that uses this catalyst engine whereas spark ml Lib uses the RDD API and so the result is that ml is supposed to be a better performing library that's not necessarily true as of yet there's there's pieces where ml live is actually still preferred so you got to do a little back and forth to try and figure out which one you can't you can't go wrong either one will produce good results it's just you might want to fiddle with you might want to fiddle with both libraries to make sure you get the right performance levels and we don't talk too much about pipelines but pipelines are you know this more generalized abstraction about how to transform your data set and you can do hyper parameter searches and lots of lots of lots of documentation to go through on the on the website and really probably could spend a few days talking about ml Lib and to cover it in sufficient detail but but I definitely want to get it to where you can load it you know get it working get it running on your laptop and and feel comfortable running the examples so I'll show you what k-means demo looks like how many people are familiar with k-means algorithm okay okay so I'll talk a little bit about key means what what k-means algorithm is I think we are in yeah we're in and this is gonna low whereas K means data okay I spark so we're loading up PI spark here and what we're going to do is cluster these data points so this is a XYZ coordinates or you can think of them as a list of coordinates they happen to be the same XYZ coordinates so this is a you know x equals 0 y equals 0 z equals 0 second one is x equals 0.1 and if you look at it it looks if you were to plot this on a 3d plot you would say oh I bet you these points would cluster around that center point zero point one right because they're kind of close and then if you look at the second one you say oh yeah that would probably cluster around this nine point one data set so that's basically what clustering is is it it takes usually Euclidean distance between points you can think of these as being you know three dimensional vector it can be extended to an n-dimensional vector and it takes a Euclidean distance between those points in it it finds the ones that are closest to the to the center of that mean and then just updates it iteratively so it creates its what's known as an unsupervised learning algorithm which means it just kind of looks for natural structure in the data so it doesn't have any labels assigned to it it just says oh I think there's a cluster of stuff here maybe that's important and there's a cluster of stuff here and so the big the big workhorse for unsupervised learning in SPARC is is is k-means that I think there are others in fact the topic modeling one that we're covering next is a is also unsupervised but okay so I really I just kind of wanted to get you used to finding you know you can find you can find snippets of code in the docs you can find snippets of code on stackoverflow cut and paste run them see see if you can reproduce the results and that is you spend a lot of time doing just that just finding you know a lot of times it's just getting the right combination of API calls and you know everything working because the libraries are already there everything is working it's just a matter of you know who hooking up the wires correctly and interpreting the results so so let's see so so this one I think Center is will tell you the names of the clusters oh no how about clusters centers okay so this this one this there's all kinds of stuff that are that are there it tells you the centers it tells you K we defined K to be two so the K NK means is how many how many clusters you you want to assign to it and so the centers are here and then each member of those clusters is in there somewhere I can't remember work to access it but so now it well what's nice is that it found the centroids that we kind of expected here right it we we saw that the data was clustering around those two points and it and it turned out that it was so-so so the those are the kinds of things that you can do with with these basic algorithms and I think this one came out of the docs so a lot of this stuff is just there in cells you copy it directly out run it see if it see if it makes sense and the next one is topic modeling and this is this one's done in in in the data science experience you can run this on you can run this code on your you know from your laptop the only reason I brought I put in data science experiences because it has it has Brunel installed and that's kind of cool to do some plotting so G so this one is so this one's got two elements to it this one has a training let's see the first let's get the we're going to I'm gonna restore assets I just want to check and see if it's still running it oh this one might we might be might be had a look on this one and that's some caching issues on that one so I want to I want to review I want to review what's happening with late and dersh lay a location so Layton Dursley allocation is a method for modeling topics so it's an unstructured learning unsupervised learning algorithm and what it does is it takes a body of articles and clusters them into topics and it says the probability and then so it'll have let's say you specify ten toxin will say each one of those topics emits words and the probability of the there's there's a there's an article that has a fraction of words that were emitted from each topic so the bite by assigning these probabilities you can say oh it's probably got 10% from from the topic about you know Trump and then another 10% as from Putin and you know various topics but it's a mixture of it's a mixture of topics that go into each going to each each model so so I'll show you it'll make more sense when slow kernel connection [Music] maybe this wall I think I have a cached version of this that I can I can show you as well while we're looking at that yes so I'm just gonna walk you through the training of the model first you guys see this okay so this is basically a notebook you you can run you can run spark and Jupiter notebook on your laptop if you want to try to install it yourself I think you can run DSX local on your machine if you want to try it there Jupiter's a fully open source and DSX is built on Jupiter so if you ever migrate if you want to migrate back and forth between the two it's a pretty transparent thing to do to get Scala running in Jupiter it's called the Tory kernel and that that can be installed from the github but if you guys have any questions about any of that just let me know and I can be happy to point you to all of that what this is doing now here is it's loading the Yahoo newsgroups data set is this one yes okay how about this one still a slow kernel okay so let's that's okay well we'll look at this cached one so it's gonna load the data set which looks like this this data set is the the Yahoo 20,000 articles that are pre categorized into 20 different directories here so each directory contains a thousand news articles and they've already gone through and curated and say well we think these are all about atheism we think these are all about graphics etc so we're gonna what we're gonna do is we're going to model those topics with no prior knowledge of the the topics that were assigned so it what we're hoping that will naturally coalesce into into topics that are similar to those that were assigned and so this is first thing you need to do is create a vocabulary array this means the counts the number of times a certain word occurs this is I saying one of those word count challenges this is definitely something that's a large data issue here we're actually loading some data and so we're back here this is just all the boilerplate that goes into creating you're gonna create the corpus and then based on the corpus you're going to you're gonna train the model here so basically there's a lot of boilerplate that gets into you create this sort of documents object and then you create this Lda object with all the topics and then you say train the model and that's and that's basically it so a lot of these machine learning it's got ETL it's got in and then it's got some kind of training operation that usually takes a very very long time the output of that will be a model and then the model will be applied to new data right so so so that's that's usually the way that these machine learning workflows go and they're much more formalized in the ml pipelines but but that's the basic idea so so we so we ran it we created a model and now what I did here is I say I had two notebooks I say I ran the model on one notebook because that's a time intensive part and then I and then I had another notebook that loads the model and so you'll see that in practice as well so you'll you'll have some batch operation that trains your model periodically and then you deploy it somehow to some other maybe spark application or maybe some other service layer and so then what we've done here is we loaded that very same model that we wrote and also loaded the vocabulary these are the topics that it came up with the first so it gives you topic one these are this the the words with the scores and it's kind of hard to read there's a lot of stuff to go through so we do this this Wordle or text cloud and this will this gives you a little bit of a feel for what the contents are and I can't remember oh yeah this one we I think we thought it had a lot of it had a lot of electrical and I had a lot of scientific terms in it so if we look back up here this is the unstructured part right so it said it just gives you a nice one was it I think we said sigh sigh dot electronics we looked at that topic cloud and said that looks and this is how this is kind of what unsupervised learning gives you gives you just a some cluster of results and you begin to assign labels yourself so we're saying okay that looks like scientific electronics now we can show these with the weights and this has you know conductors Mathematica it looks you know like that and so we load the first our article in the electronics directory and we say okay so this is just the first email I don't know it it has a bunch of stuff and transmission to a reliable underwater that looks that sounds scientific and electronic wireless communications antennas okay that seems like that might be a topic so so we take that article and we say what what is the topic distribution for that article and and now it should return something and hopefully I will identify the topic that we believed is the correct topic and it turns out that the topic was was topic six was the one that we identified as scientific electronics that one seems to be the one that's most prevalent in that so it's the beginnings of a of assigning topics to incoming items so of course this is just very preliminary right we're just starting to get out of the unsupervised clustering study so but but but that's that's sort of how you might use those so the I do want to cover the library somebody was asking about you know h2o and some of the other linear algebra packages out there so IBM has an open-source version of that an open source library that that handles mathematical operations and algorithmic development type operations so it's a it's a very compact language this is a completely open-source package it's top-level Apache project and it's got some great developers that have been working on it for many years and we really like to see people adopt it and so let us know if you're interested in it we'll be glad to help you get going on this but the system amel project really is this thing called declarative machine learning and what what it does it runs on top of spark so it calls the spark runtime but it is just a very compact way of writing certain mathematical operations think of it as like SQL for math algorithms just very compact and very you know very good for certain types of workloads and so anything that has matrix operations it's it's really great for it's got lots of optimizations on the back end on how to how to do these distributed matrix multiplies you can also do things like this is a conjugate gradient descent I don't know if anybody went to the talk yesterday where we're talking about optimizers for the neural nets sure so so this would be one of those algorithms for solving you know for the weights of a neural net it would be a one of these optimizers and these can you know these give me hundreds of lines in if you try even if you try to write in Scala or Java or some you know some some language so so for this particular mathematical notation it's very very good for writing writing this kind of algorithms so if you want to get started with this let me know basically all you have to do is go to this you can't see that at all can you let's see it if you if you if you want to try to get going with there's two examples you can go to the download site and it's it's this binary here and you unzip to the location and then there's a jar that's in there and that is in the system ml 0.14 Oh directory and then at the command line the this is just kind of a nice thing to know about running SPARC shells if there's any library that you want to run manually there's there's a variety of ways to do it but you can just download the jar and then just do dash dash jars and then it'll bring in your library and then you can import all of the features of the library so I'll quickly actually I will in the interest of time since we are in five minutes I will just walk you through the code so you see what it looks like but the idea here is you create you do some imports you create this this creature called the ml context very much like a spark context but now it just you know has all the ML System ml stuff built into it and then this it just tells you versions and things like that so this is this is kind of how you do it in SPARC you you you write a string and the string is your D ml code your system ml code and so this one just says you know create this random matrix and create this random matrix do a sum and then return the output right so this is this little snippet of code you embed the snip snippet of code in your spark code and then and then you and then you just say ml execute script and it'll you know run the it'll run the it'll run the system ml so so it's it's really great for certain kinds of things especially if you're developing your own algorithms and it literally can condense you know hundreds of lines into you know 10 to 50 lines of code there's a huge collection of algorithms that are available and so this one will also allow you to oh yeah this one will allow you so if you don't want to embed that big string into the code you can load it from a you can load the DML from a file and the github has somewhere on the order of 50 algorithms so what we what the the intention of the system ml project was to release machine learning algorithms that were complementary to the ML lips so we have a lot of algorithms that aren't present in the in the SPARC ml Lib libraries so a lot of really specialized algorithms some time series stuff in there it's it's really great so check it out if you can and let us know if you have any questions on that one this is got a k-means example I think we're gonna skip that one I was thinking I was gonna cover the I got five minutes left maybe go through the graphics demos and then if you guys are still interested afterwards I think I can you know I can we can take it offline I can show you some of the others sparks training and our four ml ones that sound good you guys want to see graphics for the last example before we sort of officially wrap up before and okay so graphics is a is a graph processing library so it's just it's got api's for running graph parallel operations it's got things like you know pull out a sub graph for all the edges that meet a certain requirement it's got a lot of common algorithms PageRank is the main one that's available it's also got this great API called graph frames which is kind of this other additional layer on top of let you do SQL on your graphs and pass it back and forth between data frames and it has some other really cool pre indexing operations and it's got its it's a really it's a really neat library and the so so here's the idea is basically an RDD the vertices and edges so each every every vertex has got its got a single value but then every edge is you know connecting certain vertices and so those can be represented as collections and those collections can be parallelized and then there's some additional bookkeeping that allows it to be treated like a graph but creating it is really a matter of just taking all these nodes and edges and and making rdd's out of them it has a great view called the triplets view which which will print out you know pretty much all the information you need to know it'll tell you tell you all of the edges and then all of the values between all of the edges so that's the one that you you know when you're debugging you tindy to use the most so we'll just do a quick simple graph demo and it was a simple graph yeah so I'll see if I can I don't know if I can do this in two minutes but I'll try to hmm that's a it's a good question so it's usually you know connecting connecting the dots between so it's always like I'm friends with you and you're friends with her but they don't know that her and I may have you know me and her might be friends because we share you in common right that it's actually kind of hard to do at scale with any other kind of data structure so it's usually that set that complex chain of connections that graphs can work out the other thing is like distances you know if you traverse a map you can find the shortest path through through through through a graph using special algorithms that work on graphs but yeah that's the one this one here is you know supply chains for example so where did a particular product come from all the pieces that go into building it will try to trace that this one here we're doing a I have one where it's a clickstream example I'm going to and so yeah maybe I'll in fact just in the interest of time I'll walk you very quickly through the simple graph one and then I'll I'll show you the graph demos and then I'll show you the output of that one so then that we won't keep you here overtime but let me let me show you the simple graph demo and then okay so again you know feel free to run these at home and let me know if you're able to get them working okay so this is this triplets V I was telling you about so basically we've got this we created this users okay you create this users array do it again so we created this users array here on which created the users array here which has got to two fields so you can have two different descriptors these are stored in tuples and then the they have a long integer identifying the nodes and then you have relationships so now three is connected to seven as a collaborator right so that's the edge with the with the value so you know this can be a floating point this can be a string this can be a variety of things and then we have this default user which is John Doe if there is a if there's a missing connection and so so Peter is the student of John Doe which means that Peter didn't have a proper edge setup so let's see if we can figure out where that is Peter right so Peter Peter is a student he's known for for has an edge of four connected to zero but actually zero doesn't isn't present in the vertex list right so that's what this this default user says oh if the nodes not there then assign a missing value to it and so the missing value is John Doe so it does a little bit of extra bookkeeping for you once you create the graph and make sure everything's connected it the way you would expect it and then you can create this triplet as triplet view which which let's just show you so triplets this returns an RDD so you have to collect it to see what it is and it's just an array of of nodes and and who they're connected to so three is connected to seven and it has all the information so kind of hard to read here but it actually has all the information you need built into it to do all the graft reversals that you if you want to write your own algorithms to do traversals it's all in the triplets structure so I got I got two minutes to show you the graph but I will probably just show you so low clickstream so what we did here I think what I'll what I'll do is just describe it to you rather than go through the code because this is a lot of code but what we did is we said we loaded the clickstream data set from Wikipedia and that just means that just says that okay I'm on for example we said Watson Watson computer was a sinner no this is okay and then it says what the clickstream data was says oh if you're on Watson you had 100 clicks to IBM and you had another and then from IBM you had another ten clicks to this that in the other so what we did is we start you start it at a center node and let's say it has five nodes Dedic that that it connects to then you take the top let's say there's 20 nodes and you take the top five of those nodes and you see what those nodes are connected to and they take the top ten of those and see what those are connected to and so you can create a series of shells of what nodes are connected to what nodes in that way right so you can create you can get a sense of the network of the connectivity so you can do this at scale I created a smaller version of this so that we can see it running but you can actually run this code on your laptop without breaking it so don't worry it's a small data set I won't I won't run it so so there's two there's run and right click graph which will which will do the analysis part and then there's another one called load click graph which will wait let me remember which with the order of operations yeah so first you load the data do some analysis and then there's another one where you run and run it and write it which actually writes the the output in a form that you can look at with HTML so then it what it does is it puts it in this site directory creates an HTML file based on a template so it has this first it creates a JSON file and then from that JSON file it creates so then it just creates a list of nodes and then those nodes all have things that they're connected to links so that's basically just the output of the graph that it created by searching that clickstream Network and then you can visualize the there's the there's the JSON file and then this is the d3 graph and you can kind of see what's connected to what so you can also if you want to visualize this graph you can do it with python the if you this graph is in there if you have 2.7 it's Python - M simple HTTP server and that'll it's d3 so it has to be served and if it's Python 3 I can't remember what it is but something like that to get a server so this is kind of cool you can see some of the things that show up in the Wikipedia entry you've got Watson you've got google deepmind is associated with it you've got and funneling in of it you know so you do Watson the computer it also connects to Watson you know the guy who discovered DNA so the molecular structure of nucleic acid shows up three clicks away from that so so there's some kind of interesting things and these things are really endlessly fascinating the Wikipedia click graph you could you could you could really look at it all day long let's connected to Ginni Rometty and then Thomas J Watson so there's this whole you see this whole ecosystem of IBM and Watts and stuff and then then there's lots of other things that that is connected to Watson jeopardy and then apparently Weird Al Yankovic was on Jeopardy I don't know why that was connected but but some interesting this is just a kind of a microcosm of what the full clickstream graph graph looks like and and so so yeah I guess on that note maybe I can answer any questions and then you know I can show you some other stuff offline or you just you know ask me later on if if you have questions so [Music]