SF Scala: Andy Petrella, Spark Notebook: beefed-up REPL for reproducible distributed data analysis
um so yeah has Alex said I picked up Scala and played a while ago already Scala was almost four four years ago okay for San Francisco is pretty much okay but in Belgium I was like I mean UFO um so yeah so they didn't really get into the functional programming now even um so yeah actually I'm doing a lot of rad spark in my daily work so because some people say I'm a data scientist I just say that a mathematician and um at some point in time I was a bit yeah you know spark reer is really cool but then you want to replay stuffs you want to show stuffs and then you're screwed your buses or your client says okay what what is this black thing with blank with white letters I don't understand word so please give me some charts okay so I didn't want to pick a tableau or all the tools like these so I as bridgew if there was a way to integrate spark into their Scala notebook I left the issue open for two months month and then I decided to pick it up and um now it has evolved so much that I cannot even merge it back so yeah um so I decided to pick the issue myself and then okay I I created the spark notebook out of it and finally I I think that 80% of the code is now own my own code actually I Evol it so much it was based on uh unfiltered and stuff like this of course I migrated to a play two and uh um I changed the UI so I integrated so many stuff so today I won't show slides I will try to do my whole demo explaining um what is spark and how you can use it you uh through the notebook itself so who knows about spark here that's great who worked with spark today that's really aesome okay that's pretty awesome cool so I'll try to give some detail about spark actually the thing that I would like to to show first is how uh okay I'm not going to give any details about a notebook if you want some you just you can just ask me but actually I find it pretty boring so maybe you can just look at what you can you can do with the notebook I I guess that I should increase the is it okay like this yes yes okay cool so yeah so basically what is this spark notebook it's just a a web interface so there was the this kind of notebook um um Talk recently uh here so I won't give more detail about what is a notebook it's just a a freaking uh HTML file but this this HTML file can uh run code here so behind the SE I have my my server running here I display just the the regular you know uh folders that I have in my notebooks configuration and okay so maybe I can really start with a um simple example of spark the simpler that I can have is the 101 that I created out there um so yeah actually maybe I can just say that this guy is just as BT run in the project that's that's nothing El that just as BT run in the project there are some drro IN Zip and and stuff like this but you can run it from SBT really easily anyway so spark one 101 there are a lot of people already knowing uh spark so I we go great quickly so what I can do in Notebook is just load all my packages but by default actually here I just clicked on the notebook and actually behind the seene uh if I don't misspell it have my spark that has been started right away so I didn't have to download the tar or whatever I just have my my spark running by clicking on the on the the link in the notebook okay so I can't have access to this UI of course there is nothing because I didn't execute anything so also the spark jars are uh yeah the spark jars are already included in the project so I can already load them and then I can start looking at what I've I'm trying to explain uh okay I move these things and now my computer is crazy okay it's back try to squeeze my my my knees um okay so I'm trying to explain what is Park and to explain what is Park the best is to have some text and some codes right and here I have my uh small mup uh Mups that come on okay enough now um so yeah so to explain spark here I explained what is a deck what is a job Etc blah blah blah and then I can run it right away and showing that actually when I trying to load a file from spark is just defining how to load a file in spark is not executing it executing it so I have a mar DD there and you see this Mar DD is actually a subtype of rdd of int so rdd is for resent date distribut the data set I guess the most of you knows about it and a rdd is typed in this case it's a name because I mapped all the string all the lines into the file the versis log the classical one into its size so finally I had this rdd of string that converted to a a rdd of in and then I can apply a distinct however nothing have been processed so see nothing has been processed it started for uh 3 minutes now almost and then just did nothing okay cool um oh um um so this mapper DD is actually something that defines that he has to it has to map over another DD and doing something restraints to in all right okay cool um so when I have this rdd I can maybe Group by um Group by the last digit um so we have have if I do a mod over 10 I will get the last digit of the size then I can Group by that means that I will have basically 10 items in my LD one for each uh digit and then it will um map it to a Nel of end so be really careful when you use Group by in spark because it can blow your memory really quickly um so if I execute it again spark is very lazy maybe is even more lazy than I do and actually see I have my Shuffle dir DD why is it shuffling because I have a group by basically if my CIS loog was really huge I have to to split it into hdfs that means that I will have several partitions and then uh in order to blah blah blah okay um in order to have the digits for zero that are coming from all the notes I have to shuffle all the data to a single note and then having this itable of in anyway so I have my my data gred by digit um everything is lazy so I have my shuffled rdd so um nothing has been processed in the in the in the in the background okay um even more functions available in spark I can map over the data adding one then taking the even out of them that means I will leave the odd in the data and finally I can even create toos with fancy computation that deserves nothing okay um so um when I have these different rdds um I can join them why because these are DDS I can show the output by e in Ho actually if you don't remember that it's O you can it Ed and you will have an help uh for the different uh shortcut that you have okay okay okay one two one 1 two one two so um all right since I have a m DD over int and int toles and the order there is a int iterable of int I can join them right uh so it like inner joining the SQL so uh to join them I can call join on one of them and then finally spark B and thec will figure out how to join the keys uh all together cool finally I have my rdd of in itable int that's the first rdd and int is is coming from the second one so now the interesting part with the notebook is that you can print out some result at some point in time so I have if I want to know show the firstand result of the Jer DD I can do it like this so I will have my data in a in a rather freacking string but if I want to have something a bit more fancy I can just return the list and I will have a table showing the list all right so what it shows it's a tole of X and Y basically so it will show the first item and then the the values is going to be the compact buer of the all the um um sizes that has been discovered into the file and is have been has been mapped by the last digit that that's why everything is four for instance there so 240 54 and so on so nothing really interesting but still you can show it really easily it's better than in in the Ripple actually when I show that to someone at least it has some sense oh it's a table great um anyway so something interesting with uh with spark is that you can show um what will be processed so in the case of join I can show what is the dag that has been created it behind the SE so actually this is the thing that has been processed by spark and I can show in the job right now that's something has been processed finally so I have these takes um that are um consisting of the distinct that I've declared before the map of the sizes the group by for the digits and so on and finally the take that has taken the data out of the of the fires and the computation sorry all right so we can debug a bit debug a bit with the to debug string and uh that's that's pretty much okay now um sorry um something in Scala that we like to have is types all right so toose is pretty interesting but finally we want to represent our data into something relevant and understandable so if I want to manipulate some financial data I would like to Define I would like to be be able to define a code class that will contain some stock date and price um fields in it and so I can declare them into my uh Ripple into my spark notebook quite easily so this uh case class has been defied defined and then I can process some CSV file quite easily and and map it to this uh to this case class so how I can do that I can show the content on the file or there but this file is actually on Amazon I should have on S3 actually so I should have already locally I hope it's in TMP so oh I'm going to download it anyway um so what I can do actually here is that I can um download some file from S3 and then process it but normally I should be able to process it straight away from S3 but I have another example for that um I hope that it's not going to take so much time otherwise I'm going to be quite screwed um I should have prepared this one um yeah see 35 persons 37 40 41 43 44 okay so it should be fine in in a few um okay the thing is that actually I I should I could also just start another notebook while this one is running and I create I can show some vistion if you like actually there are some several vistion that has been integrated into the notebook one of those that has been integrated recently is a magic plot and Magic plots is something quite interesting for uh data scientist that don't want to integrate anything from D3 or whatever actually is is something that allows you to to automatically sh plots based on the type that you are returning okay so in this case I have a simply a sequence of doubles so if I execute it what it will return it will return first the a return the um the actual numeric range but also is going to return a series of plots that has been created automatically based on the data that has been just returned okay there is a random on this microphone I mean and okay um so this is not only the this is not the single graph that can PL on on doubles there is this bar chart but also there you have the the regular plot line right okay that's fine for a list of doubles but when you have something like pairs what you can do with topper of two actually you can plot something like a table the regular table of course but you can also plot a bar plot with the in the axis you have the first entry in the tle and the second one is going to be the the the the the value in the toes but of course you can also have the uh the lineer plot Bas that has been included into your into your maybe I'll just hold the and we'll cut off which is your M yeah which is I want to turn off the the other one small okay so let me this one this one okay let's just should have a back man thanks you okay um it's better now I hope okay so that's for the tles uh but when you have a a list of tipples for which the first item is not an INT or number anyway it's a string okay in this case I'm mapping over the characters and I convert it to a string and then I want to print something that has been zipped onto this list of pair of even words uh even numbers sorry if I wrot this guy I can have the data but also I will plot the regular table and of course when you have things with strings we will have a pie chart right because actually it's about strings and doubles so come on okay though um so you have a part chart and U because I mean when you have a string a douc it should be print into a a a part chart of course right it's categor categorical values and since you have categor values you can also PL bar plots where the xais is actually the different library that you have okay that's cool with deers you can also show some different toes with this fracking underscore one 2 3 but if you are fed up with this Tios because you are lost into your underscore one underscore 2 etc etc you want to have some types actually you can Define your own type in this case I created point class which is actually a double in basically just a wrapper on two toes if you like and if I have this kind of class actually I can use it to randomize to create a list of randomized points like this and actually I will still have my table with a and b as being the the the fills name but also I will have the bar plots and and the and the uh L plot of course okay so because it's a it's a type that contains only two things a number and a number so we can PL it like this um if you want some category values again create some with a lot of Chinese text because random string is always generating these but anyway anyway even I understand the word of that I can at least look at the pi so actually this appears 19% and this 18 and this yeah apparently is not uniform okay and I can also have this power plots so it has been infer because we have a class that contains string and and and numbers and again for a random type let's say I can I can create some tables out of it and for now we are working on a on the way to click on the tables when we have a table like this with multiple entries and and uh based on I always speak louder so when we we click on these columns it draws automatically the charts okay so because the tables are fine but if you want something more fancy you need to do something all right can I ask you a question yeah uh so these charts um I remember Nathan Hamlin forked scull notebook to add the graphs yeah is this something you also had to add or is it that for of his actually I merged it on my and then I completely refractor it got it so actually if I want to show some more plots that using this um um uh reactive programming patterns I can show maybe the Rick show one which is about um some time series and okay I'm not going to show much but actually I'm going to create some time series and and a integrate the D3 uh plots and something like this anyway so um what I will do right now I'm going to create uh a plot so now have my plot there you see just a regular time service plot with different things but now I'm going to create several Futures right I'm going to launch them every second or every two second okay no it's 2 second otherwise it's going to take much time okay let's let's stick with 10 seconds and every 10 second is going to create a new time a new bunch of Time series and I will apply my new time series to the previous widgets that have been create that I created just there right I have this p is actually a wrapper onto R show TS R show TS being a just a a um time series plot plotting framework and when I apply a new time series to a widget is going to automatically listen to the data and then it will update it so if I do that in 10 seconds you will see that actually it it will just change but this code this future sequence blah blah blah will be run will be running onto my server onto the actually onto the notebook uh uh process and then it will be sent back by a web soet until until this guy picked it up so in a few seconds it should update I hope it should update it unless Murphy comes in the game or Murphy is there okay I'm screwed anyway I have other examples um you know what I'm going to show it right now I just show quickly what is the spark cor um I still have my stuff there that I can show after but I can switch directly to spark to spark streaming uh do you know about spark streaming how it works and so on okay just create a bunch of rdds out of a stream actually I can show quickly a Twitter stream and a Twitter stream a basically um well integrated into spark because spark is providing this you know the the this line there is including a dependency another dependency onto Spar streaming Twitter Okay so this is how you would Define new new dependency and for now is going to use Maven to download the dependencies and then includeed into the notebook and also it will include them into the spark configuration in order to send them to the cluster um but in the current um um in the current uh no sorry in another Branch I have an SBT implementation so I can handle both Maven and IV repository so this is how you would declare a local repository and how you would declare some dependencies like this and how you would exclude them because there is a minus a dash before so it will exclude them out of the of the environment because actually they're already there okay so if they are not already in my repository behind the SE will download them ET Etc other thing is that in spark you have to also include dependencies into your spark configuration otherwise you will run into turbo into your cluster because they won't have the jars to be executed out there so um in some sense um this guy will add them to the spark context I can show it actually so I have spark context two debex two not strength two DB string ah spark context get to Deb string there you go so here we see in my jar configuration order there it included all the jars that I needed to should be shipped to the cluster so it has been done for us um okay so what I can do is just load my different keys from Twitter that I've actually I pre previously I update up sorry I've run a script that creates my uh un varment variables that has been um created before the Spy notebook so now I can access them straight away from the system environment and put it into my system property this this is how it goes with spark streaming for Twitter and now I can create the regular streaming context onto the spark context and I want to have batches of two seconds all right and I can now create some filters so I'm going to create the filters onto the the the stream of tweets for Sparks SCA and music why music otherwise I won't have enough tweets for spark and Scala to be shown so I've included music and music means I was I would probably have also LOL ass you know and this kind of stuff but anyway so for the stream it's fairly easy to create a d stream over tweet because the there is this Twitter etails that comes from this uh you know uh this park streaming uh Twitter thingy I create the stream nothing has been processed yet because it just it's still lazy just a stream okay and then I can create I can map over and flap map over the status I can take the status which are Twitter Stratus get the tag split onto uh split on on the the blank character and then filter everything that is not a uh filter filter out which everything is not starting with a a hashtag and then I'm just creating some fancy operation onto it like okay I'm creating a t out of the word and mapping on to one and then I can create a aggregated uh summation on two different chunks of data but for a period of 60 seconds and and I will basically turn it into 20 seconds otherwise we will have to wait for 1 minute and um so yeah and after I do some some transformation order to have something relevant to show and now there we are there we are I have this Ur thingy there that creates actually a UR structure a basic HTML Ur structure but also it's a widget and widget can be applied with a new bunch of data and and knock out because it's this is the library that's used behind the scene knockout will listen to the socket and when data are coming from the server it will update it update the the UL uh uh view uh in the notebook so now from now on I have just nothing all right but now I can count the top uh the top hashtag in the Twitter stream but since everything has been defined laily in spark we don't see anything until I start the stream all right so at some point in time you'll see every two seconds you're going to have something shown in the Ur So it's processed on the server and it's converted to some some JM things it's sent over the web socket and The Knockout is picking it up and update the the uh the widget automatically so we see that actually yeah music is there on thep top nothing about spark and SCA we can be ashamed and okay that's that's basically it with this uh with this kind of automatic uh viewing I don't know what's going on with this Rick show thing you have to to look at it a quick question okay um okay that's was for for the the viewing and the and the streaming um there are other things that are quite in wait wait a minute should ah yeah we have the data right now so we can do something with that so remember we had this case class code so I can now have have my data out there I can actually I have another scope sh that stands for shell and actually I can some I can put some shell uh commands out there and actually there I'm just showing the the the the size of the the the file that I'm just downloaded and you can see it's 174 uh megabytes of text right okay that's boring yeah I don't know yeah yes it is um it's probably my fault actually um so uh so I can plug something like this I can do of course LS I will have my my my list of things anyway okay so you you got it you basically got it right um so um all right so what I will do on this file is that I I'm going just to read the file using text file on spark context okay that's one of the reason why I'm using all the time Spark all the time because I don't have to know you know fire new file reader and the path and then get llies and then whatever I don't have to do that I just have to has text file that's more or less it right and furthermore this this good guy will create some partitions out of it and it will process it in poal so I don't even have to bother with know I love haa but I don't have to bother with haa when I have to process a file so it will do in in parallel just for me right so it's very interesting to have it because I'm so lazy and okay so I have this text file out there I can map on the on the text on the different lines I split by the coma because it's basically a CSV and then I map it to a list this list I recovered in a very untight and very unsafe manner this way because I know that there is there are only three items in these files and I map it to a code right so and I have my exception so um oh no no it's not exception sorry it's just a a warning saying hey it will fail at some point in time no now in the future because your case pattern is not complete thanks um okay so now I have a map I have a mapd but it's simply a rdt of codes my type that I have just defined into the rer in some sense all right I can do some fancy computations like I can Key by the date all right so I have basically a string and code there and then I can combine by key it's it's like reduced by key but for um I mean for who for who wants to do it actually actually it's just a way to to specify your own redu by Key by specifying how you will map each item on a local note how you will process the data on a local node and how you will reduce the data onto the different nodes all right so just a way to do Reduce by key uh in a in a more fancy manner so when I have this this guy there I can just print the result is going to be M by dates so um here I'm just printing the minimal the the the stocks that has the minimum price for a day so for 17 in of October in 12 it was FC andn it was zero and for Max of course is the same ah yeah what I wanted to show out there if I I can find back the right UI is this one right so if I process this file like this okay if I run it again it will take another two seconds right so see I have this take there that took two seconds and this take there that took two seconds that yeah but however the computations are exactly the same right uh so spark is not helping us in this in this case well wait a minute actually spark is there because is so useful using the memory so air it didn't use the memory I computed something and then I had to recompute some the same thing again um um without any caching so what I have to do now in order to be able to run it run this code several times without having to spend my time or waste my time I can find now a Max by date so it's just the reverse and I can cach it over there right so I can cach it there that means that actually when the first time it will run all right it will still take two seconds all right but the second time there is going to take only 9 milliseconds why because my data has been partially put into the memory I ask spark to put my data the process data so far into the memory in me so since it's in memory um spark can directly process it and return it without having to do anything within the file whatsoever and why only 17 person because I didn't process all the data it just has for sorry for um for two items so so he process part of it and then it it put it into into the the the uh the cache all right so we did something with data that's cool um of course if you don't want to do this we can simply return the list and we will have this fancy part charts so um since we only return two of them we have this clearcut 50% one in one um okay so what could I show ah yeah you know what I'm going to show some SQL who loves SQL I love SQL just kidding um okay um so SQL is yeah one of the API that has attired a lot of people coming from different world and um I'm going to show some stuff that we can do with SQL out there with the mainly with what has been integrated into D to to the to the notebook so again I have this code class all right I can Define some stuffs to connect to S3 right then I can do some stuff like this okay I can connect to S3 I can provide just a path to my data so rather than having to download it like before I can just point to it on S3 and what I will do again I will split on the comma and then return as a code about after I will purist it on the cache I will cach it locally on my local F system why because I don't want to pull the data out of s all the time right so in this case I will puras it which is more or less a cash but on this only so you have like you have several um options for Storage level you have memory memory um sterilization um you have different kind of cash system in spark that can help you you can even cach it serialized in memory replicate replicate replicate it twice something like this and also you can name all right you can name your um your uh rdd in in in this case you will have in your spark UI a name for the cter DD so you can Rec recover your pieces all right so now I can play with my data on S3 and I will play with it using SQL all right so for that I have to deay to to to declare this SQL context built onto the spark context and I will also import this create schem rdd in order to have more inter interpolation in my data okay I creating this context now I can register the the the the new rdd you know this codes there is an rdd all right and I can register it as a temp table into the part the SQL context and the timetable will have the name codes all right and now I have this table I can do some SQL queries on it right and for this I have a specific context uh SQL and behind the be between the brackets out there the the angular Brack um the the square brackets there I can specify a name for the written value I will show it right now so I can execute it so I can show some stuff happening uh wait after um actually so this SQL is only selecting the dates and the count of um the date and the count for a filter data so I'm going to filter codes by specifying a particular date and a particular price that I want to filter on and I will finally group on the date okay so the group is going to be applied on the the the count is going to be applied on the group data by date all right so I can create this one I got what it created is just a simple form all right there where I can specify some data like this and no okay actually it detected that there is a change that it tried to compile the SQL but actually it misses something so actually the SQL is not compiling so it says before um behind um sorry below that actually it failed to compile the the SQL actually it misses a date somewhere a price sorry somewhere so I can put maybe something one 1,000 and then okay right now it's it create it detected the change and then it created my schema rdd out of this SQL with the uh date string uh included into it and the price that has been replaced as well and finally I I I I should have something older yeah oh yeah S3 is working against me it's pulling out the data from S3 and then it's it's a bit slow I guess maybe I can yeah maybe you know what I can clear it out and I will cheat a bit because because since I downloaded it there I don't need this one anymore ha so since I have it I can I can use it right away there and there there there and every if everything is fine let it should have updated yeah it's doing the work so now it work it's working I have my schema and normally this guy should show some stuffs right there all right so if I maybe putet I should have another job that has been started uh somewhere uh somewhere yeah now yeah okay so it's pretty reactive because you know changes events are sent all over the places and and behind the SE the spark SQL is is handling it it own stuff so we can see it right there uh right there so we see here all the task that has been launched Etc that's the that's the output the Delux for the for the Spy notebook okay so this is how you could play around with the with this uh SQL by generating some Force out of some form uh out of the SQL itself and actually yeah the B dat there has been mapped to a a local value on which you can react so that means that react the react things there um takes a functions that will uh produce a new output based on the rdd there so I will collect the data then to list and two string yeah I will output it into a simple pre preag okay um okay so SQL the cool guy ah and if we process some GitHub data you know actually GitHub is producing some archives and these archives are containing basically events between users repo um and actually so you know you have a user that pushed into a repo so you will have an event into this archive saying okay this user pushed even to this repo so actually what you have you have a graph of of user and repo being the nodes and the edges are actually the event between the two okay so you can process this guy um I'm going to take this one otherwise he's going to download the internet come on okay so what I will do I will use graphics and Graphics is a graph API uh part of the spark uh environment let's say ecosystem so now I'm going to declare this uh particular um dependency order there I will remove what I don't need and then I will also include Jackson because actually GitHub had the wrong IDE to publish this data in Jen um I will download it is not that big this one is around I think maybe I have already already downloaded it I don't know let's see yes I have it so there I just downloaded it and just um unzip it and put it into a regular temp file so I have it and I can show some uh some lines of it so see Jen why Jen I don't know um so actually there are two two ways to handle Json files in spark you can just use text file and then you will have an access to the lines and but there is also in sqls context a Json file that can already under the the lines has being J on you Rec create the schema and so on but it's not the purpose of this this one but still you at least you know it now so you have this text file being written right there and you can count how many uh events you have is 1,300 and what what I'm going to do is just parsing all this data per partition because I need to create a mapper I don't want to create a mapper per items I'm going to create one mapper per partition since I will have fewer partition than items and I'm going to map it to uh the the types that I want to to deal with and all right so I can take the two first items and showing that actually now I have a yeah okay so we have things that has been mapped from U this um this Json object hold there this is mapper actually we picked we picked the object from map anyway okay so now the the interesting part is how to deal with this data in graphics I'm going to import all the inform all the package that I need that said um that are rdd and graphics and now I'm going to process the data by creating rdd of vertices which contains all the information coming all the information um of the of the actors that means us as GitHub users and the repost that is our that are our repos right so I'm just paring the Jon and picking the IDS of the actors and the repo and I'm going to create an Union of these guys in order to have my vertex rdd that's it okay that that that will be the nodes in My Graph right and now since a graph is not only nose we need edges I can create the edges by again reading this file and extracting what type of event occurred between these guys the actor and the repo and I'm creating the the um sorry um the bilateral edges so I'm creating in one sense in the other because what I'm going to do what I'm going to do I'm going to actually try at least is to detect um um communities into the op open source world so using a graph and a simple algorithm like connecting component okay you can already detect some I mean very dense um uh communities in a graph so this guy is going to run the algorithm for me so now it's going to take the vertex and process them uh a lot of group bu and stuff like this um I hope it's going to take quite some time it looks like H takes quite some time all right okay see there we go so it's done so now I have my connected component okay so what is a comp connected component is a it's a sub set of the vertices that are connected in some way so there is a path between uh all pairs in this in this sub subset of the vertices so what I can do is trying to detect to to to um to show these groups and for that I'm just manipulating a bit the um the uh the results in in this s in this um count you see the number of connected components it's a lot of connect components why because I only I took only one hour of data and get up that means that I don't have that many data out there right I just have what seven Mees of data 11,000 uh um even so it's not that much so that means that everything is rather decoupled so guys from airline are committing guys from ja from JavaScript and so on so a lot of connected component has been created anyway I can try to show um some clusters so if you want to look look at the code I can explain it but it's not really worthy it's just manipulation of the vertices and the edges order to detect uh what are the vertices uh what are the the connected components sorry I can show this is just a simple uh uh sorry a simple uh visualization and now what I can do Bas based on these two calls I can cluster so what I will do I will try to detect which uh names contains for for instance the GS uh characters the GS string and then I will say okay all these guys are grouped into um I I return the connected component I say everybody has been uh part of the same cluster okay so now I'm trying to to fetch all the data from the graphs and collecting all the data in a graph in a graphic manner so normally be the C should have launched some collects yes oh there and finally I can show the result being a pie chart of the of the information but actually the layout might must be a bit better there because now I have my clusters in a in a better way that means that I have this repo order and I I light it those ones that I containing GS and see flowchart GS you have angular GS you have all these clusters have been created out of the Git Up events only right I didn't do anything than just that just collecting the data and um we can do it also for Scala yeah normally I I have also the I try to to keep it short actually has only 17 minutes of battery so um yeah normally after that I want I wanted to show some machine learning stuffs and uh and some other thingies but I will try to keep it short what I will show after this one is a more elaborated more elaborated um uh project using AKA Kafka zuke keeper and Spark streaming and the SP book all right so here is this for Scala so you see that there are fewer repos and those repos are come on dude um so yeah you see so scalaz has only two um this guy Ox 414c has committed to two different projects being Scala Z and patterns um you know yeah several small groups has been created so not that many commits has been uh done during is hour but at least we have the connecting command that has been created out of the data so if we would have more data we would have more relevant clusters all right so so we can print tables as well so the thing is now we could have a more abated example like you know uh trying to so there is this oanda uh um service it's basically a service that provides uh stock stock market options and stock uh codes actually and something like this so what we can do with this data we can try to start a Docker environment actually it's already started and what how it is started actually I cre I created a zookeeper uh zookeeper uh Docker file a CFA Docker file and there is another application that is this AA um um AKA small code that takes the data out of the PSE PSE actually the service and puts the data into Kafka and now I will show how to connect to Kafka and uh trying to have some more information about the the um uh interaction between uh you US dollars and Euro for instance because it's basically what it produces so if my battery doesn't run out um so see we had the spark streaming um uh the spark streaming for Twitter and there is also the spark streaming for Kafka so it's rather easy to connect to Kafka and start pulling data out and creating some stuff with it okay um so I will have to okay now it's gone I have my my jars I can create a streaming context I will connect to my Docker local braak okay where is the uh kfka running um and then I can I can simply um start listening to Kafka and how I do that Kafka utils create stream and then I provide all the thrift types because actually when we when I put into the data into Kafka it has been put into a using the thrift a thrift model so actually it's a binary uh uh stream that I have in Kafka and then I can rate it back using these types which are uh Thrift decoders and then I'm going to take the ticks out and uh return is a one that because I want to to to count them all right so this is how you would connect to kfka and since now we have a KFA stream we can try to Output some stuffs I hope there will be some events because we are quite late so maybe maybe the market I already closed and actually if I run this here I'm just fetching the data out which are Thrift items and then I take the count of the uh rdd because streaming is is chopping the stream into server U the stream is chopping the the data into server ldd so I can count for a particular DD uh how many items there there was there there were there and then I can take only one uh those instrument that matching this uh string and I will map it map it onto uh price and then compute the average and print it into this um to this widget out there so in order to have it running I have to start the SP the SP streaming context otherwise it's just lazy and doesn't do anything and if there are some yeah there we are so we have a a one the initial count was um 1,000 items that was returned by the polling uh behind the sea and okay no nine only then six were uh about Euro and USD and actually the conversion rate is now 106 so it computes the average onto the old tick so the seven tick is maybe 101 and 09 but there just the the Aver see the Euro drop to dollars right now live no yeah actually 00 0 that's fine okay that's basically it I think I run out of time now and battery so I can stop the context all right um I can I can help you out now thanks thanks a lot