scala.bythebay.io: Deepesh Chaudhari, Wikipedia RecSys
Recording: scala.bythebay.io: Deepesh Chaudhari, Wikipedia RecSys
I you good afternoon everyone it's delight to be here I'm data scientist and I'm currently working for a gaming company where I get to play with different machine learning platforms because we are upgrading our machine learning pipeline so among that includes different data science platforms machine learning frameworks and I get to build different projects like churn prediction recommendation system fraud detection systems prior to that I worked with data breaks skymine company behind deep learning for Jay and Wikipedia during my master's in data science so these are some of the demos that i'm going to show today the first demo will be focused on a community edition of data bricks with MLM wat to work and the second demo would be on sparkling water of the h2o and sparks combination so this is the quick agenda so this is graph that shows number of editors on wikipedia as you can see there's a sharp decline from 2007 have been wikipedia ambassador from 2010 and i contribute to wikipedia i'm pretty sure that most of you use wikipedia pretty often and it has been a problem in terms of getting more and more articles so that's that's the first challenge so the goal was to create recommendation engine for Wikipedia editors to make their tasks easier so the first problem is we have to 95 different wikipedias as as we know we use English one predominantly but we have German Arabic Indian languages so there are many different platforms the problem with traditional recommendations engines are they focus on text which is NLP approach even Stanford's NLP has limitations of five to six languages so Chinese Arabic German and obviously English so we wanted a solution that scales across all the languages the the obvious solution was to go for the text but as I mentioned it doesn't scale across all the languages people recommended us to use the links as well so i'll just show you the data set that we use on spark plug platforms so this this graph shows you click stream database and for example when I say click stream the San Francisco we keep it article gets most of its traffic from Google search engine and then another Wikipedia sources and California's Wikipedia page and most of the people from San Francisco go to another article such as list of people from San Francisco and so on so this could be one approach for recommendation engine but the problem with this approach is it doesn't consider the reading habits of users when you are reading Wikipedia articles you tend to read articles that are in the news for example you would read Bernie Sanders article along with Hillary Clinton and so on elect election related articles so they don't necessarily have links between them the data sparsity issues among the worst ones because as I mentioned Indian languages don't really have lot of articles written in them so there's definitely not enough data to create separate models just want to show you what kind of data we are dealing with so this is life edit stream of Wikipedia this is clearly a big data challenge and for that purpose we switch to platforms such as Scala and spark for obvious performance issue at this point I don't really have to explain why Spock is and Scala is ideal choice for this purpose so we used what to work what how many of you have heard about what to work so what to wake is the algorithm that basically predicts locally so in this scenario this is taken from stitch fix awesome blog where they are predicting all the words that around awesome based on the context of Awesome so if you have amazing word they would predict it based on this so it's basically words in form of vectors so in this case when we have limitation of not using textual data I use reading sessions so for example instead of using text i'm using number of articles that you are reading one after the other to train my work to work model because sequential data works really well with work to work so these are the preliminary steps and i have some scholar scripts on my github which you could use to scrape the data and load it in your data breaks notebooks and you can start using this model and do fun stuff with it so this is the clickstream scrapings which used to take one or two hours but with spark and a scholar integration it only takes 19 seconds to script one or two gb data so we broke these are the few steps I took while breaking down the requests per clients to be made sure that we are only using editors data and we are we made sure that they are highly contributing editors to understand the better patterns and turns out we have 370 million straining sessions and 1.4 billion training items across all only English Wikipedia embedding have more than 170 million embeddings and 650 million training sessions so what is embedding anybody's am bearing is basically representation of all those reading section in form of vectors and we push this to a metrics and then the traditional recommendation engine approach comes in the picture we just find the articles or vectors that are close closer to each other in the SEC in the vectors so this is the output i'm using sparks DSN a library so this this shows an interesting this is the data from last month and this shows that donald trump's article is being read together with melania trump Ivanka Trump these are totally making sense but people are reading house of cards articles along with it so it's kind of ironic but it seems like it shows the pattern of articles that are being read together pretty clearly now so the best advantage of this approach is we are not focusing on not just links and text we are also considering the different areas of signals that are coming from different people globally so we're focusing on search engine patterns they are searching on current news and all sort of other actions that human influence wikipedia in a way this is clearly multilingual and we twisted this framework on chinese languages and japanese it shows a pattern of clear succession so the future work is we will use this for translation so for example the articles for barack obama really really well written in english but they're not well written languages so if we use this recommendation engine for that approach it would automatically translate it and create indeed indic languages and German languages databases and articles more richer we can also use link recommendations for this so for example if we have seen a pattern of using political articles being read together one after the other we can just put links into the Wikipedia articles and then make it easier for readers to move along the Wikipedia so the next demo would be about sparkling water it focuses on stock exchange prediction and this is one of the products that data products that I stress tested and we built at my project so i'll give this mike to severe and it will talk about it thank you so thank you Depeche so in the in the framework of our collaboration and evaluating you know platform to to do data science we've been working with the patients on some simple data processing using spark the spark notebook and h2o because one of the characteristics of data scientists is that you know they want access to different valuable libraries they know a lot about different methodologies to get the best out of the data and the best service that we can do as provider of technologies is give them access to to really a wide range of libraries so when you're choosing a platform that is a development platform you want to have ants in many different technologies and so this is just too to illustrate the fact that it's it's quite easy actually to integrate into some platforms and specifically the spark notebook something like like h2o so it's just a simple example the spark notebook is very similar in interface to ipython because it's the same basis so you know UF cells you type code executes it it's the same as the data bricks cloud-like interface so what we did here is a is a simple process the idea is that we should make it larger we take some data from yahoo finance get it on our server from the their public server we will parse that data put it somewhere in parque then have another note book that we read that data clean it make it in the right format because we want to predict the value of one stock from the value of the others so that we have a proxy for example for the American Express price taken from the others that's a silly use case but you have some some use for it in real life set up if you want to find some arbitrage situation or if you want to work with markets that are open at different times you may need proxies like that so just a simple example where we do machine learning so this is the way the notebook works and it's the scatter conference and I like to to say that okay you you're in skala you're in a notebook you want everything to be self consistent in the notebook so why not mail from that do from time to time you know a system call to to get data that is outside of the world getting in local and I'm saving it in your infrastructure in your data like so that's the goal of this of this notebook very simple so you see you have this this system course and directly the only thing we do here you see that the data is consistent locally but we want to get rid of that that is not sustainable so directly we read the data we spark there might be a lot of data here is just a little sample and also the value of Skala of course is that you use you know easily your case classes to be able to parse structures that may be a little tricky we all like to play with dates right and when it comes to to data we often have them so here i have my specific parser and my specific structure because i want to be able to point to day month year maybe and i have a timestamp so this is my structure the way i read my data and i have the structure to build my price which is you know stock name a dates and simply a price and parsing the data is just you know for each line you will parse the line get a quote object if it's well embedded with the the options here i can get a clean data set in some way so no stray data should should get in I have no errors and then it's pretty simple I create a data frame out of it so in it's something that is having the structure of rows so it's a set of rows and each rose as a ski map so I have you know string the date structure that i just created and when I save it on file as a project file if I read it again I have for free the schema that was saved so that's really elegant way to do that cut the first notebook take data from the outside world getting it into my my storage infrastructure then of course I want to do that something with this data so i have a second notebook that aims to process with h2o these data set so again very simple operation i read data in parque forma when i do that i get as i said the same schema for free so that's the the power i would say of these frameworks but then I want to do something that is a bit out of the the sequel space so if data that basically means one row is pointing to a single point which is a stock symbol a date and a price and because I want to compare the price for each date for different stocks that means that what I want in a row actually is the price of each stock so i need to gather together a number of a froze and make them columns and expressing that it's equal is pretty hard that means that i want to get out of this equal space and that's the value i would say of the the spark interface you work with data frames with the sequel world but if you you're not comfortable with some transformation just go back to the rdd world you get back into a very functional language space where you can do whatever you want with the structure you want that's exactly what this notebook is doing so first i have a cleaning set up i I don't enter the details it's not very important here what's interesting mostly here is that in this filtering is that I use the leverage of of sequel because I do a join an inner drawing to remove wrong data so that's something that I find really cooling those those interfaces and then I convert to our DD strongly typed with my strings long double very easy to convert a row into into a tuple of objects that I that I have here and then you know grouping things together by date and ordering in such a way that I have the right column that I wanted and I said you know the right name on the right columns when I go back to a data frame and I had exactly what I wanted at the start that means each column represents the price of a given stock for different dates I've removed removed the dates out of it i just want to to see correlations in the price okay and it's not very long code you know and but conceptually it's quite tricky you wouldn't do that in sequel only and then we get into the interesting part i could do that with ml lib but i want more than that i want to leverage the power of h2o who is familiar with h2o here very few so h2o is really a great framework for distributed machine learning the reason is that they have their own implementation of MapReduce in memory so distributed but the fact that they are working with you know linear algebra and machine learning makes that their structure is really optimized for working with floats categories and stuff so it's really extremely performant so it's really it's really good to work with that and the way it works you know spark they have this worker so distributed workers the one when I do this operation of creating on h2o context what happens actually is that I will create a an h2o worker living within each worker of sparks so I have a cluster of h2o nodes living into my cluster of spark and when I have a data frame I can actually ask for conversion of this spark data frame into an h2o frame and it's a memory to memory operation so one partition translate into a partition from the other framework it's just amazingly fast and that's exactly what's happening here you see I have my data frame here and I say oh convert did for me in an implicit way into an h2o frame bomb and this data actually you can browse in the h2o interface here because they have their own notebook so now i have two notebooks living together my master notebook where i want all my codes to be sustainable i would say that this party notebook but i can explore what's happening in memory on the h2o side here so if i click on get frame i see that i have here my my main data set this data frame here and i have some statistics so that's really pretty cool and then you know i use the framework to make training and and test set here and I then call for the linear regression on these data sets and get a model out of it and then I don't want to work anymore into my h2o I have my model I have my I want to make some prediction because it's living in there in h2o and see what's the result and so I have operations to say that okay on the model you make a prediction here I want it back as a data frame a spark data frame and then I extract these data and just take the right column which is the double with all the prediction and I will zip it with the the actual value and collect it in the form of an array so I have an array of double double and a if I ask for you know a display of this array what i get is a table so the prediction and the actual value and I with several way to plot it so oh cool there is a correlation between my prediction the actual result that makes a bit of sense then of course you can compute our stuff like mean square error whatever what i want to point here at this stage is the fact that we've been doing something very classical you have a data set you write code in a notebook you process it you get an output somewhere that is saved and then when we take that output we read it from a notebook we process it we generate a model and how do you keep track of all these information actually there's a way in the spark notebook in the the unperturbed price edition that just allows to do that so we generate from the work of the data scientist a view of what is happening here that is I have data in park a format it has been read by code that is in a notebook so if I take this part and this data in parque has been read by another notebook with code generated the model which is actually data saved somewhere and that and it can go on and go on and go on and so it's a platform that allows you to know that some data set has been used for some certain purpose that from this data set you have clean version of it that is available and so that's a very important feature I think of modern a platform is to get you know an automated cook way to collect information about all that is being done by your data scientist I will leave depeche make a conclusion as you see there are many other different tutorials that I've uploaded on so there are many other tutorials like this have uploaded on my github repo and you can use them to go through them and learn more about it some tutorials were more than 40 minutes so I just tried to squeeze as much as i can in two sessions thank you so much for your time [Applause] you