Simon Chan: PredictionIO, SF Scala @StumbleUpon 20150122

Simon Chan: PredictionIO, SF Scala @StumbleUpon 20150122

Recording: Simon Chan: PredictionIO, SF Scala @StumbleUpon 20150122

thanks for being here um one of the many things we can learn from Deborah's talk and stumble app points experience 14 years experience is that building machine learning application on production is tough it's very tough but a good news is that guys you don't need to build solution from scratch you can use existing machine learning server so we prediction IO we build a machine learning server the goal is for developers and data scientists to build machine learning applications on production in a fraction of the time prediction IO Is Now showcased by giig Hub as one of the most popular machine learning products in the world and today I'm really excited to tell you the latest development the features that we have we have made major reamp from 0.7 version to 0.8 version um you you will be a great contributor you can add it into it so before I jump into the features the new features I want to answer one question first um why do you need a machine learning server so what what is a machine learning server if you're familiar with the data science open source ecosystem you must be aware of a lot of names like data processing you have Apachi Hado you have spark machine learning libraries algorithms are no longer secret right you have Apachi Mahal you have spark MLS as Debra said like they're standard algorithms collaborative filterings class regressions they're out there already there and data storage just like them they use Cassandra they use xbase see I can predict what stack they're using but what is really missing is something that would put all this stuff together and make them talk to each other nicely so that we developers data scientist can build something not only prototyping but something on production remember the architecture that they show you right like stumo points using a lot of tools putting them together working together is challenging and production IO you can just install it and everything will be in place but um what do I mean by prototyping and production let me give you an example so the textbook case classic recommended system right if you look up a tutorial machine learning tutorial Spar ml Hardo they will show you how to build a recommended system so basically they're showing you how to build a stumble a point well not the recommendation Discovery but the same thing they're telling you that you can build it with a few lines of code um so that's prototyping I'm going to show you how it works you have a mobile application right like stumo Point there's a mobile application and you have content so you want to build a recommended system or recommendation engine whatever you call it now you fit in the user Behavior data or the content into the engines and the engine will train the model with some algorithms and then you want to predict what contents user would like so now as I mentioned you don't need to build your algorithm from from scratch at least at the beginning like later on maybe you want to build something fancy but at least at the beginning there a lot of algorithms available spark ml Li is has been built in in prodiction IO so we just take them as an example if you go go to the website you will see a tutorial using spark mle build like recommendation system it looks really easy it looks so your mobile application connect to spark MLA got a predictive model and you get prediction wow so simple so why is dville app Point spending 14 years building this kind of stuff and de still working on it um let me show you the code for um this prototype okay um I'm really GL glad to be here in SF scholar so I don't need to explain the scholar syntax isn't it great like scholar is a great programming language we really love it so ready the first line of code is you read data from a file it can be a taxt file it can be hdfs as long as the data is well defined it you're fine so you just read the data and then what do you do you pass the data to an algorithm so in this case we use um Al as one of the Matrix factorization algorithm in integrated in um Spar ml lip so it's it's really easy you pass the data in the algorithm it gives you a protective model done with a protective model all you need to do is pass all the user Loop through all your users and make recommendation for them so let's say you have a thousand of users you look through them you would get like five recom recommended contents for each user so that's the end of the story machine learning prototyping and you have you happily have a recommended system what's wrong with it nothing wrong it's great as a prototype but moving beyond prototyping to production now we start to scratch the head um how do you deploy it as a service so that you can respond to Dynamic prediction quy oh it's back up okay so H how can you do that like let's say you want to recommend or discover five content to a user uh free content to another user or if you're in San Francisco right now tonight I want to recommend something to you it would be very different from the content that I want want to recommend to another user who is now currently in New York what over so it's like dynamic respon dynamic um recommendation so you need to deploy the engine as a service and also you need to you need to make the productive model persistent right like you can't just have a model look through all the users you need to make it persistent in the distributed environment and as I mentioned like there are a lot of great tools hbas spark um algorithm Library is um or other tools they're not great for being user friendly unfortunately like getting them to work together is challenging and there are a lot of other questions like how do you prepare the data um remember I mentioned prototyping they require the data to be in really well defined format but most of the time you're collecting data from multiple sources with different formats like this user clicked this button this user do some other behaviors um how can you collect the data in real time and eventually you can update the model with real or new data and also like from the software engineering perspective if you guys thinking about like really building recommendation system on production now you you're asking okay so where should I put my business Logics and how about like making it reusable modelize so that I can maintain it easily and if you have a background in web development mobile application development now you're asking where where is the separation of concept framework like we have the MVC for web development but where is the MVC for machine learning like we only have an algorithm generate model but where's the framework so I'm going to show you what I think it should looks like for machine learning to be on production it should be just two circles very easy one is the event server event server collect all types of data and there will be an engin engine represent the prediction engine the predictive model and it will be deployed as a surface so in real time it can be it can respond to data your mobile application will be sending the user behavioral data in real time to the event server the engine will be able to pull the data from the event server and build the model and be a service so that the mobile application will be able to send a realtime Dynamic cury to the engin and then the engent will be able to respond to the application with a Json Json respond and things like that and it should be able to deploy in a distributed environment that's what we want and that's what we want to provide to you everyone who is building machine learning applications so um again the goal of prediction IO is focusing on building and deploying machine learning things on applications on production and what's great about the stack the open source stack is that if you install prediction IO with one line of one line of command you get the Apache spark you get the xray you get the elastic um stack all completely available with one single line of installation so so if if you're starting a new project that's a great way to start it you get the latest technology um because we abstract all this great tools underneath so you can use it right away and you won't worry about scalability forever hopefully so um I talked about two components right two main components one is the event server the other one is the engine now let's talk about the event server because it's the most simple component event server is the one that collect data from your mobile application or from your website or from your variable device um after you install prediction IO launching an event event server is very easy you just run a command Pio event server and you got a event server um it's a very very flexible way to St um collect data and the data will be stored underneath in hbase in the events format so for example you want to collect events or actions a user one two three give a rating of five to an to product to a Content 100 you you just use the rest API or one of the many sdks provided by the community like note. JS SDK Ruby SDK um scholar SDK of course um and then you can collect the events to the event server it's it's it's a really generic way to to do it if you have used analytics analytical tools like Google analytics mix panel things like that it's really really similar and it's generic but are you ready for the fun part so um we're going to talk about building an engine in scholar so um a little bit of deep deep diving into the code um bear with us bear with me because um this is the MVC for machine learning in scholar so um we developed something called days d a s e is basically the MVC for machine learning but I find it sound much better than MVC right days you can pronounce it MVC you can't pronounce it um so d d stands for data data source data preparation which is responsible for preparing the data for training the model um a a is algorithm or algorithms so in an engine you can have more than one algorithm and you can evaluate them you can combine them together um is which is really a really common practice that you start with one simple algorithm and you want to evolve and test other algorithms so we provide a separation of concern you don't need to worry about merching the code was over and then s s stand for serving so this is where you put the Real Time Business Logics um it's a serving component component before you serve the prediction result you can add whatever business Logics to it E stand for evaluation So eventually you want some evaluation of your prediction results um so I'm going to talk about two functions of an engine so we we're not going to touch a point evaluation tonight because we don't have enough time but we can talk about um the two main functions that an engine can perform the first one is train a Deployable model the second one is after you deploy the model how does an engine respond to the realtime Cy from a mobile application part one train a predictive model now you can see that we separate the components into different classes data source perator algorithm and in each classes is you have the functions responsible for different things so let's assume I'm building a recommendation engin I defined this classes and then in on prediction AIS platform I just need to run the command Pio train what it does is it will trigger the r trining of data source which would pull data from the event server um and then prepare it to to make it like the training data format and once you return once you return the training data in the in the retraining function prediction IO will pass it on to the prepare of preparator automatically and if you familiar with spark you you you might have heard of rdd which is a distributed um data format so training data can be an rdd format which means that you can basically pass huge amount of data from from it but of course like most of the time training data um it it depends on your application some of them you can do it in single machine but most of the time you might want to do it in the distributor environment so it has been taken care of the rdd will be passed toep prepare of preparator class and prep prepare will take the training data and do whatever preparation you like so you can do the data cleansing here you can reformat the data here remove anything that you don't like here and you pass the returned prepared data so the function will just need to return the prepare data prediction ey will take it and automatically pass it to your algorithm classes to the train function of algorithm classes as mentioned you can have more than one algorithm but in this example it's just one algorithm algorithm one so training function of algorithm one class will take the prepared data and you can do whatever you like to build a predictive model and you you return the model a closer look at the retraining function of data source is it's very simple to the Prototype that you have seen basically it will use the event API provided by the event server of prediction IO um get the store of storage of it and then you find the events you like and then you reformat it to a format that is required by the algorithm and return the train training data model object sorry and another deeper look is um the train Class train function of algorithm one class again you remember the ALS function right in the Prototype um I've shown you before it's basically just the algorithm um function provided by ml Li so here you take the prepared data do whatever you like train the model and return the model prediction IO platform will help you to make the the model persistent and deployed it on as a service automatically so it will be in memory it will be distributed you don't need to worry about like how do I Reeve the model when I'm serving prediction in real time is confusing but you need to look at a source code a better way to understand it is a visual field of the an engine for the training train predictive model um process is the D components and a components data source data preparator and algorithms when you trigger the Pio train data source will pass the training data to data preparator data preparator will pass the prepared data to one or many algorithm classes that you have defined and in the algorithm classes they will produce models and prediction IO will store the model and make it Deployable that's training it it's the logic is very similar to what you're doing in a prototyping stage except it's it's a separation of concern so some of the data scientists can be working on one algorithm while another data engineer doing the data cleansing can be working on the data preparator or like if you want to make the data source reusable for another project you can just take the component put it on another project it's exactly like what MVC is doing for um web development so now you have an engine that is Deployable you have deployed a model what is a source code looks like in an engine that is responsible for responding to realtime cury when I say cury it means that the the input from your mobile application to the engine to request something in the recommendation recommendation engine scenario one of the possible ways is um I give you an user ID I tell you how many products I want to recommend to to this user ID and you return something to me right so the Json request um through rest a API would be probably be a user and a number number of product to take this Json all you need to do is to define a case class called cury in your engine code and obviously you define a variable user and you define a variable number and make it serializable prediction IO will automatically load the Json um data that is sent by the rest API through the RS API interface into an object of this case class and pass it to um the components that you have built and similarly to the output we want to Output a list of products um after prediction after we predicted what products we want to recommend to this user ID we want to pass a list of items with a score the score would probably be like the higher the score the more likely the user would like it so again you define a case class predicted result um Define an array which means that is a list of list of products basically make it serializable and of course you define what does it mean what is it in the array it is a string of item item id underscore so that's it um in in the engines all you need to do is return a predicted result object the platform will automatically convert it to a Json format and return it to the mobile application through the rest API so in terms of the code um we actually triggering the a component again but this time not in the um train function but the predict function let's say your mobile application create a cery through the rest interface the cury as mentioned the Json cury will be transformed to a cury object pass it to the predict function automatically along with the model that you have built before with the model with the query you can make prediction and you return the predicted result predicted result will be passed to surfing um the surf of serving why is that because you can have multiple algorithms so all the predicted results from multiple algorithms will be passed to the serf of serving class as a sequence and you can combine them you can compare them you can do whatever you like and here you might want to add some real time business Logics here as well like in a recommendation case if you don't want to show the products that are no longer available or like not in the inventory this is where you put the business Logics so you won't be massing up the the code of the algorithm play or the data preparation case and you can have different people working on different business Logics um a closer look to the algorithm predict function is take the query take the model the model remember we use um the alsos model of ml Li right you can use any any other um um machine learning libraries like deep learning library um Apachi Mahal any any kind of Library you can use it so take the model just use the build-in function predict um recommend products you return the result as a predicted result object and it will be passed to the serving so a summary of what's happening in a deployed engine responding to realtime prediction Cur is you trigger the algorithm class and you trigger the serving Class A cury will be pass to the algorithm class that you have defined it which contains the model and then you will return the predicted result to serving and serving will return a final predicted result objects which will be automatically converted to Jon respond and of course you can have multiple algorithms here so this is really beautiful like you you're not only prototyping machine learning but you have a framework you have a design pattern that is reusable modulized providing you separation of concern and that's really the new core features of um 0.8 of prediction IO so if you're starting a new project like you don't need to build everything from scratch this is the framework with spark with hbas with elas search with a separation of concern of for for your development and in the engine code what bring all this component Das e component together is a simple object called Factory um pointing to the fun pointing to it to all the classes and now you have already built a production Deployable recommendation engine in scholar running it on production in a distributed environment is very simple one line of code AS mention one line of command you can install the whole stack prediction IO spark ex Bas elastic search everything talking together nicely event server one command you start a event server that can collect almost any kind of event in a really scalable way build the engine have fun a scholar so you can look at a source code modified it change it and once you have got the engine you run Pio build to build the engine you run Pio train train the model you run Pio deploy deploy the model on production you can deploy multiple instances on large scale Custer and if you have new data coming in um in a bat mode all you need to do is run Pio train again and then deploy it it will automatically replace the old model with the new one um of course we're also working on the real time streaming part which is really sexy in in and I'm glad to hear that stum point is also looking at it we we um definitely love to get more feedback on it so this is what we envision in the production environment of machine learning Not only would you have one mobile application like stumble app point you might have mobile application website um email campaign basically whatever channels you're interacting with your users you collect all the data to a simple event server through rest API sdks and then under the event server within prediction IO equal system the platform you can build multiple engines for different type of prediction problems you can have one engine for recommendation on the front page another engine for discovery surrend dity on another part of front page one for the email campaign you can have fraud detection Trend prediction basically you can use different type of algorithm in this really flexible um framework and it's all again the code is given you the separation of concerns and it scalability has been taken care of automatically by the framework so if you like what you see if you think it's an exciting project the next step is don't worry about the complexity because we provide templates you don't need to write things from scratch we have templates for recommendation we have templates for classification we even have templates for some business with some business Logics like discovering similar products and things like that all written in scholar and you can just go to the gig Hub look at it um see how the Das components are working and subscribe to the newsletter on our website um if you're sitting here um not knowing scholar just coming for the food I strongly encourage you to learn scholar it's a beautiful language and finally it's a privilege to share prediction with you guys here we need more people who understand scholar um interested in machine learning to participate in in the project tell us things like from the small things like your documentation this here you need Improvement to contributing to the whole architecture to machine learning algorithms deep learning um stream gaming anything we really need help it's a very early stage product but at the same time right now it's powering hundreds of applications includ including the largest one of the largest Media company here um mainly like e-commerce large scale media um media and e-commerce companies so it's an exciting project focusing on production thank you questions um any questions yes make scalable um so the question is how do we make it scalable we depends heavily on Spar so um as as long as you f you following the rdd formats of a data transformation um we at scalable as spark um so basically um we we in in the early version we support hard do but now we believe that spark is more suitable for a machine learning um task it's not saying that we won't support hard um is basically portable you can port to hard as well couple of questions so how do you actually SC for online serving for example you have so do you take care of all the load balancing and everything also so the question is how do we make it scalable in terms of the online serving um prediction cury let's say like trillions of requests at the same time um I mentioned that you can deploy the engines the model to multiple servers so how do we do it um again spark is the spirit in in in the project um we love them we work with them so the model you built actually resid in in Spar in memory in the distributed custom and you can deploy engines on the same engines on multiple servers and we have a Custer on AWS where you can just use a few clicks you can launch the whole um distributed Custer if not after you deploy the engines on multiple servers you can use um op op tools like Chef or like other things to point it to different servers um engines to do M um low balancing so um it and and also we have a very lightweight um layer um handling the rest API interface um it's the spray IO um framework so it's it's very lightwe and it can serve a lot of request hi sorry yeah [Music] theolog data yeah um so the question is there a lot of Technology underneath like spark uh Sue Keepers and things like that do you need how much you need to understand those technology in order to use prediction IO um we spend tremendous amount of time making sure that all the components are talking together nicely um prediction iio is one of the few products nowaday certified on spar so um we're compatible with the latest Spock um specification um we make sure that the version that you're using um ml x Bas spark um elastic search and prediction IO server all talk together nicely um if you in install a clean installation but if you have existing like Cloud era um xas or things like that um we we're also happy to help to support as much as possible so one of the many one of the very main missions of prediction IO stack is we want to do the heavy lifting for Developers for data scientist like modeling finding smart data is fun it's a job that data scientists would love to do but like setting up Su Keepers making sure things are scalable um this is stuff that should be taken care of by software and also buy one maybe one or two expert in in a company but not a job of a data scientist we think thank [Applause] you