data.bythebay.io: Simon Chan, Building an A.I. Cloudy Sky: What We Learned at PredictionIO
so let's get started um Matt has done a great introduction so I can skip that so um basically I created um prediction with a great team of Engineers data scientist and we chose the open source approach very early on so um we have big components in the company which is open source prediction iio and is one of it's always one of the top five machine learning projects on on ghub and recently we joined Salesforce and I'm doing um product management there continue to build um AI machine learning Platforms in the company so um today I just want to share what we have learned at prediction IO and because that's a long time of reflection to think of what we have built how should we move forward and what were the challenges are so um the big thing the only one thing that I want to talk about today is the AI developer platform dynamer um ranging so there are two extreme one is on the Simplicity side like think about the platform for developers to build predictive applications um what are some of the Simple Solutions out there it would be probably Google Vision a Pi um stiff science basically very vertically focused single domain but as from the development perspective if that is the problem you want to solve you just plug in your data and you get prediction out sometimes they have public data so you don't even need to have your own data like image moderations Services by um companies you don't need a huge data set to train the model because they use public data to train it already the limitation is it's not flexible you can't change such a system from image moderation to fraud detection image recognition system is image recognition system so that's on the very simple side of developer platform developer AI platform On The Other Extreme which is kind of the path prediction IO has been taking so far um we believe that we want to give developers the flexibility to build whatever applications they want so if you you see use cases on prediction iio for recommendation that all kinds of recommendations like wine recommendations movie recommendations product recommendations video recommendations fraud detection Trend prediction natural language processing video recognition someone's thinking about using prediction AIS for like cancer research stuff like that so it's like so flexible you can build all kind of stuff but what's the limitation the limitation is Simplicity even though we say like prediction out is already making a a big achievement to make machine learning development so much easier it's still very difficult to learn that you need to learn the DD architecture you need to actually write code uh we start prediction out with scholar but we support Java but still you need to learn how to use like for example if you talking about um distributed processing you need to kind of know how to use write in spark how to write in scolar Java stuff like that and it's not a plug and play model so there are two extreme and on the flexibility of course you have r side kit learn spark whatever you can build anything you like that's the flexibility without Simplicity on the other side Simplicity without flexibility so there's no solution on the market is really about how we want to balance it and the reason why Simplicity and flexibility cannot come together it's very simple it's because every prediction problem is unique set whom set me because I've seen a lot of applications used using prediction and IO they are totally different everyone is unique so today in this short period of time um if we want to talk about all the challenges we have faced we it might probably take like 40 days instead of 40 minutes so I just focus on one Dynam today every prediction problem is unique so the Dynam is when we developing a machine learning platform for developers should it focus on making it simple or should it focus on making it flexible and because every prediction problem is unique we need customization right so let's say we we're building a platform for various companies various applications various use case in order for the platforms to customize the predictive applications there mainly fre Solutions one is automations people nowadays start talking about AI for AI which is very exciting like AI is great but it's so complicated so we build another AI to learn how to customize AI for everyone one that's AI for AI it's it's a great approach it might eventually solve the customization and simplicity problem but that's a very complicated solution um in the Middle Ground between Simplicity and flexibility is um graphical user interface right so um would that be a solution where you can just drag and drop so that you can build classification solution for all types of applications so kind of a tradeoff for example you just focus on classification it's a limited scope it's not extremely flexible but then it's easy to use because it can provide you with a graphical user interface track and drop and build it um Google prediction API Amazon ml is kind of in that approach um they have pre-build um Solutions like classification regression recommendations and theoretically you can just plug in all your like any types of applications into that framework and use it another extreme is mentioned is custom code and scripts so data breaks they have a notebook you can do all kind of data visualization analysis you can build machine learning so providing a framework so developers can write custom codes solve every problem that's on the flexibility extreme but we you sacrificing simplicity so let's be more concrete about it um I'm going to go through 10 key steps and in each steps there are a lot of choices a developer would need to make in order to custom customiz the predictive applications in other words the machine learning platforms need to support customization in this 10 main steps and I'm going to use three examples um just to illustrate to to basically is a kind of like a interactive thinking process to think about what would you do if you're building a machine learning platforms um and to give people an idea about machine learning platforms we're not talk about building machine learning solution just for one applications it's actually a platform for any applications so now here is here are three examples one is e-commerce product recommendations the classic one another one is um subscription based model like Netflix or like Amazon Prime they want to predict turn right like uh which customers which subscribers going to turn um leaving the leaving the plan so they can spam you with all disc SC stuff like that uh the third example is social network um spam data social networks means um one of the key elements of social network is everyone can contribute content and spam detection is a big one so the challenge there is how do we design a platforms um so that we can support various kinds of applications and here are three examples so in the first stage many people many people jumped right into algorithms um so when you're talking about spam detections the people email me which algorithm should I use or when they're doing Trend prediction which which algorithm would it be deep learning would it be logistic regressions would it be everything most people jump right to algorithms but there's actually complications about defining the prediction problem be clear about the goal here are a lot of questions for example what is the business goal um just take e-commerce product recommendation as an example um would your goal be improving the user experience sometimes it's really just about discovering great products that you like um and what do you mean by products that you like is also very controversial should I recommend products that you are going to buy anyway or should I be recommending products that you may buy but you may not buy um kind of like some surprise surity there so what is the business goal or would it be about maximizing Revenue spam detection the business goal is kind of um clear is just we want to detect unwanted content um so what's the input cury would the cury involve anything Dynamic so for example in the subscription turn prediction case um a simple way to think about it is the input of the prediction is just um the user ID right I want to know if this user is going to turn I want to know that user is going to CH so that's a basic way to think about it but you can also add other colors on top of it so let's say this user has just referred a friend so this kind of like a dynamic c um user ID one just refer a friend to this um applications what's the chance of chant this user just sent free complains emails what's the chance to change so the input quy matters when you define a prediction problem and how do we enable developers to Define this customization on the platform would it be like Automation in this case very difficult to automate the business goal make the decision for you so probably it would be like user interface or probably it would be um custom code but again custom code is flexible but it's not simple and there all different thing what what do you mean by a good predictions like turn prediction what what what does that even mean um let's say Matt signed up for um Netflix he it we predict that he's going to leave after 10 years is it a turn or is it like not a turn or someone else like is sign up and leave after the free trial is it a trend or is it not a trend is actually not a Pay customer so this is actually all the business decisions the developers needs to understand before choosing the algorithms the second point is still not algorithm unfortunately um after you design the prediction problem you need to decide how you're going to show the prediction results um at the end of the day we data scientists have a lot of ideas about evaluating where um the prediction application is working whether it's better than all our competitors everyone say they have the better algorithms better than any other algorithms in the world at the end of it's really like human perceptions when you make a predictions you show it on your mobile app you show it on your website you send report to customers do they think it's accurate do they think it's a intelligent system so um a classic example is the Toleration to um predict bad predictions which is related to the second point should we present the prediction result in a decisive way or is a suggestive way right um I guess like everyone knows here prediction is still not 100% accurate for example um spam detection probably you want to eliminate spam but at the same time the Toleration for bad predictions the tolerations to say you upload gr content but I flat it as spam and I delete it um Toleration to that in most applications is very low so should this prediction be suggestive or predict um suggestive or decisive decisive mean like okay um we predict this 65% chance is a spam we delete it or would we just create a list send it to a moderator say we kind of think that it might be a Spam do you want to take a look at it um why decide why deciding on the presentation matters because it affects how you build your model um um for example in this case in the spam case you probably don't want a classification to just say yes and no maybe you want to make it like a regression to say is like 100 score um is a spam or 50 score is is a spam and so on so it depends on how you design the presentations and to make it to to to to build models basically and the challenge again facing in the past few years is this is so flexible that so many choices developers can make how how do we build a platforms to support it um again there's a lot of things like human interactions scoring for example when we predicting um the turn would it be a score like U five is 10 or would it be like a ranking he has a higher chance to turn than you so that the business people can prioritize the support prioritize the discount and stuff like that would it be a graph Custer different users this is the highrisk user this is low risk users so there are so many things you can decide now if you're customing is about customing it's not about regression classification anymore so you get my point about importancy of deciding on presentation before you actually work on the data science cool stuff the third point is now it's cool is importing free form data um most of us when we're in Academia we work at very cool big um data set like uh movie L Min like for they're all so well format like user ID movie ID events this is the titles but how about the missing data and all this stuff in the real world I never see people building applications for such a beautiful data set in in real life we're ping thousands of applications and I I just see that data source is always a problem as a first step so a few examples of different data types um that how we categorize when we build the event server for prediction iio uh one is user user is always important and users you have user attributes like age gender location whatever um U whatever information you can gather basically items item is a very generic terms but in this world or if you're coming from programming object oriented backgrounds everything is an object so everything is an item so a movie is an item book is an item laptop is an item um you can even say your friends if you're building a social graph your friends can be an item so what's the relationship between items what's the attributes of that activities most things happen in events activity based when you're interacting with an with a web or mobile apps right like you click this button you purchase something you browse something um you sign up something you so there's um activities and usually we link users with activities and items items would have the content so this is the article you you read and this is the content of the item um change lock is when when we when we're talking about more production use cases change logs become um important because most of the time when the huge amount of data is residing in like um orle database or whatever database myle database company using they need to extract the data form a standard data set and we can run modeling on top of it but people don't want to generate that huge whole data set every time you retrain the model so a lot of the case in Productions they want to send update so what's the update from last time to this time so the system would also need to handle this kind of change lock and integrate it to the data set um but at the end of the day at this stage we just estimating which is a fancy word for guessing we're just guessing what algorithms want but you do have some idea because you kind of know the prediction problem you're solving you kind of know what you want to present and now from your Arco database from your whatever database um meato whatever data warehouse you kind of guess what kind of data you need for the modeling and you extract them from it um a lot of machine learning algorithms libraries or tools they don't handle this um you might just need to use um hbas or other tools to handle it by yourself but at prediction AIS we want to create a end to endend solution so we have something called a event server it's still on the very flexible end meaning it's not very simple um so there are a lot of space to improve as a machine learning platform how do we support this four types of data um easily in a simple sense but also flexible and and there are different way to transfer data um sometimes people like transactional like you set a JavaScript on the website so whenever someone click a button you send the data to the machine learning platform immediately um most of the time people already have this kind of database or they're using like Google analytics or like mix panel to collect all this event data so they want to send those data in batch to your machine learning platforms um batch frequency is important the reason why it's important is um especially in turn prediction for example there's a Time issues here how fast you want to predict turn if you're updating the data every week um you might miss out a lot of and after you retrain the model it might be too late for you to detect a lot of CH turns already so when a developer using machine learning platforms to build this kind of applications a lot of the time he or she would need to talk to the data guys the business guys say like what's the batch frequency and don't forget data sity check it is extremely important 90% of the cases um that we we saw using um asking for support from prediction IO um the data input because of the complexity of this that's usually some broken stuff in the data pipeline to send data from wherever they have to the machine learning server so I originally I was thinking hey this is nothing about machine learning I I don't need to deal with it but at the end of the day if you building a machine learning platform handling practical stuff like keep making sure the data is actually importing correctly is extremely important the fourth step is so now we have a problem we have know how to present it we kind of get the data um from data we need to generate features which is again most of the solution is on the flexible end right now um you just need to write your own script in Python using Jupiter whatever look at it see what features relevant do whatever you like um if we want to create a simple platforms and we have a dream of making prediction out also like a cloud service so every developer without knowing the infrastructure can just click drops and use it so how do we build a tools or software interface that can support feature generation feature selection that's extremely challenging because there's so many way of transformation first of all you need to transform qu Quant quantitative to numerics because most algorithms they they don't know if like the if if it's a tax based categories the algorithms cannot handle that so um a lot of the time you need to transfer to numbers and the number has weights right and you need to normalize that um are we talk about aggregating the data of the last week or we talk about aggregating the data of the last month like which way is better for recommendation product recommendation which one is better for um TR prediction and which one is better for spam detections um these are all decisions the developer would need to decide and experiment and giving a platform to allow developers to set all this different um choices and experiment that would would be very beneficial to the community so um again today I'm not trying to give you an answer like we I'm not going to sell you a great product that we have built that solve all this problem but rather I really want to encourage the community to think more about all this challenges so more people work on it PR again prediction now is open source that also all kinds of Open Source projects in the world that need help on many of this Quest problems um the fifth point the five the fifth step is um specifi label um if you're familiar with machine learning algorithms you probably know that in order to train a productive model you need a matrix of features which um represent basically the input and you also need labels labels means something you want to prodict so example features can be a customers um age gender for example and probably not the best example in the world but um but the label can means um the user is going to um the user churn um that's the label or the user has not churn in in 30 days so that's the label how do we construct the label is very challenging um again is every business is unique um and you also need to understand what you're actually trying to predict um and there's always delayed in the feedback as well so turn is again the very challenging and classic case um many business want to make Trend predictions without defining how long the period is to Define it is is a turn um so when we're defining specifying label you have all the challenges you face when you're doing feature selections plus the delayed of feedback because things most of the time things happen happen later and in terms of like reliability of explicit opinions um spam would be a classic example there's a content on the social network someone some guy flatted as a Spam is it a label is does it mean that that content is actually spam U we don't know is is it reliable that's explicit opinion but is that reliable um how about two people flat a Spam is it reliable or free people or do you need to rely on The credibility of the people who flighted as a Spam so you can see how application specific how business decision specific it is it's not a technical problem it's more about the platforms we need to support this kind of business Logics how people Define um the labels the sixth point is very interesting um every time we present so today I'm not presenting prediction iio but every time we present prediction I there must be a questions do you support real time so I add this point just to answer this questions the same word can means many different things um different scenarios right um in a non realtime case you update the model you update a predictive model in batch and you send a batch curious cous means like I want to predict um whether this list of customers is going to turn I want to predict this list of content is this spam um so nothing real time here but real time can also means no we're not updating the model in real time we still updating the spam detective model in batch but the cury is real time so um for example um the input of the spam detection is about the text but it's also about the location you post the content so now you can't model the location of the person who post it in the model because it's so Dynamic I can post it here I can go outside and post that I can take a train go down P out and post it so it's it is real time right it's basically on the spot like now I posted an article is it a Spam this is a real time cury but it doesn't means that we need to update the model in real time and of course there's other case where we actually need to update a model in real time for example like if you're modeling about um stock trading stock price there's so many informations you have new labels you have new features um continuously you want a streaming system to keep updating the model and sometimes it's a hybrid of that so um a lot of cases I saw so far is on the second case where the cury is real time but the model it doesn't need to be real time so it's just something to think about but then if the model is update in bash um how frequently should we update it um the model the the machine learning platforms probably need to support different retraining schedule like spam detection we might not need to train a model every hour recommendation maybe every day just for an example um subscription based model Maybe week or month something like that so it's not like a one platform I set it the model will be trained every day and it's fine it's like if you're building a platform that truly want to support multiple applications need this kind of features this kind of flexibility in the platform um so so the seventh step is um finding the right model finally we go to what people always talk about the model the algorithms the hyper parameter we find the best algorithms finally there a seven stuff and now we're here um this one is very well discussed in the industry in the machine learning platforms we need to let the developer choose is it classification regressions recommendations or something else like customing what whatever supervised unsupervised so um ideally um the more types of recommendations you support I mean the more types of prediction you support the more flexible your platform is um but again if one platform trying to support multiple applications the definations of overfitting under fitting is is very very different sometimes you have an algorithms you have a set of parameters it works super well for a large e-commerce companies the same thing applied it to a smaller e-commerce companies in exactly same industry it won't work like you might overfit the model or you might underfit the model so this is actually where I think possible the possibly the AI for AI part will kick this like how do you automatically find tune the hyper parameters for this specific client I mean this specific um applications how do you automatically fine-tune it for another one um we really want to eliminate the needs of tuning all the things manually in the flexible side because kind of think that Simplicity is very important here even though um there's still a lot of challenges for example co-art um co-stars meaning like there's something new you have never learned it before so what's the default mode to handle it um recommendation case there's a new customer coming in no previous knowledge um no information have not registered an account nothing um would the default mode be recommending the top five products overall or would it be a random so you want to learn um his or her behavior as quick as possible these are all um business decisions but again when we saying we provideed platforms for people to build stuff they would ask for all this flexibility like I want to do this I want to do that oh you only support this it's not a platform I want I'll bu it by myself so you you can see a chance we need to support a lot of applications at same time um the more meanings is not simple um the eth step is set evaluation metrics so um it is actually related to the AI for AI points in order for machine to learn that we already have a good model you need to tell the machine what it mean by a good model in a quantitative way um in evaluation there are many broadly speaking two types like online and offline online means is it's life like you're not using historical data to see if the model is good offline means I'm using historical data to see if if the model is good um a lot of the time let's just take recommendation as an example um people would just look at research paper and say oh rmsd is a great uh metrics every competitions every research paper use it like it it was like way back then like few years ago that that's what's happening um but RBS is a very bad metric to represent most business goals so that's why remember in the first slide or second slide mentioned understanding the business goal is extremely important if you don't understand the business goal you can't set the right evaluation metrics if you can't set the right evaluation metrics you can't automate your modeling steps and that's why understanding the business goal is so important um running out of time but again like um one example is recommendation what does it mean by by good recommendations um the or like subscription what does it mean by good turn predictions the business goal can be extending the subscription so I want to identify all people who are going to turn so I can reduce the turn turn overall turn rate that can be one business goal or my goal can just be find out who is going to turn and with the prioritize them it it's a total total opposite and in your valuation metrics you need to reflect this business decisions um one one point I also want to mention is how do you present the result to everyone it's super super important um I've seen data science team they come up with really robust metrics which that only they understand and they finish all the whole projects and now they go to the business side or the customer side and they say hey we have the best prediction in the world but what does that mean like what does that score mean 0.35 and 0.37 is that is that better is it is it is it good so there's a lot of um issues about evaluation metrics you need one that reflects the business goal and you also need another set of metrics that is human understandable um the ninth step is surf predictions which um at prediction IO we mainly see two cases realtime scoring and batch scoring batch scoring essentially means we have a model we do a batch of prediction for millions of customers and we need to find a way to store the result um real time scoring of course is more flexible one user at a time and we do a realtime calculation but it's more computationally expensive so um it depends the approach you choose but the note is really um in most cases I see that we need to add a business Logics layer on top of it um not going through all three example but again recommendations means hey the product is no longer available why are you recommending that product it's it's it's so stupid like imagine you go to a website it recommend you free products you click into it and it say out of stock so it's a perfect recommendation maybe but it's out of stock so you need to add real-time business Logics on top of your predictive model most of the time is it the job of the machine learning platform is it a job of the applications the line is bur depends how you see what is mean by Machine learning platform final step thanks for going through all this 10 steps um is a condensed experience in this challenging World um keep monitoring a lot of the efforts in the community is about build one model run it one time and show great result on production though things break like you need real time alert data is not coming in the model is working yesterday but next month um is basically flatting all good contents to to to spam like how do you have a mechanism for your users or the system itself to tell you hey your model is not working anymore like you don't want to you don't want to have a data scientist to keep running evaluation manually every day that's why you need an automated way to monitor it um real time alert report and a lot of the time we we see people ask us to integrate with existing monitoring platform because most companies they already have a production ready um monitoring system um this is a summary um the whole point is to scare you guys um it's very challenging to build a machine learning platforms and there's a lot of challenge it's very exciting um but I believe that to balance the Simplicity and flexibility the future is really to automate AI which is a very exciting things and we're building a new machine learning platforms um prediction IO we're continuing support it is not we're not killing the project after the Acquisitions we're putting more resources on it and we need people to to help join our journey and we're hiring thank you so do we still have time for questions yes we have we could probably two or three questions okay can you speak to some of the business challenges in building something like prodat sure so the question is um can I share some business challenge of building business like prediction IO um one of the first challenge is it's really this Dynamic if we make it simple it means that we're not supporting many business cases and when a team of data scientists they say hey this platform we have 10 requirements it only support five of them they tend to build a new platform by themselves at least that's the first respond um we see cases where after nine months they come back and say hey we are not building it um inide we want to use use prediction I want to cont back but it is such an exciting feel everyone's want to build from scratch and there are so many tools available there are so many random questions why are you using Spar and not Flink um instead of thinking about the Practical 10 questions so I guess like a lot of hpes um we need to filter noise to see this is actually valuable to your business and we need to communicate that I think that's the big challenge in building a business on machine learning platform 10 things you mention how you see prediction I address so for the 10 points um I mentioned how do I see prediction I address them we are still very very early stage um we have event server to kind of address the issues of like you have multiple source of data you can input it um the reason why we chose to open source it not sure if it's a great business decision is because we go for the flexibility size but now the dymer start to kick it to say hey this is so flexible but it's so difficult to use um so we kind of have different placeholder for the 10 problems but I won't say we have to solved it like we don't have a great graphical user interface to for you to drag and drop to choose your features that would be some great open source contributions or like it might be something we want to build in prediction a project or outside prediction project so thanks thanks very much Simon thank you thanks for a great [Applause] talk