Cognifest NYC 2017: Simon Chan, The Evolution of Machine Learning Platforms

Cognifest NYC 2017: Simon Chan, The Evolution of Machine Learning Platforms

Recording: Cognifest NYC 2017: Simon Chan, The Evolution of Machine Learning Platforms

[Music] thanks for having me here I'm so glad to come to New York for this presentation and thank you mark and Dennis setting up this stage to talk about how we can make money in the AI time and how we build applications essentially for one specific domain conversations how we build a really production-ready applications that people can use daily I spent the last 13 15 years in software development AI space without knowing it without noticing it I first started consumer applications is kind of like newest recommendation applications where you can share news that you like to read with your friends and of course with machine learning try to predict what you like to read and predict what you want to share and after that there's some other startups in a space try to build teams try to build products that create this kind of cool applications at that time it wasn't that cool to call the applications ai ai was kind of like long-term were like invented 50 years ago in research ai and then it somehow hit there's no period where no one's talked about AI until recently everyone's talked about machine learning and AI applications so now we're again but the idea is that we want to build smart applications and when I see all the teams all the products that we try to build it was hard it was really hard and then I have a fault one day when we build search when we build database application where you just store your data and you want to search some information what do you do you pick up some algorithm and you build a whole like data storage file system you create a search indexing technology and then you pick you build 100 people team to build a database server before you build your website no you didn't do that you just download the database server and then you use it like it's just like five minutes work right and there's a company going IPO new york-based company it was IPO actually recently database company which is great so the idea came where why do we need to do it why do we need to recreate a whole AI of machine learning server from scratch for every single project so we invented prediction IO an open-source machine learning server obviously we try to mimic the successful story of a lot of open source database businesses right we created it open source and we hope everyone contributed to it and we create something that is useful for production to build smart applications and we made it frictional became one of the most popular machine learning projects and it's used by thousands of companies and the company was acquired by Salesforce and I'm leading the Salesforce einstein platform projects on the product site right now so Einstein is the AI initiative and Einstein platform we power all the AI applications in different verticals in Salesforce so that's the background and today I want to share just one thing the journey given all these years starting from scratch trying to create a server create a platform and now running Einstein every time we make one baby step we thought ok we're done this is the one problem we need to solve and turns out is a big journey just like baby learning how to crawl learning how to walk then learning running Yoshi doesn't know what's coming up next but then if you come to the long life journey to pick up new skills and then you move forward so that's what I'm going to talk about and just a very quick background machine learning is about getting data set mark talks about having great training set data set that's your goal in mind if you have some proprietary labeled data set you can train the really powerful machine learning model or deep learning model and then you can classify data and then you can have data that you want to classify it and you make prediction it is just that simple that's machine learning that's teaching the computer how to learn to predict which translate technically to free lines of code it's not a technical talk but I think this reliance is very useful and surprisingly the previous speaker talked about scholar how cool it is the team built-in scholar and it looks like a scholar language so the first line is read from a data set second line is you build a prediction model with one of the algorithms existing in the market and then now you have a model you run the full loop you try to make predictions right for everyone here we want to predict what you like so that's the most basic machine learning or AI or recommendations applications that you see in textbooks or in any tutorial so you download any machine learning tools that's probably the first tutorial you want to learn from and we feel like ok that's it we know how to build AI now that's the real thing first step building applications real application learning application the first thing we learned is the biggest challenge is how to be stable now we download machine learning libraries that tons of them now say like you you don't really need to worry about not having enough algorithms there a lot of open source not open source libraries that you can use but how do you make it stable three things I learned of course there are many things but three things the most important one is garbage in garbage out your data is super important and having a healthy stable data pipelines really really important I can give you an example we have a teams back then built system for customer and then we figure out the parameters we figure out everything is is fine and then we launched it on production what's next classic scenario right the next day data is not flowing in or some crappy data flowing in a system the model is corrupt the beauty or the interesting thing about machine learning is it's a life system it's not used playing something static and it kept it like that it's a machine learning system that keeps learning from new data meaning you need to deal with new data meaning unless yours you you want to dedicate people to look at every single new transactions of data going into the system it is automated you better have a really robust system to make sure that you're pulling the right data do a sanity check make sure you're not pulling in the wrong data on corrupted data make sure you're actually processing the data run some statistics and metrics to monetary system 20 24 hours a day systematically make sure you're spoiling the right data because you don't know what data you will need in the future make sure you handle problems like real-time changing data as well as batch update of the data so turns out when we think we're building machine learning applications we're building the very robust data pipeline so that's the learning number one learning number two if you guys software developers how many of you actually use the software release process like you have a development environment test environment really release and RAM and production environment so if you all have this release process for software you also need that for machine learning that's not what you see in tutorial in fact in text book but remember your models keep updating every day or every hour or even in real-time sometimes if your models keeps updating and you're serving your production customers imagine if you're a bank you have four rigs model you were determining whether you want to loan some money to your customers and this is the model that's affects people's life where the people can buy a car where people can buy a home and then if you're not monitoring the release process you're not really that responsible right you monitor your software release cycle we have to monitor your model as well so we see more and more companies coming up and provide model management system this is a great area that I believe that we need more companies most solutions in it how do you monitor how do you manage the model process if something goes wrong one button you need to reverse it to the old versions and you also want to keep track of a number of things the source code that has been changed the data snapshot that has been changed and whether the algorithm has been changed so you can debug it just like any regular software development process and the third thing is machine learning models or AI algorithms often time gives you a score but you don't show the score to customers most of the times like if you're building something to recommend mosaics to people you don't say hey this music score 0.56 and that's zero point nine five that doesn't make sense and usually we need to combine score with real time context you don't want to sell the product that no longer exists and if you have let's go to San Francisco you want to see the mids up the events that's available in San Francisco and not in New York so combining real time contacts into the AI applications and make that scalable that is important all of the time we focus on the scalability of model building how many GPUs do you have how do you process the data but in real time when the QE hits your system and when hundred thousands people using a system in real time it's not just the score but also the layer combining with the score so that's the hot lessons we learned in the process three things now we have a really cool applications on productions after solving all the stability issues what's next people are greedy even though we try to focus on solving one problem when we work with customers when they see some success in the email campaign it becomes smarter they want to make other parts of the business smarter there are many many places you can apply it in your business sales marketing service security analytics HR finance everywhere then people will say pay we really want to build moldable applications we have the same data source or we have some external extra data we won't add into it but at the other day we want to build many applications cool just come the project and rebuild everything no that's not what you do in in a software development firm right unless you're doing prototype that might make sense but we really want something that is reusable so that's the next stage after you can crawl you mix things stable you start to walk is to make the development of AI reusable fonts is a little bit small but let me read it out for you as a machine-learning developer that new things I want reusability and talk ability of components less code less boilerplate maintainability of multiple applications the principle of not repeating myself yourself very very creative very important statements which you have seen in the last 100 years already essentially if you if you have been building software that's the problem is provided by software development framework like if you use Python it's die jingo if you use javascript that might be like note jr. as we have is actually any software development framework provides you with this software design pattern reusability and we just need the same thing in machine learning development so take a deep breath and then we look at every project we created there's some common patterns we discovered retraining data from the sauce ain't training data is very important about machine learning model prepare the data so you need to extract information from your data like what if was the features if your data raw data is an address maybe the distance is the real data that you want to put into the model maybe it's something else trainer model find the best model now you have a model you need to serve what we call Curie can actually tell me if this application is the fraud or not that's a Curie and then you want to surf the prediction yes yes maybe no or like a level of it how do you wanna serve it at the end of the day you also want to evaluate the model as well right 99 percent of the projects machine learning projects AI projects follow this pattern so cool just put some framework around this and a prediction I'll open source by the way we after the acquisitions of company we have donated the open source part of prediction IOT Apache so you can use open source Apache friction I'll just like you use Apache spark Apache Hadoop so that's the MVC Model View controller equivalent for machine learning that we have created the kind of training cheating because these stand for two words this sauce dat evaporator is d a is algorithm or algorithms as is serving is the real-time logics serving layer and E is the evaluator so in every single component of that you can reuse them let's say you're building simple example recommendation again you're building one recommendation engines in your email one recommendation engine on your web front page in the email you might just want to show free latest products and recommend them to your customers on the front page that might be more personalized based on what they have purchased before and then you personalize that so one with time one without time you can share the same data source data set you don't need to rewrite all this stuff right you're reading from the same data you might be using a different algorithms or you might be using some different real time logics this create the separation of concerns where maybe one of your teams can work in the data preparation other teams or multiple data scientist or data science team can create different algorithms and later on you can compare to see which one is the best some team might be good at evaluation statistically see how good engines performing so this creative framework this is just one example of framework and prediction I'll pioneer to that but I believe does and area where if you guys looking for area to participate contribute we believe that we need more machine learning development framework like that there's post and cons for this model we created a prediction I'll very strict very clean but it's not completely flexible so just like software development framework they're more different flavors for different projects I believe that more people more companies will create different machine learning frameworks in the future so that's pretty an IO one server you can create multiple engines same data source and the results of that is very powerful now people the community if you are expert in sentiment analysis you can create an sentiment analysis template with all these components and share that open source other companies they can pick it up just modify maybe the data source part or maybe add it with their own algorithms and then they can customize it if they don't want to do any of this they can just download your template and use it as this so a prediction IO that's vibrant open-source community with ready to use applications that you can download there are many many other things you can make it reusable feature stores metrics how do you run experiment algorithms and many many mores and I do hope to see more collaborations among different communities either free open-source or maybe through some galleries or collaboration I I think that that's the future so you know how to crawl you know how to walk next thing is really running and that is very interesting for me because when several years ago when we were creating prediction L we thought oh that's the perfect model now we make it stable now we make things reusable that's that's the journey not until I saw companies like Salesforce where they have 100,000 customers and we see the need of automation why simple math questions how many data scientists do you need when you need to build custom model for one customer how many people the data cell in his company is 150 people under something people right then how many people would they need if they need to build not just one extra layer applications but maybe 10 different external applications thousand five hundred how about you want to build how about if your company is that you need to serve 200,000 unique customers running different prisoners or having different business problems the answer is not scalable so there really a lot of processes that labor-intensive in the AI development process and that's why then is kept kept saying like their company is still small even though they have over a hundred people like pre-processed data collect the data pre-processed data select appropriate features choose to appropriate algorithms tune the model parameters after you to name all parameters you need to tune the models and then evaluate the results every single steps are many more of them are interestingly we always say AI replace people but every single step is labor-intensive we need a lot of people in that it's just isn't scalable think about companies like Salesforce when we launch predictive lead scoring just an application that predict who you should call first in the first ring in your day we we have small companies like a flower shops we have large companies we have like telecom company financial company every single one different data set unique data different size of things they need different things how do we have so many teams to clean the data to choose the algorithms for every one of them we can't so that's the beginning of Auto ml where we call automated machine learning the idea is actually very simple each customer has unique data set and then for every one of them we try different methods to process stater tried out different algorithms and automatically find the best one so that's the oversimplified idea behind Auto ml of course it involves a lot of statistic mathematics models to make it more efficient because unless you have unlimited processing power this is very instead of human resources intensive is now computing process intensive so we're changing the human scalable scalability problem into a mathematical problem how do we solve this problem efficiently mathematically and this is a super exciting research area so if you're into the math in just statistic we really I think the industry really need more more people in this area we need a lot of solutions and maybe startups maybe products in this kind of area so think about that if you're the kind of math mind if machine learning is about teaching computer to learn automated machine learning is about teaching the computer to learn how to learn one last things you know how to walk you know how to run everyone has the same dream when they were kid looking at the airplane and they want to fly and by wanting to fly by flying I actually means you give the flexibility you give the freedom to your customers let them build on your platform mass version Watson is a platform it's not just one product for one solution it's a platform you can build on top of it so the key word is be self soft which actually built on top of period steps right of course we don't expect every companies to have sophisticated team's resources to build like a huge data science team but with the technology like robust data pipeline with automated ml we're making it very easy for every company or even every individual to build AI applications several examples the different way to do it it's actually not an easy solution a prediction I we tried year years well sounds like a long time but actually we did spend several years trying out different models to see how we can enable developers to build AI one is the API the new words API as people say it at Einstein for example we launched Einstein AI you can create settlement analysis vision detection with things like that with the API as I suggest it's very developer friendly is you just input your data select the model and then you extract predictions you don't even need to write the modeling code that's one solution but some people might want something more flexible or like easy easier to use than API then I would want something point-and-click write official at the very early version of prediction IO we have something like this if you need to build a recommender system you don't need to write a single line of code you just point and click that's what I want that's the data I need and then you can create prediction and of course some developers they really really want extreme flexibility and nothing is more flexible than giving them access to your source code to your github make the project open-source the thing I really really want to talk about is this line one hand is simple one and it's flexible when we create a product when we make it super super simple what talk to customers talk to developers most of the time they ask you hey but how can i integrate my own our model into it this is so simple but how about I want advanced features and then we make it very flexible everyone can create their own custom algorithms and then there will be other groups of people say ok that's great but I want to build something in three minutes how can I do it so the learning I got is actually there's no perfect solution which is the beauty of our technology world there are so many solutions there's no one-size-fits-all solution so at the end at prediction ah we took the very I would say interesting approach we make it super flexible with the design patterns so you can create anything you like but we also create a template gallery ready to use engines built on top of the flexible system so people who want to use it as is they can just download it and use it data scientists if they want the TWiT algorithms if they want to create their own evaluation metrics look at the source code is open source you can customize it and share that with the public that's the approach we took at them at prediction IO and for Salesforce we have very quite different customers right business customers so we also have a range of products it's not one product for everyone we have a range of products on Einstein platform for them but the learning is really even though it's a hard problem give them a platform every time every time I see very impressive successful cases created by our customers they always exceed our imagination so that's what I wanted to share crawling walking running and flying that's the journey not sure where you guys are at but I still feel that there are a lot of opportunities for developers for startups for companies to participate to disrupt this market thank you [Music] so I had a question so I about your probably hope I had a question about your Auto ml point so actually two questions one is just if Einstein dot a I actually offers an auto ml API and the other pieces if you could talk a little bit about how you guys went about doing it is it just purely grid search did you try to try to turn neural network to do it did you do like how do you optimize the datasets as you apply different models etc to it we actually do I say Donny is the combinations of many services so we have wishes we have so sometimes there's pre-built tomorrow you don't need to retrain your pre-trained already and we also allow customers to upload a common data set so there are a lot of interesting cases where we want to recognize most in the refrigerator so they can upload their own data set as well what one more interesting example sentiment analysis which usually when you say it's a hot product it's positive right yeah once it is hot Einstein is half the products a hot but that's one customer there they sell air conditioner so your property your air conditioner is the hottest on the market that has actually another positive Saturday that's quite the opposite so we allow customers to follow the proprietary or data to retrain the model and the system automatically trained a model for you so you don't need to select to the model training model so yes its automated already and in the broader sense in Einstein platform memo is not just research we need as I said we need that practical we just cannot afford to build a unique little scoring model of personalization company so we're running we spend a lot of resources and will continue to research and develop this area so we're looking for more people in that area as well thank you [Applause] sorry you think yes so the question is that I show a slight different template yeah so if you go to Apache prediction I'll there's a link called template template gallery a bunch of them we do not list all of them because we realize it's a couple projects they're different people building stuff but you can just a quick question so as we with actually using business you have those false positive or negative known as type 1 type 2 arrows quite high remember then is talked about low accuracy accuracy I think and more scientifically what's your point like true positive or negative that is a hard problem in business we can use very sophisticated mathematical metrics to say you're OLC spore is such-and-such or your are mascius session session but that numbers may not make sense to some customers think when they look at the numbers what does that mean is it helping business like helping us save time so another area that I did not mention which were actively developing is evaluation - Bal explain ability and I believe that one area in machine learning research is explained about mission money that is super interesting and valuable in the business world I create a model for your business why why we make this decision why is it accurate and more importantly how much have we helped you gain more like save we will usually don't want to go into that cost savings i but more like was the baddest business panelists we have provided so this is another expertise in the area maybe it is visualization maybe this human-computer interactions how do we interact how do we help the interaction between end-users and the model so a big [Music]