data.bythebay.io: Evion Kim, Black Magic: How to apply Machine Learning to real-world problems
all right uh ladies and gentlemen thanks for coming to my talk in this early morning so I named it as my T talk as a black magic and I know it sounds cheesy but I just wanted to attract more people so basically I'm going to talking about lessons learned that I while I was applying some of the machine learning algorithms from my current company mattermark so I start the talk by this metaphor here so I think matter the machine learning is a hammer it actually is a magic hammer it is a great tool for some purposes so in case of the magic hammer it will be good for nail to the war or for hitting someone maybe one proverb that I want to mention here is that if all you have is a hammer everything looks like a nail what does that have any relation to what I just talked about so once you know machine learning you will have intention to like apply this to any problem that you may have in the word well it's not actually the going to work that way cuz let's say if you are trying to solve the problem of building the house let's say we're building this place galvanized then you need lot more things than just to put the nail onto the W like you need like wood you need cement you need the like shovel you need like all this different tools combined together with the hammer however this machine learning is a like really really powerful Hammer it's really good at doing what it's supposed to do so that's why I call it a magic hammer so it's not like the regular Hammer once you know right way to use it once you know how you can apply it to your problems once you know find the like where to put the nails on on your home building system like it will work out very well so I will introduced myself so I'm Avon Kim I'm lead machine learning engineer at mattermark prior to mattermark I was senior software engineer SL dat scientist at LinkedIn I've been working on the recommendation systems there prior to that I did my study at Stanford and the K so today's talk is going to be in this steps so firstly I'll talk about so I I I decare that machine learning is a tool machine learning is a hammer so I'll be a little bit more detailed explain detail explain about what that means and after that I will do some quick advertisement of our company mattermark and after that I will talk about the one of the most biggest problem that we were facing to solve at mattermark namely the funding extraction problem and I will let you guys know what kind of magic spells that we were able to learn while working on that problem so this talk is not about learning deep academic techn technical knowledge about the machine learning algorithms although I mentioned some names of the techniques this talk is not about the data infrastructure or do I name some infrastructures this talk is mainly about tips and tricks on solving the problem that can be done with machine learning all right so the first part of the my talk machine learning the powerful Hammer so I just searched for the Google with the query number game number trick game so I got this cheesy only for geniuses qu anyone can guess what the answer would be like anyone so there so yeah there is actually a debate going on between if it's 90 or 72 I'm on the 90 I think it's 90 I I just think it's the given X return x * x + 1 like there's a so I came up with this example because I wanted to give you guys the idea of what's the difference between the traditional computer algorithms versus the machine learning based algorithm so I I I I I'm going to assume that the answer for that is 90 from now on the traditional way is that okay I want to build a system that's returns x * x + 1 given the X so this there's a that's actually the only only code in my whole Slide the defining the function that's returning x * x + one which is python code give it to the computer and computer is going to translate into the machine the zeros and ones and then it will know that okay I can use the y = x * x + 1 as a solution for the given data from now on the difference between this and the Machine learning way is this so what machine learning is doing is instead of just we tell the machine what you need to do to solve the problem it's rather giving you the data and the model so that machine can be trained by itself so in this case we are giving it the data of if input is two output is six and input is three output is 12 blah blah blah and it we can give it the motor so this motor can be like there there are like numbers of numbers of different different models logistic regression support Vector machine the Deep learning neuron Network all this thing can be the model so what model basically say is that given this data try to apply this model to come up with the actual algorithm you will be applying later on when the future data is given so in this case for example we can give the model like let's think about it it as a polinomial model so whatever the number you got the Pol polinomial combination of the those moders will give you the output of six or 12 or 20 so if we say that machine learning will go through the data and finding the model that will fit to that data best in this case which will be the y = x * x + 1 so in the end it have the same answer same answer for the solutions to come it in traditional way it was it got the idea y = x * x + 1 the machine learning way it also got the idea of Y = X Plus x x * x + 1 but the way how it's being done is different so it's kind of similar in a way that how babies are learning the language like not not in the school but when they are learning the language for the first time in their life so my mother is taking a look at this red thing and say the word Apple like multiple times so that's the data for the baby so whenever I see that red thing I that's the input the output is the the word Apple so after that happen like repeatedly multiple times baby is trained in a way that he or she can say the word Apple when he or she see the Apple so how does it so like I heard some questions from the folks asking that so I heard the term machine learning I heard the term deep learning what's the difference how they are related so I have a one slide for that so deep learning is actually the sset of the machine learning so we can compare it to to this slide here so in here the the left is data right is the output then the bottom part this model in deep learning you are using the model called like neural networks the neural network is the model that's trying to mimic the way how human brain works not exactly how the human brain works because we don't know how the human brain works yet but we got some concepts of neurons and networks from there and tried to apply it as a model and actually recently it's surp it worked surprisingly well on a lot of different problems so from there like them learning become really hard so the reason why I talked about all this concept of the machine learning is to tell this so it is not Sky Net at least not yet it is just like really really powerful cool tool for coming up with the mathematical function that can be applied to the problems you have later I just say not yet because like maybe if we keep doing this maybe if we keep building this like the mathematical models on top of one of each other maybe at some point it will start thinking itself start coming over its own free will and trying to take over the world I have no idea but at least from my knowledge at least from my colleague knowledge it's not there yet so it it's not Skynet then what what it is I again it is a tool it's a tool for you to use it to build something to nail down something that can be used for some specific type of problems so that's my con conceptual understanding of myself about the machine learning so just a quick advertisement from here so what is matter Mark so I'm going to talk about this not just be for the advertisement but I will introduce mattermark so that we can explain more about like what kind of machine learning problems we have at mattermark and how we Ed the machine learning to solve that so what we do the mattermark is the in in very simple sentence mattermark is the startup database it's the database that collect and organize the world business information especially the business information about the fast growing startups so we are collecting this lot of information from the web public web around the world and we are applying our techniques to there to make it as a structur database and we store it into our database and we are serving it to the users paid users with the lot of different mediums like web app the exer plugin Google plugin salesp integration API you can name it so whenever I say about the matter Mark I get the question of how it's different from crunch Bas so this is the page that you will see when you are log into mattermark actually let me access the mattermark on live now because we have Wi-Fi this is the matter mark.com if you sign in It'll ask for the password so so this is the page once you go into the old company's page it will show the giant big CSV of the list of the companies so on top here it says it has 1.3 million entries and out of them we're currently showing the 50s what we can do is we have all the structured data about it so for example we can do something like okay we since we are talking about machine learning let's find the some companies that's related to machine learning so it shows something like this let's say we are filtering it by the companies in the Bay Area sear oh yeah sorry by the way someone who just like talked to me is my colleague Samir ran he's going to actually talk about the search functionality on 4M today so and I want to do something like I will just be interested in any company that's not too big in a way that it has series b or series a or and also so by doing this you can see all this kind of companies that's filter down so from the 1.5 million companies it has been filtered down into the 1,300 companies we can even do something like let's filter them by employee account and least I want some people companies that has 50 or more employees so yeah I didn't meant it but the matter Market is in the third place so anyway so let's try click in there to see what kind of informations we have in there so it has information such as like what's the description of the company what kind of tag is related to it who are the key people what are the effect what are the similar companies to matter Mark and what kind of fundings we received today up until today and the trajectory of the companies as well so that's quick short advertisement of the my company mattermark so as I mentioned we are using lot of different machine learning algorithms on solving this problem of building the matter Mark and mainly we are using it in the in places where we want to collect the data un structured data natural language data and then transform it into the structure data so one of the big challenge we had to facing was the get the unstructured data about the funding event the start of funding event and then converted into the this format how much money which company who are the investors what do the date what do the the of it so I'm going for the last next part of the talk I'm going to talk about how we solve that problem using machine learning so prior to actually jumping into the question so uh let me talk about one thing really quick so I used to work at biger company now I'm working at smaller company so there was lot of difference that I was able to see between the bigger companies and smaller companies especially in terms of the machine learning team so I just wanted to quickly mention that prior to jumping into the actual case study so good thing or bad thing about the bigger company is that they have lot of data already that's the way how they become big uh one thing that was very interesting for me to compare was that when I was building the recommendation product at LinkedIn like for example I built the people you menu so let's say you go to your phone now turn on the LinkedIn and going into people you menu and out of 10 recommendations seven of them are the one you actually know what's going to happen is you will freak out and you think like wow how did they do that it's really freaky it's awesome but 10 out of seven basically means that Precision of 70% so when we were building the recommendation product Precision was important but it was not that important so I just stick to the number 70% because that sounds like the freaky enough so we have millions of millions of user interaction data training data but we our goal Precision was 70% so I joined this smaller newer Shiner company mattermark and I joined the team first thing I realized is that we don't have that much training data firstly we are BB companies so instead of like using the user interaction data we had to use the manually crafted data andal te data to train the motor which is in terms of hundreds or thousands thousands if I'm lucky so let's say I'm you are turning on the your the matter Mark app and going into the funding history and take a look at the 10 funding history event and let's say that seven out of those 10 were correct and three were wrong what did you say you say it's really really really bad because like you you are the paying customer and we are giving you the right in we're selling the product by saying that we are giving you the right information so what ended up happening was that I now have much much less number of training data from Millions to thousands but my preent requirement has been gone up from 70s to 90 95% so at first that sounds like a nonsense to me how how can I even do that like am I in the wrong company so throughout the last part left over part of the talk I will come up with this Chi slide namely the spell so these are the magic spells that I learned and we will have total of five spells so after the the first spell that I learned from this bigger company smaller company the comparison was this one no your enemy so at first it didn't make didn't seem like making sense to coming up with the higher Precision from the lower number of training data but later I was able to to I was able to realize that there are a bunch of tradeoffs I can make so when I say no you're enemy what I meant was that there are a lot of tradeoffs scalability versus accuracy precision versus recur engineering versus machine learning there are trade-offs you can make in terms of where you are putting your resources on so you need to know what's the bottom like for that specific problem so that you can easy out the other one yeah can you give an example prision yeah I I I I get into that after I talk about this so the question was he was asking what's the example of the the tradeoff between precision and Reco so the firstly the bottleneck the the the bottom neck here the trade-off here was that the millions of millions of training data points sounds awesome but that also means that you need to build a scaleable system that can solve the millions of millions of users so using the very complicated model on that through the realtime system big giant system you need like like lot of engineering there however for smaller company since we only have we don't have that scalable data yet we are serving to the only much less than million users we can use the much much more complicated data complicated model to build the model that's getting us to the higher accuracy so I'll talk about it a little bit later but instead of using very simple model like logistic we were able to using dim learning based feature extraction model making the vectors Dimension much larger than what is supposed to be and we were able to do a lot of semantic Ro labeling all we we are able to all put all this different much complicated techniques in there because we had smaller data and smaller scalability issue so in here that was the example of the first one the scalability or accuracy so you if you want to achieve higher accuracy see you may can put more resources in there and ease out the resources you would put in the scalability so the Precision or record that's the question so basically the Precision and record what that means is that like out of the output data how many of them are correct that's the Precision and out of the all the data available in the word for example in the funding case let's say there are like 10,000 fundings around in the world how many of them you were able to catch so that's the recoil problem so every classification problem you can make the tradeoff between precision and vle if you want to make it like more and more precise you can make the threshold higher in a way that you will only get the out of 10,000 you will get only like half of that but half of them has like 90% Precision or vice versa if you say out of 10,000 like you just like say everything I received is the right information then you will achieve higher recer but you will receive like lower Precision so that's the so when we are building the clut fire we can also do the trade-off between pre and recer in a way that how much coverage you want to have versus how much accuracy you want to have that's the better of precent record here so get back to the funding thing based on matter data on year 2015 there were about total of of 150 billion 156 billion dollar of wec funding and that was comprised of 8,532 funding events how did we collect this data that's the part I'm going to talk about from now on the problem to solve is not simple but simple in a way that given the left part given the article news article about the funding or maybe not about the funding given the news article curling them and find the funding articles and how can we trans transform that into the right one we want the data to be structured in a way that we have the numbers we have the series we have the list of investors we want to have the date so that it can be used for search later it can be used for the further prediction later so here comes my spell number two is the slice and dice so just given the fact that there is a big giant this problem of let's find the funding information around the world that sounds like a pretty big problem that's the like building the house the just trying to apply the machine learning algorithm somehow directly into there doesn't really make that much sense because machine learning algorithms on its origin is has like classification problem clustering problem reinforcement problem that's or like smaller problems so you need to slice and dice the problem into the smaller chunks so that the machine learning algorithm can be applied to this bigger problem so this is how we chunk down the problem into the smaller problem firstly given the funding all the Articles so I'm not I'm not going to talk about how we do the build the crawler how we build the pipeline of the data I'm just going to talk about the machine learning part here given that we C the information the Articles from The Source we first build the classification algorithm for say if this article is about funding or not after that let's give say that given the funding article we build the Clash fire that's only extracting the funding related sentences there and then after that from the sentence we were able to extract the funding related entities and then finally we build a confidence score which is saying How likely this information that we extracted is correct or wrong so the first part was to classify the funding article so that's pretty simple machine learning algorithm so given the news article we use the TFI IDF based feature extraction actually we used a bunch of different algorithms but we ended up using this and then just sbm classifier that says if it's the funding related article or not and that gives us pretty good Precision so suddenly there's another spell three here so understand your domain so while working on this I realized that know more about the domain of the problem that I working on is actually helping me a lot on solving the problem I'm saying this because after classify the funding articles we realized one one interesting thing out of the old the funding articles that's relate listed in there so noral funding article is like comp compris of the like 50 60 sentences there are lot of unnecessary information there are information such thing as like this found also built the other company on year 2012 which was successful or someone even Mak the like joke like like who ate the breakfast for what or there are a lot of like unnecessary informations in there but we just wanted to find the information that we wanted and we realized that but by taking a look at the the funding articles we were able to find the three different type of main patterns that are the information that we'd like to get so by taking a look at the data we were able to find the three different types of the funding sentences patterns the one that looks like company blah blah blah has raised this much amount of money with the series a funding that's the amount and series type of the sentence there are also other types of sentences that says this inter Capital lad around with the participant from the this this this this company this this investor that's the investor only and also there's like the combination of both amount series investor this company a has close the 3.5 million series a round from led by this capital and that capital and blah blah so how after realize that what we did was we decided to build a binary clut fire for this three different types of sentences and this is the result so here we used a little bit more complicated approach so we built the three different Clash fire using the word to back which is previously trained from the word Chun to come up with the feature and we did some semantic Ro labeling in there and then we put it into the gradient boosting classifier so as a reg given the article like left we build a system that can find the sentences like right the color sentences are the one that we found out that is the funding related sentences after that the next step was much easier cuz since we know that what kind of sentences are important in there and what are the types of sentences we can just apply the reject paring and name entity recognition in there so for the investor only sentences we can just apply the name dentity recognition and believe all of them are the investors or like for the amount related sentences we can just apply the reject paring of finding the dollar amount and the number so that we can say that's the funding amount so in the end from the left article with color sentences we are able to paring out the right ones so the spare number four is think probabilistically so using the models that probabilistic is helping us helping people a lot many times what that means is so we have this entity is extracted from the funding article so what are we just putting into the database actually no cuz even though I said everything went well like like smoothly there can be the room for error on every single corner like there it it's possible that it found the wrong sentences it's possible that it found the hum people person's name as an investor name blah blah blah so in the end we build one other algorithm called confidence scoring so given the everything we extracted everything we have done make that as a feature and we build another classifier in this case it's trying to classify if information extracted here is 100% correct or not and then for that classifier we pull out the 0 to1 scale probability score so that we can Know How likely is the information we extracted is correct this probability the outut from the are youa so the question was if this 0 to one scale probability score was created from the previous gradiant boosting three Clash fire actually we built the additional Clash fire on top of everything we extracted so the gradient boosting CL tree Clash fire was being used for each type of the sentences so there were like three different gr gradiant boosting tree crash fire and given that all information is in the database we just just added like additional step of the classier so what were the inputs to this so input to this classier was something like the how many sentences are there in the article how many like how many like funding type of sentences in the article how many investors did you find out like what was the position of the sentences because like normally as you can see from this example like normally the important sentences are coming in front of the article so that the where the position was at it's a good indication that if we find out the right information or not but that kind of there are like numbers of the information the features but that to name a few that they were some what are the data sources say again data sources oh data sources so the the way how we find out the data sources is so I'm saying as if like we were building it from first till the end but that's like this is the result of lot of different iterations so at first we were doing it like total manually so when we are doing total manually we find out the going into the like something that's well known for having a lot of Articles like Tech crunch is one of them like phub is one of them and we added like them gradually as data analy find out the new sources so yeah there are like tens of hundreds of like new sources there and every time we add them we are testing the system again so it's basically any new article for now that's in English all right so confidence scoring is done the reason why we did the confidence scoring is actually to come up with the last point the last spell that I named it is the human plus machine so let's get the heal from mighty human being not entirely reli on machine what that means is that when I'm trying to solve this when people are trying to solve the problem using machine learning like getting into the 80% getting to 90% and getting to 99% getting to 100% there is a step function of how much resource and effort you need on each additional 1% gain so for example if I build a system that can achieve the 90% accuracy going there going from there to 100% is like totally different story instead if you just come up with the 90% solution and combine it with the humans manual approach that actually ended up being working much better so what we did was since we built a confidence score we build a web view version of all the Articles we found out is about the funding and for anything that has confidence score higher than like 0.9 or 0.8 we put them automatically to the system for anything else that's between like certain confidence score we are showing it to the our data analyst so that they can decide what to do from there so by this approach we were able to get this much amount of the data with only four data analysts on year 2015 we still have only four data analyst were you able to the results from the data analyst and Fe to your so the question was were we able to use the feedback from data analyst and feed it back to the algorithm so that it can be better the answer is yes so the originally the way how it's trained was come the the training data was coming from the data analyst and as data analyst started like putting more information to here of course we have more data and then we can fit it back so yeah yes that's definitely the case but we are in still middle of building this ecosystem of like feedback and like this output is fitting to this input so it's still to be done more of course so that's how we solve this problem so this is the summary Slide the six thing here the firstly machine learning is a powerful hammer it's not like robot yet spell number one is know your enemy there are lot of trade-off you need to make and when thring something you need to understand what problem you want to focus on to get rid of this bottleneck spell number two is the slice and dice given the chunky big problem trying to dissect it into the smaller problem so that each problem can be either solved by Machine learning or some other easier solution spare three is understand your domain know more about your domain will help you get coming up with a better machine learning approach spare number four is probabilistic like building the model in probabilistic way so that it can be instead of like saying that this is correct or this is wrong if you know that this is like 80% correct then you can come up with the better way to manage it spread number five is human plus machine basically machines are really dumb humans are really clever the only reason why we are using machine learning is machine is dum but they are cheaper that's the reason why we are using machine learning so by combining human and machine approach you can get things done better in less resource so we are hiring we have three machine learning engineers at the team we are hiring machine learning Engineers we are hiring data analyst we are hiring every everyone so if you are interested you can contact us also my colleague who's sitting in front line here Samir ran is actually going to talk about more of our machine learning approach so here be talking about building the semantic search based on all the information we collected using this machine learning idea so I believe there it he will talk about more about the actual machine learning techniques we use and the infrastructure we use so if you are interested in go to that talk at 4 pm thanks for listening to my talk these are the contact information you can get reach to me and any questions the question was is this approach the POS available to be used for the or Clash fire classification problems the answer is maybe so the reason why I said named it like black magic not magic is because a lot of the the things in the machine learning even the algorithm itself is like deterministic the way how people are using it is really undetermined so for example I show you the three phases of our machine learning Engineers for any given problem the way how we're solving the problem is totally different like I'm I used to be the fan of support Vector machine Samir ran was the fan of the gradient boosting tree I like to make use less features someone like to use more features someone like to do the Cur validation someone like to do things manually so the spells that I came up here is the lessons that I learned while working on the problem so lot of them can be useful but I'm not going to say these are the theorem or AUM on building the stuff um for the smaller company that you actually mentioned you did quote the number of for me your sample set is small that's why it's not necessary statis sufficient sample set compared with the large one that's why if you have outliner and so on you will impact more your result so actually it's very different from dealing with a domain I actually have sufficient sample it's more statistical stastical and then you can do a more uh variation still being accurate and not probably the power want to do because it's harder to analyze a large scale data with machine learning and it's the right approach you probably can solve them more automatically do you agree so his question was how would I put it I think you appli to a smaller s set and it's not sufficient yep and that's why your approach is are different so what he said is that since the we are working on the smaller data set there's a possibility that there's a bias in the data set and statistically and that's why we are trying to do the things in different way than when we have a lot of data that's exactly 100% correct so the trend in machine learning for the big companies is that collecting more data put them all into the GPU power machine with deep learning let the let the neur network take care of everything as you build more and more and more machine power it will get better that's actually kind of trend where things are going the things that I talked about here is mainly more about like how can we fight the better when we don't have enough data cuz any any startups in the world that when they are created they will have not that much data so from there and going to the like bigger data part how can so there there of course will be the bias in the data and statistical s then how can we fight that bias how can we prevent it from overfeeding all the things can be done by using some sort of different techniques than when we have larger data and I think some of the things that I mentioned today is about that so in the information Theory there's a way to say given a sample set if you have this Dimension and then whether you have sufficient SLE set yes I think that's very important to say how should I solve this problem yes yes so what he said is in information Theory there is a theory that you can say how much data is sufficient for you to say this is sufficient amount of the data to make the decision and yeah that's that's what I said so can I ask how many data entry you have per company how many data entries I have the for the compy I don't think I can I can answer that question thank you any other question yeah yeah so in one of the slides I think the initial part of machine learning you show like something svm buyer you also mentioned you tried to svm is probably so I guess my question is for most of the problems I've done I found svm and logistic progression kind of perform similar or maybe asbm is a little better but I still go with listic regression it's easier to explain it's also easier to transport it to other people or easier to scale so do you have any thoughts on that so her question was and most of the problems that she were dealing with she used the spbm or logistic regression and they both show the similar the similar the performance but she anyway went with the logistic regression because that's easier for her to explain it to the people who are not machine learning person so the question was what kind of thought I have about that right yeah like did you go for sbm because like why did you go for sbm I guess the simple answer is it worked better in this case it work better compared to logistic regression and all other approaches so for here as I mentioned like the getting the better accuracy was really important so the explainability versus like accuracy that's the but our scalability was also not scalability was also not a problem because like as you as you already know a logistic evation is really easy to scale compared to the other algorithm so yeah that's the probably the reason why we did choose that any other question yeah I so this is the going to be the last question as increases will your model change of and today may sound fantastic for this problem but your dat increases probably some other model May a better so how manage that change so the question was as data increases for now svm or Guardian boosting Tre was working best but like as data increases some other approach may work better and how do we deal with that change how we plan to deal with that change that was the question so our plan is to we are planning to building the whole pipeline of like doing the retraining and com compare the retrain model to the previous model in a way that if this makes sense to making the change to the retrain motor because as so we are only scratching the surface of a lot of different things and we are actually in mid of building the ecosystem between this Clash fire that Clash fire and that Clash fire so as we change the one p part of it the other part will be affected so we are planning to building the system that's given the new M compared to Old M and if it's working better switch it if not leave the existing M kind of thing so that building that kind of system is our scheduled solution for that okay I think that's it for my talk today thank you very [Applause] much