data.bythebay.io: Lukas Biewald, Active Learning and Human-in-the-Loop Machine Learning
Recording: data.bythebay.io: Lukas Biewald, Active Learning and Human-in-the-Loop Machine Learning
can I get a little sense of um of you guys before I start um how many of you guys are students I'm curious and and how many of you guys like are like say would like have like data science on your on your business card got it interesting um how's the conference going so far yeah um is it like too technical not technical enough but uh that's what I always try to gauge with these talks I've never know how uh how technical to go Prett pretty technical pretty Technical and you want even more technical is that nice well all right how how many you guys like do machine learning like in your day job oh awesome all right well you you might find this talk not technical enough I don't I won't be offended if you if you leave um so I'm I'm Lucas you know I I um used to work on machine learning algorithms at um at Stamford and then at Yahoo and then at a startup called pow set um and and about six or seven years ago um I started a company called crowdflower anybody heard of crowd flow I'm curious anybody use crowd flow nice awesome sweet um so uh CR briefly uh we have a a a we crowdsource data collection so we have people all around the world that um can collect data for you um and so this supports a lot of machine learning tasks because machine learning often you know machine learning generally starts with training data and often you need people to collect that training data so um I actually don't do machine learning in my day-to-day job but I think a lot about machine learning because most of our customers are taking the data that we collect and using it to build machine learning models and then doing a thing called um human in the loop where they take um places where the algorithm is not confident and send it back to people so I'm super passionate about um making machine learning really work and that's actually why I started crowdflower um and I even have I have a couple graphs that are sort of the um epiphany of of why I started crowd flower which is um you know the first graph is the effect of better algorithms which is what I really worked on in the beginning of my career um and I should really add neural net to this but this is kind of on the left you have a toy data set probably all of you guys have had this experience I just did it with kind of vanilla psyit learn on a um a uh a document classification task and you know you try naive Bay which is kind of like the simplest algorithm you could think of and then you know maxent is maybe a little bit more machinery and SP spms maybe a little bit more um and you could try a um you know these days right like a CNN or something um and and but what happens is you you don't get that much improvement really from the the algorithms right I mean you know here you have um on this data set I went from like 19% accuracy it's a 17% accuracy um which you know if I was trying to publish a paper I'd say hey that's like over 10% um error reduction and and that would be um pretty exciting right if I was the first to come up with a an svm but um you know in the real world this isn't um this isn't super helpful right like you know a lot of our customers would look at this you know if they're not actually data scientists or machine learning people if they're kind of their boss they'd say well you know 19% accurate classifier can't really use that 70% accurate classifier well still can't use that um so I think what data science teams and machine learning teams spend most of their time really doing is um is feature selection so I was actually surprised when I got my first job in Industry um you know there was basically one person working on um improving algorithms and probably 30 or 40 people working on um finding better features right so um you know the feature selection process is um you know kind of going from the data that you have to uh a set of numbers that you feed into your into your algorithm so there there's lots of different ways to do it right so you know unigrams is basically like you look at every single um word in your document and assign a number to it byrams is is a little more advanced right you look at each pair of words and and that can sometimes you know work better right you know if you say the word great that probably means positive sentiment but if you say o great you know that might mean um you know negative sentiment right so the pairs can have more information um and so you know on the same task I tried it with um I tried to do my classification with unigrams and it's you know 26% error rate you know you try biogs and it goes down to to 22% then if you combine unigrams and biogs um in this model that I had it goes down to about a 17% error rate and that's you know that's kind of the typical thing that that you see right like these the features often super super helpful often more helpful than um you know kind of optimizing your algorithm finding the right um parameters but it's really hard to know what's going to work I mean I remember um at Yahoo kind of working on document quality you know I remember we discovered that um semi colons were actually a pretty good predictor that the document wasn't like a crazy internet rant right like people kind of pounding on their keyboards um they never really use semicolons but they actually will there's certain words that if they're misspelled and they're misspelled with all capital letters are like guaranteed it's kind of a crazy internet rant that you wouldn't want to show um high on your search results um so you kind of you look for these um these patterns but I think this is really a big part of the art of machine learning is finding what are the right features that make sense and you know you see the paper on the right has kind of a similar um outcome to me where they see sometimes the bams are working better sometimes the unigrams and byrams are are are showing even higher accuracy but then you know not always right like they had parts of speech of the words um and that doesn't help in with some of their their classifiers so um these are the things that that that um you know we don't do right this is what what our customers do but there's a thing that that we do that almost always helps the accuracy of algorithms um which is the effect of more data right so um you know here I had about 10,000 records that I showed my classifier um and then when it went up to 20,000 it actually went from 13% error rate down to 8% and then when it showed it around 40,000 the error rate went down to um a little over 4% right which is a really meaningful um result that I just got from from adding more data to my to my training set and there's actually a paper on the right which I I think is kind of funny right because the point of this graph on the right so it's the same the same axis right where the the Y axis here is is error rate so down is better and they're basically showing you know kind of red is the random thing right that's like the the naive thing um and then the other colored lines are improvements right that actually got the the paper published and so they're showing that like okay the the green line is maybe you know a couple percent um improvement over the red line right and so they published the paper kind of saying that and that's the point of this graph but the kind of more obvious effect that they completely ignore is the fact that each time they add more training data the the a rate goes down a lot right so they start with a th records and I think they go up to 200,000 um so it's it's log scale but um each time they basically add you know in order to magnitude more training data the error rate goes down more than the effect of the paper that they're publishing and so you could say well you know you can't always add more data but you know sometimes you can and if there's cases where you can actually add more training data you can do amazing things um there's actually one more effect that I think is kind of even more meaningful than um more data and actually can never find a paper that really talks about this if any of you guys are are actually in research I think this is a really interesting um research topic so I can only show you kind of my anecdotal evidence but we see it in almost every customer that we um work with and and everybody that we talk to which is the effect of cleaner data is humongous right so here I actually went back through my data and I added noise so um I fed the classifier 90% accurate data right so 10% I kind of scrambled and then um the the accuracy is of the the actual classifier itself is is 12% right so the kind of input training data is only 90% accuracy doesn't learn as good a classifier right that classifier is 12% and then if it's 95% accurate data fed in it's it's you know maybe 7% error rate and then 4% with 100% accurate data right so this is a humongous effect right and this is potentially even easier to do than collecting more training data and yet it's really rare that people want to go back over um all this sets of data that they have um and and clean them up themselves right they think oh this is too much work but um the effect is is powerful and so you know is there a question yeah oh yeah let me try to repeat it because it's I'm actually having a little trouble hearing you so I think you said in the real world training data isn't always clean clean the data by model but the real data is not always ah yeah yeah so I mean I guess I I guess if you mean um yes so I think there's sort of two types I mean in the real world data is certainly messy and you want it to kind of handle the messy cases I think though if your labels are wrong that's sort of like a different thing right so you know say you're trying to I mean like a simple say you're trying to do face recognition right um you know there there is like a real thing that you're trying to do that I think someone could say you know kind of did you get that face recognition task right or wrong right so you know I think that's a pretty I mean there actually probably are cases that are not well- defined for sure right but I think you can still kind of Define the task as as there is sort of like a platonic ideal of of what you're trying to accomplish and um I think like you know your testing data may also be not clean right and so you know that may mess up your results but that would kind of only underestimate the value of clean data right so um you know whatever you estim on your test data I think you know by cleaning up your test data it would even more show the effect of of uh of of actually cleaning your your training data but I think there are other ways that you could clean clean your training data that would scrip the distribution like if you actually cleaned the underlying features that wouldn't be something you'd want to do right like you know spell correcting your data you definitely not want to do that right because in the real world people misspell stuff all the time um but yeah yeah good point yeah training you if you train on unclean data and then you uhhuh you're saying you think that's true or is the question I mean I I guess there's probably different situations so the question is if you trained on unclean data and tested un clean data would you get better results um I mean there's probably some some mathematicians in the audience that' be better equipped to answer this I mean here's what I would say like you'll get better results if you train and clean data I'm really sure about that yeah com like to make so the way I see things is like why he just said the data in fundamentally statistical then it's only a matter when your s set is to deal with this statistical behavior and you can see that data you have I can curve down vertically by the gain of having more data I can plau yeah C point you already have measure those physical labels of parameters you want to meure in order recognize orass yeah that's the way so my my think my thinking is actually you should live and die even okay so here's the comment let me try to let me try to express it so the point is um kind of two things right one is that um you know your data the the effect of more training data will Plateau right um and then the second point is that you should um you know because in the real world data is messy you should take lots of messy data and use that to to to train your model um and so I think um I think I guess um I I I I hear your point um I I disagree um but I think like this is something you can empirically test right so I'm sure there are situations where the amount of training that you data that you have plateaus um but I think it's it's a good thing to ask yourself like am I at a place where it's plateaus right because certainly there's diminishing returns right so you know as you get more and more data the next piece of data matters less and less right that's undeniably true right but um you still might want to get lots more data right because you know if you get orders of magnitude more data that effect might be good and like you know if you're paying millions of dollars for a machine learning team maybe you should allocate some of that money to um data collection and see what if I really collected a ton of data what would that do so all I say is is test it right um you know I think we're we're all data people we all really empirical right so I think that's like a it's actually kind of a testable hypothesis um and I I think like another thing that I see is like a lot of the really exciting advances in algorithms seem to me like they've come from the fact that data has become so plentiful and in some situations like you know Facebook gets you know crazy amounts of of data with the face labeled right and so they can do like really really interesting Advanced things in in facial recognition right because they have data sets that you know when when we were grad students we'd only dream of right so um you know I think even if you think you're at a plateau with your data um a new more advanced algorithm might be able to actually use the fact that you have um more data but you know again I think like all I'd say is is test it on your data I would just encourage you to do that and then the second thing is you know does cleaning your data really help um you know I think that that like what you're actually trying to do with your data is usually pretty clean right you usually have a pretty clear idea of what you are trying to um accomplish and feeding in dirty examples right dirty feature sets you definitely want to do right that'll make your model more robust but I think actually like cleaning the um output of what you're trying to predict um is typically helpful but maybe not I mean I I actually think uh you know if you have examples where it's not I'd be really interested in in seeing them but I'll say like you know what what we see well what I saw with this particular data set and what um our customers tell us is that um when they clean their data they often get better results so um you know if you have any data sets or or evidence I I'd love to see kind of more examples to put in this slide actually feel kind of bad that I don't have a you kind of research paper to site or show on the you know the right side of this graph this is definitely my own toy toy data so you should be skeptical of my uh my result here um but what what I do know for sure is this next point which is that people you know data scientists spend most of their time cleaning up data right I mean I think everybody kind of realizes this right you know in the last year or two the media um has really picked up on this um and I think that you know whether you're doing modeling or um business intelligence if you input dirty data it's it's not going to work as well and and data can get dirty enough to the point where you're actually you know you can't build a good model or um the results that you're collecting on your data is actually wrong right so um you know I think what's for sure is companies hire lots of data scientists and machine learning people and these machine learning people end up spending a lot of their time going through record by record cleaning up um data I I know this history because I was doing this right in my you know I was doing this in grad school I was doing this you know in my jobs um after grad school like I think everybody's had the experience of like opening up your data set kind of looking for the weird outliers taking them out or dealing with them in some way um you know this is kind of this is like the real what you end up really doing when you're you're actually trying to make a model work um and so you know I'll just say like where I'm coming from right this is this is crowdflower and some of you guys I guess use it um and so we're we're a platform where you can kind of crowdsource the the data cleanup and enrichment so um you can send out tasks to lots of people so this is a a vision task right where you can um you know basically like label car and trees and and Roads and things like that um but we do a lot of kind of NLP based work where it's like you know basically like categorize a piece of text um you know categorize an issue that one of your customers had kind of all the different machine learning things that that folks work on um and I just kind of want to talk about some of the the data sets that we've collected um you know when we very when we just first started our company um you know we had no customers and we had kind of a crowd sour work force we wanted to try it so you know one of the fun data sets that we collected back in I think like 2009 um was a color data set so we basically showed people um solid blocks of color and asked them what would you call this color right so um you know you might call the top one blue or sky blue and the bottom one red or purple or something like that um and you know we created this this visualization um around uh what all the different people said which I think is really kind of fun to look at right like you you kind of like you know first you see okay people call this stuff blue but then you dig in deeper and it's like there's like Cerulean in here and that's like a really tight um circle like that's a color people have a really good sense of of what it is um and we actually took that data set and put it online so that you guys could use it and people could do um interesting analyses of it and one of the first things that happened was this really nice guy at at sap went through and he cleaned up um the the data for us and I think this is a great point you know people are talking about like what's uh you know do you actually want to clean up a data set um and I think this really shows you how it depends like you know we basically had a ton of misspellings and um you know a ton of weird stuff in here like somebody said this color is baby poop right and like you know this guy took that out um and it's like well you know do you want the misspellings like do you want consistent capitalization really depends on the use case right so I was happy that this guy um cleaned it up and published a second data set it's almost better than open source in a way because you know the first version and the second version um are useful to everyone um and then this this guy Martin wattenberg uh who was at at IBM at the time I think made this really cool visualization where you could see the um the size of the text is how many people said it um and then the location is sort of like the average of all the people that said that color right so this green is in some sense the average of all the things that people called green and you can see that green is kind of the most popular thing to call a color whereas turquoise is a little smaller and aqua is a little smaller than that um and we ended up doing this in a whole bunch of languages and comparing them and and I think it led to a lot of kind of interesting work um and then companies visualization companies started using it as a um as a test data set um for for for their tools um and then actually some students took it and they made an app for colorblind people where you could look at a pixel and it would say here's kind of the closest colors to that pixel so you know if you you say like okay this is like considered to be um sky blue right so it's not just like like the normal kind of color blend thing I think shows the red green blue values but this actually showed kind of Tex of of how people would describe the color um and then you know we collected some other data sets over the years like um Apple watch we asked people hey you know what do you guys think about um the Apple watch um or actually this labeling social media uh labels the Apple watch and we sent it up to journalists and got all these different articles right so kind of you know kind of fed people that kind of wanted to be data journalists data and saw what they did with it so um you know one person says okay you know gets a 55% approval rate another person um you know it says like just license data to Texas right it's a Texas column that says you know Texans aren't so thrilled about the Apple watch um and then USA Today says okay women love Apple watch more than men right because they think that the apps are better um and so you know we would publish these data sets from time to time but you know what we saw is that is that people were collecting the same data set over and over right this is a while back um someone is collecting 11,000 or there's 11,000 jobs that the word sentiment in it right so there's just lots of people on the platform kind of collecting sentiment data sets and we thought you know I think a lot of these people would be willing to share it with each other if we made it easy um so we launched the thing saying like Okay um we're going to make we're going to make try to make data sets public so that that people could use them um and we add a little button where you could say Okay publish my data set and um you know we we in the first year so we got about 12 million um data sets released which which felt good um but then we actually changed our our so we launched a a data forever in library um that you guys can actually go to where you can download lots of um interesting data sets that that customers have collected that are labeled by people um and then um but then we actually change our pricing so you can use crowdflower um for free if you make your data public and that actually got tons and tons of adoption right so um at that point we got lots and lots of data published um and I really love it if if folks like you actually go um and use this right so um you know we have an API where you can download all the data sets um and and at this point it's millions of of um data sets in fact I think um it'd be really interesting AI challenge to try to label all the different um data sets at once because it's kind of all the different things that people are building on machine learning on top of including you know URL categorization um you know kind of record data from various types of PDFs extracting names and titles which is you know a harder job than You' think um you know summarizing articles even like weird stuff like is this image funny right so this is probably an unusual um image labeling task um that I'd be interested to see how well something could do um you know there's there's medical images there's um classifying attributes of people um and so you know one of the things that we we do with this is we run um kaggle competitions from time to time because we have all this data available and I just thought this was a fun one because it's uh it's actually a search relevance data set and I kind of worry in a world where um training data is so powerful for making algorith work is it going to be hard for small startups and kind of upstarts to build high quality machine learning algorithms when they don't have that same core data so we took a big chunk of data um and we put it on kaggle um and it was funny you know in the first couple days we saw that um you know the Baseline accuracy of just doing kind of the dumbest thing on this task was around 30% and you know in the first five days it went to a little over 60% and I was I was really watching it like it was really fun you know cuz I I worked on search relevance in a different life and I I was thinking I wonder how good they're going to get this classification um you know if in the first five days they' gone from 30 to 60% maybe they'll get to you know 90% over the next uh next month or two um and in fact what happened is like they completely they were completely stuck right so you know like it turned out um getting to about getting to a little below 70% was actually really the maximum um accuracy that they could get in the data set despite um tons and tons of teams trying right so this competition actually got more and more popular um as time went on um but people couldn't um kind of couldn't break through the um the quality level they got in the first um significantly higher than the first couple days of the competition um maybe because they weren't allowed to clean up the data or add training data um and so then you know we were thinking about um you know what could we do because we see this all the time with our customers I felt this you know in my job where you kind of get stuck at a certain level of acury and it's not good enough for the kind of thing that you want to do um and that got us really excited about human Loop Computing um and and so just to Define human loot Computing for for those of you that you know don't do this you basically start with an AI classifier where it's confident you send you take the output and where it's not confident you send it to a human to get it annotate and you send it back right so this is a really good model if you can't get a classifier um as accurate as you need for the task at hand so you know for example um you labeling support tickets um is something that a lot of companies wouldn't tolerate in accuracy below like 99% but you know that's probably too too high of a reach um for for modern you know kind of Cutting Edge natural language processing algorithms for many applications um but what you can do is you can say okay if if I have an algorithm that's 80% accurate but half the time it's super confident in its answer and it's and it's like always accurate um you can use that 50% where it's very high confidence right and now you've actually saved half of your cost um using machine learning and you don't have to worry about the fact that your algorithm is only 80% accurate and so that's just the first part right the second part is you know you're sending the the not confident examples to humans to get them labeled and there's a whole field um that around kind of like what's a good label to get and it turns out that basically the not confident labels are the best labels to feed back into your algorithm right in a process known as um Active Learning so you set up a a labeling process here not only is this useful right but you have a labeling process now where your algorithm is going to get better and better um much faster than if you were picking random examples um to get labeled and I think it's been interesting to see all the different ways that um you know people are using Active Learning because it really really works right this is one of those things that um you know you don't have to kind of squint your eyes to see the difference right so you know as you collect more data um using Active Learning you get um order of magnitude effects on the the power of the data that you collect typically um and and I think like this is kind of studied from just a labeling perspective but if you look at um kind of real world applications you see um really interesting human the loop um applications right so you see like self-driving cars um you know have this this so Google published their metric of you know kind of miles per disengage right so they're in Google's case the car is mostly driving right but you know every 4500 miles or so it actually hands control of the car over to a human right or it would crash right so the car isn't yet driving as well as humans but it's kind of you know has a human the loop that it hands it over right and you know Toyota actually has a really different Paradigm where the human is mostly driving but they're trying to build a system that stops the the car from crashing right so this is sort of using the algorith the case where it's just really really confident something bad's going to happen the algorithm actually takes control right so I think self-driving cards again it's an example where you need really really high accuracy levels um higher than than Vision algorithms can get today maybe um and so you know people switch to to this human Loop model it works super well for them um you know there's also adaptive cruise control and then I think Advanced chess is super cool right so um you know I I love games and I remember when um deep blue beat Kasparov um and it turns turns out that actually until really recently um if you wanted to make the best possible chess algorithm you would still keep a human in the loop right so there's a thing called Advanced chess where they actually have a human operator that's kind of using a computer to suggest moves and in kind of closed positions or positions where the computer struggling the human operator actually make the decision about the the move to make right so actually I think this became um I've heard that this became um not useful about a year ago right but it was about you know 20 years ago that deep blue um really beat beat C so those computers got better than the best human but still humans stayed relevant for 20 more years in terms of making the best possible um chess algorithm um and then I I I like love go you know I grew up uh like playing go and and kind of want to be a professional go player when I was a kid and and um it's really interesting to watch Alpha go um you know beat the best or maybe maybe the best uh human and I think again you know it was interesting to see you know I think the one game that the computer lost it made a really really funny mistake right so it won the the match three games to one but you know made a really bizarre move that I think a human operat would have caught so I think you actually saw in the the playing of of alphago um that the um you know if you had actually added a human operator I think it would have won U it would have completely swept the series so I think go is probably still in a state where human computer hybrid would be better and I think it was also interesting to learn that they you know they bootstrapped Alpha go off of um lots and lots of games played by strong um humans on a server right so the initial um definition of the problem was predict what a human would play in these records of um high quality humans playing each other right and you know from there it it trained and got better and better um but you know they were talking about the the Deep go folks were talking about like you know we'd be interested to um to see if we could do this without bootstrapping but we couldn't do it yet so you know I think like training did actually really mattered for making um Alpha go work well and I think human the loop still is still really important for um you know applications like like go playing or even boring applications like supports classification or or um making cars drive on their own so uh thanks my my Twitter is l2k and I'd love to talk with you guys all right uh we have a few minutes for questions there is a break after this talk and the next Talk starts at 4: so if you want to hang out and talk with Lucas that's fine and right now if you have questions uh raise your hand and talking to the mic okay let's give one more hand Round of Applause for Lucas thank [Applause] you