Text By the Bay 2015: Omar Alonso, Label Quality in the Era of Big Data
Recording: Text By the Bay 2015: Omar Alonso, Label Quality in the Era of Big Data
today I'm going to talk about uh practical stuff so I don't work for Microsoft research and I work on a product team so real things but before I move forward the usual disclaimer these are just my opinions have nothing to do with Microsoft um this is join work with a bunch of folks um current and former Microsoft is um from the Bay Area and also from redmund the outline I'm going to give a um quick overview of what do I mean by label quality in the context of big uh big data and machine learning I'm going to talk about we we programming which means how to program people um real problems when not mention algorithms for managing crowds and quality control and then a few takeaways and you can interrupt me at any time I'll give the slides back to you I have a lot of examples and every slide usually has a paper so plenty of takeaways okay A bit of a context here so uh we're going to assume a supervised or semi-supervised learning uh setup which means that we need a human somewhere either to help us create a training set to help us evaluate an output of an algorithm or to Signal something so there's there's somewhere somewhere a human um obviously this is going to be different from uh normal users because the users are not interacting on a real system but it's basically um on a bit of a synthetic system and we're going to ask expert or users to do stuff for us I'm going to add two constraints uh from Productions and from real stuff the first one is scalability okay so I don't want a 100 labels for publishing a paper or just r a demo this is about thousands of millions of labels and the second is continuity it's not a one shot so this is not a one off exercise this is a daily monthly or weekly activity depends on your domain so the last two are very very important in production systems why we need data and why we need labels if you happen to work on machine learning most of the stuff for building future engineering and for building training sets it's about data and labels if you work on data science analyzing data involves analyzing high quality data and analyzing how how people label those dat set and how can you use them for something else and finally whatever you do at some point you need to measure things and you need to measure you know Precision recording resarch engines reer curves uh whatever you want but you have to measure something um data skin so we all already know this there's a ton of of free data available on on the web that you can use to train in particular learning systems and crowd when I say Crowd series all of us um you know we we can use anybody in the planet to do cheap labeling for data and I cross Big Data because this is just data okay forget if it's small big medium it's just label and you need good labels for data but also the quality is super important here you may remember garbage in garbage out so we want to minimize the amount of garbage that we put into these systems and I included three um I if you can read them but you know there are three references of people who talk about label data and machine learning and I really like um an article that Pedro Dominguez wrote on ACM a couple years ago which he menly in in one of the on one of the sections of the paper it's all about feature engineering and dealing with data okay so you spend most of your time cleaning data duping data managing data wrangling data and not a lot of time on the machine learning part okay so this talk for the next 35 minutes is about the boring stuff not about the cool stuff about machine learning what is a label label basically means that I'm going to assign some sort of a category or some sort of value at a particular class in the case of a relevance evaluation um task in building a search changing I can have the query text by the bay and then I have the homepage of text by the bay and I'm want to ask is the page relevant and I need three type of labels that says yes it is very relevant somewhat relevant or not relevant then you have examples like you know spam in emails and I have a spammy potentially spammy email message and I'm going to say this is Spam yes no so those are the things that you collect as labels however this not the whole story and let's talk a little bit more about the importance of labels and this is this is kind of important right so say that I have to label a set of documents in the context of information retrieval which means building a search engine and you know I'm from South America I love soccer I love meat so I have my own judge characteristics right and then they show me a bunch of documents and I'm going to be judging those documents okay I produce a judgment but then Peter and Bob are going to do the same for the same document for the same query and at the end we'll produce what we call creation of a label that is a label okay we're going to use that label to learn a model when we know that then you can predict labels for the rest of the collection why because we cannot label the entire world byde web we're going to label a subset and then we're going to use a machine learning model to label the rest once you can do predictions then you can evaluate and see how good your algorithm is why I'm doing this because if you don't put the right labels as you start on this workflow you're going to learn the wrong things you're going to predict the wrong things and you're going to evaluate the wrong stuff okay so hugely important to make sure that your labels are of high quality what do you mean by label quality so data Gathering is not a free lunch okay so you have to own it um you cannot Outsource label quality we're going to talk about crowdsourcing but do not Outsource the acquisition and label quality labels for the machines are different from labels for the humans okay and if you talk to anybody who is working on ML basically there's an ml engineer who is going to build the classification task or classing or whatever you want and he or she needs a particular set of labels for w or some sort of open toolkit and then you're going to reverse engineer the task for people okay so it says oh my system needs x y and C then I'm going to take x y and C make a form so people can give me those labels wrong there's PE there there are certain scheme of labels for people and then you have certain scheme of labels for the machine and we need to talk about that shortly but they're not the same okay so if you need yes and no that's doesn't mean that people are going to say yes and no and I'll show examples of this I was talking with some folks before the the talk and you know there's always a lot of emphasis on algorithms models optimizations some money from the label so when you talk to anybody who's working in machine learning is like oh I'm an sbm guy or I do neuron networks or I I like this some sort of optimization model etc etc but nobody talks about this okay how can I use algorithms to make sure that I get high quality labels this is the last part is the boring stuff right this is super important if you're going to build training sets Okay extremely crucial if you're going to build training sets so careful with that data in the area of uh Big Data machine learning like I said the workflow is you have some labels you're going to compute some features you're going to build a predictive model and then you're going to optimize that's the workflow everything that is labeling and experimentation is perceived as boring but we're going to say do not rush labels we're going to talk about the difference between humans and machines and as I said before do not Outsource you have to own it end to end and more important if you're going to do it at Large Scale you have to really Own It N to end okay enough for kind of the motivation for the talk so let's just get into more details what we progaming or how to ask people to do stuff okay human computation this is kind of the label for this area I'm going to use humans as processors in a distributed system um you know worker a Raider an annotator a judge kind of different names for a person doing stuff for us now people are very good at doing uh and basically solving problems that computer are not good and in case you don't know all of us we are computers so every time you log into Facebook and like a page you are Computing something for Facebook okay every time you retweet you are you know probably helping the Twitter algorithm to detect a trend topic every time you enter a query in Bing or Google you're just giving me data so can show you better results all of us are Computing things a few years ago Lis vanan when he did his PhD at CMU um he call the term you know human computation and you know all this notion of games with a purpose so what about playing like you know annotating uh images with labels they're going to use that data for training systems um non platforms today are you know Amazon Mechanical Turk and crab flow which are probably the two most uh popular platforms on how to collect data Microsoft we have our own platform briefly are going to mention uh I'm going to mention about that but in basically if you want to run any human computation today you go to m Turk or you go to Cloud flower and then you know spend a few a few cents a few dollars and get your labels why this is interesting because it's extremely easy to prototype and Test new experiment very very easy okay you don't have to build anything the platform is there you just create a tab ask upload data collect the labels you're done so cheap and fast there's a lot of papers on cheap and fast for NLP cheap and fast for machine translation Etc this allows you to introduce experimentation early on in the cycle so if you're developing stuff you know as you develop you do all these experimentation all these you know labeling Gathering task um allows you to implement an experiment as you go so you test something didn't work you go back to the Whiteboard redo the experiment redo the labels etc etc and if you get a new ideas this is extremely useful because you can move very very rapidly however humans are different from machines okay so the instruction set for a human is somewhat unknown okay the instruction set for a machine is wellknown ADD substract move jump move to the register for humans it's like yeah I can add to much two plus blue but don't ask me to add you know decimals with you know seven point digitals because I'm not to do it it's very very difficult for that machines are very quick human latency is not so quick for certain things you don't have to pay a machine you have to pay a person uh assuming your algorithm is correct there are no errors in the machines unless you have a haror problem but humans we do make a lot of Errors there's not a concept of a task difficulty in a machine there's a lot of concepts of task difficult for humans and then there's a bunch of human factors you know I'm be boring I'm tired uh I'm sick I don't want to do this I'm going to do it in a sloppy way etc welcome so how can we ask questions asking questions uh asking questions to people is basically part of an art and part of science okay um instructions are obviously key you have to know how to ask questions and in the case of you know your your domain or your M Turk or your craft flower the workers are not experts so you cannot assume they know exactly everything about what you're trying to do probably they don't okay you have to always show examples and always try to hire a writer because the engineer will write the specification but you need a writer who is going to write the communications to people and this is a very common mistake people just do not do this basically is you have a a developer who's going to write instruction on how to do stuff and guess what it's not going to work the second part is usern interface design so you probably have learned this at grad school or undergrad or you have an idea of how to design a good user interface right so the experimental all the task they have to be self-contained you have to keep it short and simple okay short and simple uh very clear what you need to do what you don't need to do engage with the worker you know do this here's good data uh good presentation at the document level same as a design you need to grab attention you're working and competing with other uh workers trying to do stuff as well um and always ask for feedback so if something is not working just ask you know is this clear is not clear what I'm doing wrong Etc um localization is also a problem you may have task that you know say spelling correction in English you know French Spanish and German it's much better to get the task first in English making sure it works and then you do the localization not do everything in parallel because is it's not going to work other principles of you know text alignment legibility reading level some of stuff is easy to read some stuff is very difficult to read um you know there's multilingual Multicultural issues um basic stuff people with you know color blinding problems and then the mental rigor to perform a task not everything is straightforward some of stuff it's pretty pretty uh difficult uh another area that people kind of don't don't um take a look is this notion of exposure effect which is if I show you the same document and the same query over and over over time you're going to start liking it when the first time you didn't like it so kind of randomizing the content and make sure that always there's fresh things to judge and label is good okay so far so good just to building a task uh the other part is quality control and general um so the quality control problem is basically uh bir directional so there's quality control for the workers but there also quality control for the requesters okay so you may you may think that the worker is doing a bad job which is basically the the facto approach okay you don't get good labels you blame the workers they're bad but maybe the workers you know perceive that your extremely allows you request it because you're not clear on what needs to be done so always treat it as a bidirectional problem and again everything I'm saying here is is not just for crowdsourcing but this is for human raters judges in your company annotators Etc same when to assess work quality so basically you have three checkpoints before so before you start the task going to assess quality usually using qualification test or a similar mechanism for training okay so the the goal is to screen select recruit during while people are doing work you're going to assess labels so as I click yes or no you know going to do a random check did I pass the Honeypot did I miss did I agree with the rest of the majority how am I doing right and after that um then you're just going to compute how I did on the entire task how I compare to my appears how I compare to my past performance why we need this because at the end of the day what you want is to make sure that the labels that you're getting from people have high agreement this is this is key okay so you want to measure agreement reliability and validity um in statistics it's called inter agreement an inter agreement is between judges so you can have two experts and then I'm going to measure if the two experts agree on a particular label or then or I can measure an expert or a judge or a worker against a goal set and you can use Coen scapa which is a well-known stat for two Raiders FL which is an extension for any number of Raiders or kryp andorf which is any number of Raiders and in certain cases you have missing values you can use that okay this is solid and please use these things okay don't say oh it just got this and looks like the stuff is cool and looks like it's decent no solid statistics for getting these things um and I'm going to see that what happen when you don't achieve agreements and what are the problems on that other practical tips um you're going to sign up as a worker in your company or in a platform do some work you have to eat your own dog food um if there is problems like poor guidelines the payment is low Etc you have to address and everything counts um a bit of The Human Side as a worker you know you you hate when the structures are not clear uh you might not you're not as Hammer but you know you just don't actually understand what you're supposed to be doing uh the task might be boring it's always good to get a good pay but sometimes it's always good to have something really cool to do because I'm going to learn something now as a requestor what you want is attrition to manage your attrition because you need people who actually do the labels for you always and it's a balancing act because you want to get the right results but at the same time you want to you know have a cool task so you know workers can do things for you and sometimes you just want honest answer you know I don't care don't don't try to be good to me I just want your honest answer maybe um the document is not relevant just tell me it's not relevant it's not good we want that um you also want qualified workers but there's a lot of problems in the platforms today to kind of pre-select qualified workers we're going to talk this about later and managing crowds and task is usually uh a daily activity and it's much much difficult in managing a data cluster okay because all these things change on a daily basis and then you're going to run into other issues this is a great article that appeared a few months ago last year on Wire magazine and it's about um how Facebook is using a company in Philippines to um sity check that every picture is good so when you get into your Facebook feed you don't see any surprises the article is great because it goes into um kind of the dark side of human computation which a lot of people have to look at this things that might are not going to be useful or for them obviously um you know but the company needs the labeles to tr to train their rankers okay so far you know this looks very good but you know looks like a ton of work and um you know the original goal is to you know provide good labels so data quality and experimental designs in our case are kind of preconditions to make sure that we got the right stuff because the labels are going to be used for rankers machine learning models evaluations Etc so please do not cut Corners now we're going to move into the real stuff okay so so far I've been saying do this do that but I'm going to kind of put uh the money where my mouth is and I'm going to say this is exactly how you do it in real life and this is the framework that we use in production you can go to this paper information retrieval that explains this in detail but the goal is the following you start with a particular labeling task whatever is the labeling task okay so the first one is the Prototype development you have a very small data set it's your internal team five four people your internal team okay and it's mainly for testing so someone builds a design and you're going to get into the room and everybody will do labeling sorry guys everybody will do labeling for two or three hours okay now you do labeling you do your statistical measures and you say did we agree on it okay if you didn't agree back to the whing board okay you don't move it's not working very similar to building a user interface okay I don't get it where's the pull down menu what is this red what is a hyperlink if you don't do it correctly you don't move once there's agreement between within your team that the task is good the data they collecting is collection is good the labels are good statistically etc etc then you move into the next stage which is an early stage production is still the small data set okay but now we're going to crowdsource so instead of building the team you're going to the crowd okay and you don't care that much about the quality of the crowd yet but you want to make sure that this agreement between us the little team with the crowd okay and this is mainly for calibration so you're going to calibrate the experiment the incentives data Etc okay kind of works more or less if that works then you move into the next one which is then you go on production then you you are going to partition the data set because you're not going to upload a million documents to label in one shot it's not going to work okay you're going to Partition by hundred or thousands so so far but you're not going to upload a massive data set at the end you want your massive data set but you're going to partition a little bit of data every day okay you're going to do you're going to crowdsource this you're going to put all the quality controls mechanisms that you know we're going to talk about in the on the last section and you're going to continuously evaluate this thing okay so if I upload on Monday Monday afterno the data is done then I evaluate the quality okay if it's not good I do not upload the second batch if it's good then I move forward and upload the second batch but you do this over time and you move from design to Quality so you have to build a workflow that get the data uploads into whatever your C sourcing platform is comput all the metrics if the metrics are not working you stop okay if the metrics are working you keep doing this over and over and over okay so let me show you a real example this is a project that we did um a couple years ago on a real problem on a search engine how to detect near duplicates okay there's a lot of duplicates on the web and there's a lot of near duplicates on the web we and we thought hey this is very simple I just show you two documents and you have to tell me they are dupes or near dupes right wrong didn't work and this is a near dup so there's so the green the sort the uh the yellow stuff is the dup the stuff that is not yellow is the near dup and you have to read and you know there's a lot of cat and pay this might be a BBC document this might be a writers document they're talking about the same stuff and it's actually more as the same document and you want to show only one okay on The Hit List you don't want to show both you just want to show one or maybe you cluster them so how we solve this problem we use that framework okay so we're going to scale up by NE by uh crowd sourcing and we're going to use two crowd sourcing platforms UHS which is an internal Microsoft crowdsourcing platform and and mechanical turque and because we found that it was very difficult to detect and evaluate if two documents um were the same dupes or near dupes we partition the task and there are two tasks here the first one is is this a document about news because even something very simple like this is a news document it's not clear you may have a picture of Obama with a footnote and people say it's news you say no it's not news I need that two or three paragraph but if you say Wikipedia page that's not news so even deciding what's a news article was tough so how we start we have one task in Mechanical Turk that will say is this document a news doc document yes no if it yes that's a potential candidate for them building a collection of PE pairs so then we can detect if the pairs are duplicate or not if not we throw the document okay so um documents labels are uh the first phase phase one okay we sample the documents is this a news document or not if it's news we go to the second phase we show the two two articles and then we say this about the same story or not and based on that we collect the labels how we did this the three authors okay the guys wrote the paper three of us did this and did this with a small data set to make sure everything works fine everything works fine when that is working fine then we scale it up and then we use uh MK because it was cheaper to do this the yes and no in MK and we wanted high quality workers so we use uhrs to do the near dup okay so two ask two different platforms same data set at the end you want one label it's a near dup but here which is kind of partition into two two platforms and two tasks so two- face approach assess the documents um that are sample ASP news not news and then we we do a separate task to assess if the document is it's a duplicate or non duplicate you follow me more less here's the actual real designs and you can go here to this URL you can download all the source code and this is source code for the uh for the templates okay so the first template is looks very very simple um here are the guidelines this is what a document we consider to be a news document this is not a news document here's an example that you have to tell me yes it's a news article no it's not a new a news article I do not know there's a broken image a broken link I cannot tell and this is how we assess if two documents are near dos and if one is containing to the other one okay so far so good for just collecting labels that's the only thing we've done is collecting labels no quality yet how we assess the quality again another circles another kind of uh circuit sorry not Circle another circuit um so we're going to use um we're going to use our own labels as goal sets or honey Poots okay so from the first step we the authors we produce the honey pots the honey Poots are injected into the crowdsourcing platform was going to basically do the quality controls going to check again our stuff then there's a second filter on the labels of the near dupes and in case there's disagreement the authors will break the pars and do a majority vote sounds complex but very doable again we started with the authors we produced the honeypots the honey are injected into the set that goes into this case Mechanical Turk of what is a new article then all the assessments of the news article pass no pass then we create the pairs we also have honey Poots for the pairs if you didn't pass the pairs we the authors will resolve the this agreements okay so um looks like it works it's a fast turn around easy to experiment you know a few dollars to test but um you have to design the experiments carefully I mean this as you can see all these little things takes a lot of time okay and every little detail counts um you have to build the workflows script the data produce everything you can automate you're going to build a workflow but actually takes quite a bit of time and work so um if you don't design the experiments carefully you're basically Doom um you have Quality Pro uh you have quality problems you have platform limitation ET this is kind of the real world right the real life now crowd sourcing in production um if you want to scale to millions of labels millions of labels and you want to do this continuous execution you're going to face the following problem how to debug how to debug if your task is working okay so far I show you one task that kind of works but you're you're going to have the following problems how do you know if the task is working we came up with this framework um going appear on on a on a conference this year but you have three parallel problems the work may be boring the worker may be a spammer and your task or your task may be ill defined you cannot debug this in par okay the usual kind of reaction is the workers are bad remove them don't pay them or pay them more block them but that's not a problem so how do we deal with the problem we say well we need to establish a baseline signal the first one we're going to look at work we're just going to look at the data how is the data and we're going to vary the data genre recency news articles no news articles Wikipedia documents not documents we want to see we just want to play with the data and see if we got some signals or not after we've done that and say check the work is not actually a problem we're going to look at the workers so check if the worker is reliable we're going to compare against the crowd we're going to compare the reliability spam detection etc etc once the workers check are good we're going to look at the task are we asking people to do the right stuff you know can they do this can they do it in a separate way etc etc so I'm going to go through this the task is on social very very simple okay and how difficult this could be you know we're going to classify tweets and I want to say I want to build a classifier that will tell me a tweet is interesting why because there's a lot of noise there's a lot of duplicates in tweets and I just want to build an index that is of high quality that's all very very simple right it's a textbook problem collect the labels build the features put it in w w linear regression or whatever is your favorite classifier boom no okay we couldn't actually get it done the way we wanted because it's first of all a subjective activity uh and then we found that we we couldn't move the needle on the interr agreement it was like just not working I mean we were able to build something but it was just not there and we tested many designs two different crowd platforms mechanical ter uhrs and we just couldn't move so we go back to our framework and start debugging the task okay one by one and we came with this idea that we borrow from captas from Rec captur and we call this a hidden so if you if you if you're obviously familiar with the capture you have this control terms so one of those terms it's it's well known the other one is not known and you have to pass the test okay so we're going to use the same um so we're going to adapt the the con of a control um term to captas we build this notion called hidden you know human intelligent task datadriven inquiry so you can automatically build this things I'm going to ask two questions one is an algorithmic question so you can precompute the answer before you upload the test and the second is a semantic question that is slightly more difficult okay you're very puzzled but it's not that complicated okay so here's a tweet this is the real tweet okay so it's a tweet that is De branded so there's no phase nothing just a tweet 140 characters the main question is here but we don't ask this question we're going to ask it in later the first one is how many hashtag words so words that begin with a hash are in this tweet okay no hashtags one two or three okay so you have to as a work you have to read the Tweet say uh and then input your answer and the second is does it tweet name a specific person again you have to read a tweet answer the question and then yes or question question do you think the TR is interesting or not that's what we want so we enfor people to read this at least twice answer these questions and then you know what we're looking for I'm going to measure our interr agreement okay for question one is perfect agreement question two is a little bit less because you have to know that you know Paul Allen is a person Etc smell smell of Victory right no B okay it's just like what's going on well it's very clear so these are not spammers I mean people are just passing the questions but there's something else here so this question do you think that t is interesting to a brother audience yes no is not working that's what we want but for the workers it's not working so we're going to desegregate this okay so we're going to go back and say we're going to leave question one and question two but here we're going to use what we call labels for the humans okay are the labels for the machines yes no we're going to go labels for the humans and these are more human readable labels please uh check all the boxes that apply to this tweet could be worthless trivial funny Mi me curious contains useful information or it's important news okay we pass we pass and then we are getting some signals okay and I do not claim this is perfect but at least we know right that on the K levels Alpha levels sorry of important news and worthless hey we're getting something maybe this stuff doesn't cut it but at least I know that people can detect what is an important news and what is junk or crap that's good enough to build something but it wasn't good enough to say yes and no okay how am I doing in time sorry you have uh seven seven all right findings uh from the designs there were no quality control issues okay eliminating workers who didn't who did a poor job on q1 didn't affect the inet agreement on questions Q2 and Q3 so the idea of this is if you don't pass q1 and Q2 you're out okay I cut you because you're just not doing basic stuff um so we can still build the classifier that if tweets and are interesting to a majority of users not the best idea because we wanted something really really cool but at least we know how to pinpoint and identify the problems with the with the task okay into the last part of the talk which is the overall area of algorithms on work quality so how do we measure work quality we basically are going to measure we're going to compare the workers labels versus known correct and trusted labels other workers labels or we're going to build a model where we're going to predict the set of labels and uh it's also good to verify the workers labels as well you can verify someone else labels or you can use a tier approach so we use lowlevel workers for certain task and you know more experts to the other one etc etc so let's talk about uh a few design patterns one is called iterative refinement so this is someone does something and then you have another worker who is going to fix bit and do something and then you have another worker who going to fix on on that and just until you get this stuff right another one which is very nice is called Fine fix verify so this is by Michael berstein who has this cool demo project called soilent so you have one crowd who is going to find the problem separate crowd is going to fix the problem third crowd is going to verify the solution okay and then you have the other one which is do verify this is more of a tier so someone one who is an expert identifies a problem and then fix it okay so if I'm a rookie I made an error my supervisor say no this is an error this is how you fix it and you're done so s like usually the three patterns that people use uh to make sure that data the quality is on the task is there how you compare to known answers which is usually called gold honey Poots or verifiable answers so assumes you have known answers you have to produce these answers okay uh so you have to hire the experts they're going to create the gold or the Honeypot and then you have to match if I pass or I didn't pass this is very similar to a test now there's a cost and benefit the first one is you have to produce this which are not trivial and they're not cheap and and then you have to spend a certain amount of work kind of reproducing them because you don't have a go set of Honey pods and then you're done because people going to guess your honey pods so you have to always maintain these honey pods and then they have to be done in C in certain ways that people are not going to say oh come on this is a Honeypot then I pass the Honeypot and then you kind of watch TV while you do the rest another way is to compare to other workers so it's kind of consensus poity redundant labeling um and again you know cost benefits you're going to ask five people yeah we're going to lose a bit of money here but you know we just need to make sure that the labels are good I'm going to speed up a little bit uh um another one is to create a qualification test so I'm going to pre-qualify you this is good so I'm going to wait out people who are not um don't have the expertise to do my task but the disadvantage again is it's uh expensive I have to create a qualification test uh if might turn off people so it's like oh I don't want you to pre-qualify me I can I'm able to do the work you have to refresh the test you have to retest people again and again right and in C in certain cases like for example how how do I know that you're good at doing label um for for relevance is very very difficult to create a a qualification test what are the algorithms that people use in practice so you have the three references there but mainly a majority vote kind of works very well in a lot of the cases then there's a bunch of kind of expectation maximization algorithms as well as maximum likelihood all different uh approaches uh Panos who is a NYU he has this um an also an open source version of get another label which um is going to decide how many labels to use on a given hit based on the distribution of the previous hit so it's going to be an Adaptive then box POI um who is going to kind of aggregate labels as kind of approximate to a goal set of a ground truth and it's going to without people who are kind of off the goal set so it's going to remove poor performance workers and then programmatic gold this is a paper by the guys from cflower who are kind of spotting stuff that is uh not well done and then you just remove that from the from the data set Active Learning it's an area that you know could have a lot of benefits from this because what you want is you want to minimize the number of labels uh to achieve the the accuracy rate of the classifier so the idea is I'm just going to select certain subset the problem that I see with active learning in in label qualities if you don't get the right labels I mean this your Active Learning is not going to work um but also you have to you know make sure that you know the workers are not going to starve okay so you cannot be a lot of selective on on the number of workers because you're going to lose them and then you're going to be able to finish your task so there are suggestions of how to keep the workers occupied you know you pay for the waiting room so while you're are waiting for the next thing just hang on um you can paralyze some of the labelings and um you know you can use the workers to do other stuff in the meantime now things that we and Microsoft have been working for the last uh three or four years is this notion of up adaptivity uh which it's pretty cool so it's this this entire uh research area called explore exploit and you want to explore and exploit at the same time so it's a there's a trade-off between quality and cost so you want to minimize the cost but you want high quality this is exactly what you know real life people want to do and you don't want to spend a lot of money but at same time you know I want high quality so how we do this so we used again the framework called explore so a lot of Alternatives and to see which one is going to work best and uh you can going to exploit once you have an alternative that you know has been performing well in the very well in the past and you just exploit that particular strategy here's an example this is brand new stuff at I'm at Microsoft uh we call adaptive exploration so you say that you have the query Facebook and you have dubdub facebook.com how many workers probably one worker but for the query solar storms. website maybe five or seven okay so how can I kind of move how can I change how many workers and when to stop so you have to have a a stopping rule when to stop I get another label and if you do this very well you can build your honeypots automatically last last slide take away uh repeatable labeling quality at scale works but requires a solid framework if you don't have the framework not going to work um there are three aspects that you need that need attention workers work on design it's important pay attention uh labeling social data is very very difficult compared to web data like a web page or a traditional information retrieval document it's important to know the limitations your own limitations and be ready to work with other folks so there's a lot of skills that are needed okay so social behavioral scientists human factors algorithms economics in particular how to do the incentives distributed systems and all the statistical stuff and I'm done that's my Twitter account thank you very much