Label Quality in the Era of Big Data

Organizations that develop and use technologies around information retrieval, machine learning, recommender systems, and natural language processing depend on labels for engineering and experimentation. These labels, usually gathered via human computation, are used in machine learned models for prediction and evaluation purposes. In such scenarios collecting high quality labels is a very important part of the overall process. We elaborate on these challenges and explore possible solutions for collecting high quality labels at large scale.