Text By the Bay 2015: Joaquin Delgado and Diana Hu, ML Scoring: Where Machine Learning Meets Search
okay I think uh we're uh ready to start um let me first introduce uh myself and and Diana the speaker my name is H Delgado uh I'm the director of engineering at uh enq uh Verizon enq is a Internet and mobile TV um product that we will be launching soon um interestingly I'm I lead the group that does advertising over the U of the internet for internet TV as well as recommender systems I'm going to talk today about some of the learnings that we've had with respect to the intersection of machine learning and and search and kind of commonalities that um we've tried to address uh with a a a open source product that we or proof of concept that we're launching this this is just the legal stuff uh that that lawyers want to see you guys can read it yourselves um so I'd like to start with talking a little bit about the the words that have been used to describe um what I what I believe in general um a a the the the retrieval problem of fetching uh data you know we have heard U obviously the word search is used everywhere so search what does search mean really and how does it differentiate from information retrieval which is the original kind of uh Nemesis of search and what what how is it related to machine learning so a little bit of that as background so in general search is uh like a method for solving what I call or it's what is called constraint satisfaction problem so the objective is really to find so search is the this action to find all possible solutions right uh given a set of constraints and if you think about it in in terms of uh the traditional search As We Know It uh the objects would be documents and the queries would be the constraints and the variables would be the the fields that you're you're searching on uh now that doesn't necessarily imply ranking or scoring it's optional and sometimes rank ranking is needed to find Optimal or faster uh uh the results in this space that you're searching on I give you an example of a generic search problem a constraint satisfaction problem would be the n Queen problem you want put on a checkboard u n number of queens and so that each of the Queens cannot U interfere or eat eat each other so it is a u problem that is not easy to solve in fact as you can see here the number of possible solutions and then the number of Optimal Solutions uh grow exponentially and um therefore the as the space of that you're looking and you're searching increases you need opt uh optional scoring or ways to um to wait what possible uh Solutions are more important more faster to reach where are optimal in terms of the number of of operations you do to find the solution um if you map it back to what we usually talk to as search or refer to as search it's really more the information retrieval uh point of view which is yes it is search there's some constraints there's some uh results that will satisfy those constraints but then the scoring or ranking is really applied to the domain of text and that's the traditional information retrieval uh you nem the the the it was born as that field that would be able to rank and score text in a way that would be more uh efficient and also would mean more for humans it would be more relevant uh ranking than anything else so in in that sense the solution that uh information retrieval uh came up with is really representing uh documents a bag of words and then ultimately as vectors in a in a n-dimensional space where each term and you know most of the people are uh familiar with this uh the tfidf model would look at the term frequencies in the document with respect to the query terms and and the inverse document frequency with with is uh how how common are terms across all the documents and then weigh them differently this creates a weight so that when a query term is posed I could find uh where the distance between the the query represented as a document and all the documents in the space in the solution space uh and then calculate the kind of a distance cosine similarity there's other other approaches like language models and also uh probalistic ranking but all of them try to produce a uh model uh that would mimic uh relevancy in in the sense that it would try to find what how close is the query uh in terms of text similarity to the uh documents that I'm trying to search um machine learning which is also used for solving the search or retrieval problem particularly for ranking is a little bit different right we particularly uh when we talk about supervisor learning uh we have uh the task of predicting labels for an unseen object based on the training of uh instances of of that we know the labels for so basically it's an algorithm or a set of algorithms that you you put as input a set of instances whether those are documents images sounds whatever they are you extract features you run them through this learning algorithms also feed them the labels for those instances and it creates a predictive model now interestingly this approach has been used for search and the information retrieval uh Way by saying okay can I compute relevancy using a predictive model but it's more generic than that because we know that machine learning can be used to classify pretty much any problem in the space of having uh the ability to extract feature vectors and then predict on unseen or new objects that I haven't uh um seen before using this predictive model to generalize and and and perform that classification task uh by similarity with the instances that have been learned um so and when you do that with the task of producing a number instead of a label it's called a regression task uh you effectively you're not only predicting you're really predicting a probability or a number that represents uh a a a in a in a scale um the a score that can be used for ranking so effectively regression is a way to use predictive modeling for ranking I'll come back why this is interesting in in the scope of this talk uh this is just an example of what a model would be in this case we're uh uh talking about a logistic regression that effectively is a better way if you have some thresholds or a binary classification to approximate uh a set of instances you have in in a space uh you have probably have have seen look linear regression which is really looking at in a linear model just a uh uh the the distance between the instances and in a linear uh a line in in in the in the uh space here it's a little bit different because it uses more approximation of a curve and then it it doesn't exceed the threshold that you're looking for to not exceed um but search is not restricted to text if you come back to the original constraint satisfaction problem we see that that all over the place right we see it in database systems where you effectively a SQL query is a statement of constraint you see it in games in AI systems like the queen uh you know even chess the way chess is played is is a really a search in a a variety of space of solutions of the different moves that a person can perform um in e-commerce systems as well because now the task is not to rank based on the similarity of text but other factors such as distance if you're doing mobile e-commerce you want to show the products that are closer by or maybe you know Geographic if you're doing a search on a um let's say on a Trulia or any of the other uh uh Real Estate websites so now the objective or the ranking function is no longer uh the traditional information retrieval tfidf recommender systems now introduces yet another factor of um uh user similarity as well as contextual similarity for example nowadays Sim recommended systems use things like time of the day or effectively where in which website in which section of your app is the recommendation being produced so you don't no longer just have u values of of data points or features that are that are related to the objects you recommending you also have other features that you add to the process of learning that can allow you to U score things uh differently and finally advertising my favorite topic is really matching or being able to show the right ad to the to the right user at the right time it's a Confluence of many things people uh talk about advertising as a search problem but it's also a uh economic problem because you want to maximize returns and if you're a publisher you want to maximize the am amount of money you can make if you're an Advertiser you're trying to bid for a uh for showing an ad that is relevant to that user that user in particular particular and and being able to get a return in terms of clicks or conversions so there is a whole Marketplace bidding um uh ecosystem that really is a different way to rank uh but it's still doing pretty much search in the sense that it has to weigh down and and and retrieve a set of ads that are relevant to the user prior to ranking um so why is this uh for me very interesting because it turns out that when I in my experience every single one of these um problems have been ultimately tried or solved to some extent with something like a search engine um why let's say why are search engines so powerful and can be used to attack most of these problems even if you have your documents or your information in traditional databases or no SQL databases a lot of the times we represent and we use either elastic search or solar or some of these search engines to perform these tasks first of all because they're widely available you have elastic search solar they're open source they're fast and scalable they a distributed system that can you know load and Shard the indexes and be able to do it at Mass scale and it integrates well you can just suck the data get out the data from the different repositories and then load them and represent them in a very generic way in Json or that represents documents and Fields and and it's it's very easy to use use um what's the problem though so search engines were originally designed and let's not forget about that Lucine which is the underlying um uh engine or or library that is used in both enas search and solar originally designed for information retrieval so the scoring mechanisms that there it contains is is very much tailored to the original problem of text um sometimes and I've seen this again and again people use uh let's say the search engine as one phas retrieve the documents retrieve whatever the objects that they want to um find and then do a second phase of uh of actually scoring so let me illustrate that so in information retrieval a ranking the original ranking functions right now used in elastic search just to come back to the uh why we may not use this scoring and why may it not fit our our our objective is again very much using the tfid F model yes it has something different you can do boosting so that's why you can do boosting on some fields or some specific uh words and you can also do normalization based on size of the document um and and so on so forth but in essence it is looking at term frequency document frequency inverse document frequency so what is the tlex uh face approach that typically um you know we perform filter and rank so what is the problem with this um many of the times the way we filter retrieves a large number of of documents that well we can do some scoring on but if we do a complex scoring that requires say for example a machine learning model then there's the alternative of doing the retrieval based on some proxy score maybe the original text scoring engine would help us retrieve a subset remember we have these constraints that you want to follow and then off of those top K we the ranking what happens here is that when you have constraints such as budget for advertising or you really want to look at that particular um uh relevant uh recommendation that you want to show to the user and you won't know until that point in time then figuring out what is the best topk based on an algorithm that is not the one suited for the task becomes a problem and therefore there's always this idea or or kind of the the the uh the best case scenario is that oh can we can we marry those things can we marry the search as a retrieval function and still be able to use something like machine learning that is generic enough to produce the scores in such a way that I don't have to do this Two-Face approach so we thought about this problem and we thought about how to solve this problem and um which is applicable to any of these systems advertising system recommender systems um and so on so forth so what we came uh uh uh to do and again this is still a pretty much a just a disclaimer a prototype and it's a it's a proof of concept but you will be able to go down and download it and play with it after this is well there is a concept of a plugin within um um elastic search and solar has the same concept for scoring so yes it's not something new to be able to score based on some function that you you deter determine what we thought it was cool is what if you can just plug in any machine learning algorithm and what if you can use the machine learning algorithms that are existing and available for us today and then we created this uh kind of three-phase approach where you can create an elastic search index off of any document that represents your IND your instances that you're trying to work on you can train a supervised learning algorithm um from that subset or a subset of those instances and then it generates automatically a plug for you so that you can use elastic search to you do any constraints that you want to do to narrow down the set of of valid um uh instances but it also does that scoring based on that machine learning algorithm you selected to do the scoring for you so in other words it tries to do what I guess in an internal scoring based on a Model that is machine learning based instead of the tfidf information retrieval base uh so let it Diana show you the steps and also a demo thanks so here we did a couple implementations of the ml scoring with the idea that hen was mentioning so I mean this is still very rough code it's mostly the interesting idea of kind of hacking your search engine to use it as your online evaluator for your data which typically people don't do so you save some amount of code in the service so the idea is that you first have your data some sort of data that you're going to use offline and it has labels then you train it and also index it in your search engine so you could have all of it your train create your model and then once you have that you have to be able to serialize this ml model and be able to load it up later into your ml scoring plugin so you have at the end of the results of the this demo you have uh the instances in the sense the data set in elastic search loaded and then then a second part is the ml the machine learning scoring plugin which has which takes in the serialized ml model and have it creates a binary sort of like the script that elastic search takes in and can do any kind of evaluation so you could take your model that you train in on let's say an svm then you serialize and you learn the weights and the intercepts and then you apply that create the binary and then it creates a plugin and and one of the interesting things is that during query time let's say at online evaluation the this is a very specific to elastic search you have this concept of passing functions where you can pass a script and they have various ways of passing script in different languages they have native which it compiles in Java because um elastic search is all implemented in Java off of Lucine uh but it also supports other scripting jvm languages like groovy um the implementation we have shown here here we are using Java um so this is an example of a query that you could then at the at the query level you could pass in other the whole idea that H was mentioning the constraint search constraint constraint search part of it where you could for example um do a you have a spot where you can only do certain kinds of advertisement based on let's say this is a category that you should only show to adults and that are certain age and then you satisfy that so during evaluation you would only evaluate it into the instances that match that so it allows you to do the filtering plus the ranking using the scripting score so and some of the support that we tried out is like okay so you could um what about if we implement this you could train any model in W and then load it as the binary that I mentioned so during runtime and evaluation time you pretty much have a trained W model that evaluates your your score for for your instances in elastic search and then we said okay so the problem with WKA is that you could only uh train on models that fit in a single machine so what about uh trying the same idea with spark so you could train a model in a lot of data and then serialize the model and then somehow load it up so during evalution time elastic search uses the train model in spark and right now it only works for linear models and there's some details on that on the code on why is but just by trying out the implementation in W and Spark it doesn't mean that you're constrained to just using these two it's just the idea of being able to somehow represent and serialize and load the binary code in elastic search jvm you could pretty much do any library and machine learning any train model and um so that's that's part of it so let me show you a little bit right now uh a demo of the code I mean right now let's see actually let me show the next slide first so let's see so here we have um the way we configure we're we're taking this data set from UC arvine the UC Irvine ml repository there's this data that has the census data and it's a classification problem where it's trying to predict uh either uh the probability of someone making more than 50,000k a year which is this last number here zero or either one and you have the all the different features and as you see the features are either numerical or they are categorical uh for example H is obviously numerical then you have work class which is either state government or self-employed or other um so we taken this data and create to train the model and I'm going to show you the spark example so here the way we implement that there's this configuration file where you specify here the data set options where the data F data resides then columns where you can select uh subset but maybe you don't want to work with all the features so you select some of them so for example here this education num we're not using that we're just using education you select a subset subset of them then we have all the other trainer options so here we select a classifier spark logistic regression there's other ones that are implemented for example spark um um linear regression there's also spvm among others uh and then specify sort of all your traditional supervised training task the training percentage so you save 20% for testing uh then you want to do cross validation and here model file name is the custom serial serialization that we did for spark but for WKA you could save it as um The W arff file which is just loaded but for this case we save it as a Json file and for linear models the nice thing is that all you need is just the weights and the intercepts and you could calculate it on real time and here are just the elastic search options where you load uh which point the Clusters you point all the details of your host name the name and create the the name of the index and here are specific uh two spark options we're selecting the logistic regression model type it's not a regression because this is a classification task and we are uh there's no need for a binary threshold because already zero and one but here we can specify custom things about the particular classifier here for example the number of Ovations for the grading descent we just set 30 so that we finish on time so then the regularization parameter and blah blah blah and number of classes and here's some other details of specific to configuring your spark cluster here's we just running it locally here the driver memory um so let me show you then we're going to fire up a elastic search cluster locally here and then I'm going to show you that we don't have a index here is local that is empty right now you'll see here that I don't have anything stored here so the first part that I'm running here I pre-compile already the binaries here because to make it work with spark you need to make a fat jar and it takes a while to get all the dependencies in spark so it's all pre-compiled but you could try it out in the code so we just run the run it here with the property file that I showed you with all the configurations for the training and indexing and where the data lives so I hit this and then here's running and if we look at here it's creating all the index for the data it's creating blah blah blah you see education hours per week BL and I think right now it's firing spark and then we should be able to see here the Local Host uh spark UI and as you see here um stages zero to three is where you're loading the data into an rdd and you could at some point read it also directly from elastic search but right just reading it locally from the CSV then here's running all the tree aggregate are the specific iterations during the SGD um optimization algorithm for for the logistic regression just running it and I guess to finish right now because I wanted to finish on time and here we have the results and the results are okayish because we wanted to finish on time so we didn't let it run for too long and you see here the results of the training Precision recall and here we take a look at the data in elastic search actually we could take a look here we can see that a index got created called demo which is what I specified in the properties file and it has the number of documents index there and it doesn't Index right now whoops if we take a look at the data here you can see all the columns that we selected and obviously it doesn't have the ility because that's our prediction that we our Target variable so we could foresee in a production environment you wouldn't have that score and that's the actual score you're looking to use for your ranking so now the second step of this is once we have created we need to um create the plug-in which is another jar file with all the dependencies that get with the binaries and we need to install it into elastic search so some of this is not documented some of is documented in elastic search was reading through the source code to figure out what is going on but um here um again I think so here I'm passing the promile plug right here uh so I'm installing here search predictor is the plugin and I'm passing the file that I'm already pre-compiled this uh passing the plugin that got created after you pass in some of the configurations here that once you create the model it points to where the model lives this is the model file that got generated these are all the mappings so we know how to handle the types for example you saw some things are string some things are double and the specific uh Spark um trainer so here we installed the plugin so installed it and then as you see some of the files that got generated demo model is the serialize spark model that got trained and to make it work with spark right now it only works with linear model because I can I looked at the the source code in ml lip and you can stract the weight the intercept and that's enough for a linear model to do a classification so then you don't really need to run a spark context in your elastic search because it gets messy so all you need is just this data but other things I'm kind of investigating is how can you if you have models that are distributed how can you have a elastic search and also have a spark cluster running and do the evaluation or maybe not but the general idea is that you could train on spark and then evaluate on elastic search so once we install the plugin I need to actually restart the elastic stretch cluster and then notice here when I restart it there's going to be something it's going to say plugin something plugin and this is the plugin we just installed here something something plugin and then we could do the query here I think this is too small right so I'm going to do the structure query is the query that we showed here in the presentation which is using here the search predictor plugin that we asked and I'm just matching all just interest and time in the index here we call the demo so right now it's evaluating for all the document is fairly fast and here you see the score either one or zero but since this is ranking by order I guess it took all the documents that match one because it only shows the top five right now just the way the plug head end works but then it shows you here I as a debug here I'm printing out all the conversions of the vector this is the actual feature Vector that gets generated that gets passed to the the weight multiplication for your linear model and I think this is it for the demo and I think the code is still very raw but it's just the concept of having um the proof of concept that this could work just kind of hacking your elastic search and not using it for for the interned purpose you're just reusing the scoring function instead of your traditional tfidf you could pass in a custom function of anything and this Custom Custom function is really could be a model because at the end of the day when you have a supervised trained model it just a function that got learned so you could just apply the function and use it and the way it's implemented right now there's certain things that could be changed a bit a little bit different uh for example is the way right now is like loading the model every time you do aquer but if you had a model that you're very sure about it you could hardcode the values in there the actual weight so you don't have to load and sterilize dilize the model every time uh and then that's really fast but what you lose is that the nice thing about this once you install the plug-in you could swap the model file as long as your features doesn't change then you want to re relearn the weights that's fine and you don't have to reinstall the plugin and it just works and part of it the I guess what is a bit part of this work is a bit um if you see the code there's a Scala code doing all the spark spark abstractions and there's there the Java code to do the plugin and you can see the inter operation of having Scala code that gets called in in the plugin and it works it just works so that's that's it I don't know if you guys have any questions oh we do have some potential issues that we want to discuss that uh okay so this is as as as I mentioned this is a bit of a hack there might be a performance issues if your search Bas is very large and that's why you want to do the constraint search approach where you limit the amount of documents that's going to evaluate and pass through this custom function uh that's one way of doing and other things that are definitely very not not not for production operations is that how do you do versioning and binary B binary compatibility because if you change your plugin and your data data model changes things get very messy and at the end of the day you're really changing your database at the database layer you're really going really low level instead of having a service chilling you off of that you're really changing the this your database so that's things that you want to consider but it's trade-offs that you want to do but it's just a it's an interesting idea that we explored and um if you have any questions right now we're open goad um how do you uh for what you're doing because you're partitioning across multiple machines and it would only apply to that chart but it would use the machine learning model that was learned and it would not use the tfidf score so it basically replaces the tfidf score and it uses the score from the model to predict the actual rank now here it was very simple because it was one and zero but in regression task it could be a probability it could be anything so um each runs the model on that set the way it works is the plugin we're really just leveraging all the work that elastic search and um Lucine has done yeah and Cloud so you get it for free they do all the optimizations and there's this kind of score multiplication that they already do anyways for retrieving documents is the exact same instead you're kind of injecting this custom function to right yeah but the cool thing is that it could really apply any machine learning algorithm that you you you want uh I I don't see why it doesn't work I mean I I kind of prototype this and it it works if there's a something with jbm we could get it um it's just a matter of matter of serialization and there's things like jython that you could do and pickle it and jython and it should work I me it was really rough but it works it's just okay any other question yeah go ahead so we're not tackling the problem of feature engineering anyways you need to do that to create your models one interesting thing that I did not mention is that not all the features need to be in in the document so you could pass some of the features as part of your query so if you have a real-time feature that you don't have it as part of the document a good example of that is an advertising we use a lot time of day or you know because depending on the time of day people have different moods and and appetite for advertising or not so that would be a query based feature but still the model you can train it with that feature it's just that it's not part of the document it can still use as an input and then evaluate the the learning so the the problem of feature engineering but you could use any any real machine learning algorithms if you wanted to use I guess uh yeah I mean part of the challenge is also if you are doing additional at query time if your uh feature Vector is not in the same space of space as in the same dimensions if they have normalization or some conversions Transformations if they don't have it you need to add it in your in your predictor engine to in the binary and of course it takes longer but if you have simple things that you just taking the data as it is you could do that and as work mentioned that's actually something that you could do uh you train on a document that has an additional column or feature Dimension like as you said the time of the day that uh at query time your documents don't have that but in your training set you do you could fetch that from the service and get that and add it in as part of your query and then still get the results so it's quite flexible in that sense it's not just restricted to the feature vectors that are in your document set in elastic search you could add in additional Dimensions some of these algorithms also handle very well uh missing values so that still applies if missing values are handled by the original algorithm this would handle it as well because we're not Reinventing the algorithm anything else okay oh one more so you have to so we we created an API that then you can Implement uh with any function as you said that that takes an instance it's very similar to the concept of instance evaluation by W or by even by by any of these machine learning algorithms it just takes an instance and then communicate and then that instance the the fields could be read from the document which is what we do or it could be read for the document and the query and then pass to the evaluator and then it returns back the score and then that score is what the internal plugin returns back to elastic search well if you want I can show you a little bit uh the interface I mean it's still very I guess still need to work a bit on the documentation to be more useful uh of time I this is available in GitHub so you guys will be able to uh yeah so for example here this is the plugin which this is the script that gets called in by elastic search it extends this um abstract interface abstract search script and part of the trick is that you need to it has this function doc which is what it gets returned for every query and what you need to make sure is that when you get the doc is obviously Json and Json doesn't have order so you have to make sure to be able to get your features in order because you train it for example age is the is in the zero and I don't know work class is two and things could be swapped so this is part of it what we do to make sure that things are in order and provided and it's the same input output for your training and testing I mean evaluation um so that's mostly it I mean still this is not not meant to be used in production but just a proof of the idea of uh not using search as the intended but well not at Verizon um but obviously um the idea the idea this is more of a generalization of an implementation that came that is used salware and that is is being used in a large e-commerce site that I cannot talk about thank you thank you for