data.bythebay.io: Michelle Casbon, Using Spark MLlib for NLP
Recording: data.bythebay.io: Michelle Casbon, Using Spark MLlib for NLP
yeah so that's kind of what we'll be talking about today one of the things that we built at any bond it's called EML so I purposely chose a generic title using spark for ML lib so what exactly are we here to talk about we are here to talk about the construction of predictive models trained on features extracted from raw text what exactly does that mean that's kind of a mouthful essentially what we're going to be doing is turning text into numbers doing some math on that text and doing some math on those numbers and then turning it back into text why would we go through that process it sounds a bit tedious what does that actually buy us what's the point well there's a few things mainly time we want to save time and we also want to be able to look at things that haven't necessarily happened yet so having a peek into the future a couple of things like a save as time are let's say we have if anyone was here at last year's text by the bay there was a talk by Rob Monroe and John awkward and they talked about doing some classification for Edmunds they received edmunds com receives PDFs from car manufacturers and they were able to read that PDF extract extract components of that and put it into the right form the right column in a database that saves a lot of human time from reading a PDF and putting things into a structured format if anyone has used an IVR system if anyone has made a phone call and been really frustrated by the person not being able to understand you speech recognition that needs NLP as well gentle pecan really the timing is very important in those systems let's say your house is on fire you don't have time for an IVR system that doesn't recognize your voice SMS prioritization that something that we've done for UNICEF they receive a lot of text messages from their community and they have a very limited number of staff to respond to that so NOP helps them prioritize those messages and bubble the most important ones to the top so that they're not wasting their time multilingual news can help us understand where companies are can help us understand what's going on in the world if you have a brand or a vertical that you're in there's no way to read all of the news that's coming out about that topic so i hope you can help us there if you are a video game company who puts out code releases every two weeks your players might be sitting on a reddit forum talking all about what's broken or what they really like about the new characters that just showed up you probably want to know about that NLP can help us there as well it's also possible to find a correlation between what people are saying and their intention to buy a product if you're selling anything it can be very helpful to predict the demand for your product this is where NLP can help so in order to do all of these things we need one really important piece and that is a prediction prediction is really the heart and soul of this that's the entire point and that's what we're looking for the prediction is really important because if you aren't using the actual data that you want information about you could be flying blind you could be listening to your gut you could be making decisions based on something that's not even real you could be completely guessing so how do we get to that prediction and how do we get a really good prediction well this is where the math comes in what we need is a recipe or a formula something that can help us find a prediction that makes sense with high confidence I won't go through this math but this is an equation this is the logistic regression function and the top one you can see there a couple coefficients we're looking for that a and that be and those are all the same equation we're looking for that p that p is our prediction so obviously we're not going to be computing that ourselves that's where ml loop comes into the picture Emily will do all of that number crunching for us and all we have to do is give it some training data what is training data training data is what mo lib uses to find out those coefficients to build a model that is representative of our data set there are different ways that you could generate that training data I won't go into that too much now but essentially training data is just an answer key it's it's a bunch of documents in the domain that you're looking to make predictions on along with a label so whatever question you're trying to answer whatever that prediction needs to be you label your documents according to what that should be so you send em a live your answer key and so you annotate all these documents you tell it what you want that what kind of prediction you want to see at the end and well not yet and oh and that model is little build a model for you and that is what you'll use to generate your prediction so you get all of your annotations you get all your documents together get all your labels and you send it mo lib and what does it do it barfs because it doesn't know how to understand languages what emily is looking for is a vector of doubles so how do you get that how do you turn that text into a feature vector how do you turn it from text into numbers this is the feature ization component and that is where a DML comes into play this is something that we spent a few months building and we knew that we wanted to use spark we knew the ML lib had some really useful functionality and that it was really fast but we also needed the feature ization component so we needed a platform that helped us operationalize that process so it's really flexible and could allow us to implement our own features use pretty standard ones and provide provide a persistence layer to do all of this in a scalable environment so that we could support continuous streaming predictions so what is an email its main function is to perform the three core pieces that that any NLP system needs to have that's feature extraction model training and prediction and we've already talked about these so a feature extraction is when you translate you transform your piece of text into a vector of doubles that goes into your training algorithm which is where you apply a label you give it the answer key and it generates a model that you can then use to predict on new text that you haven't used in your training set so in a bit more detail features asian pipelines can be incredibly complicated there can be a lot of different steps this is a really simple one normally I would do something a bit more complex but essentially you are taking your piece of text that maybe something with a lot of metadata maybe a JSON object with author information location information time all that usually what we care about is the the content field itself we may use those other fields as well but usually we want to just extract the content and the first thing we want to do with that is split up into its component parts so that's the tokenization piece in English it's really simple it's generally just white space and punctuation but in languages like Japanese and Chinese it or in Chinese in particular it can be very difficult and that segmentation piece is a very difficult challenge in and of itself so after you tokenize everything that's where you can turn it into an Engram you may be just looking at you know grams diagrams trigrams all any combination of those you put that together you add that to your feature vector and then that's still in the form of text that's where you do a lookup and you turn that into a number and that could that could have meaning it could just be an arbitrary index number but that's kind of less important but that's where you actually do the transformation from text to a number and what you get at the end is your feature vector and that is representative of your entire training set that is well there's one for every for every document in your set and that's what the training function uses the that's what training uses to come up with the coefficients for the logistic regression model so that's the model training piece that's where we take that feature ization vector that that large group of feature ization vectors and we add our label to it this is the annotation that we've collected earlier in the process and we turn that into a number as well we put that into a labeled point object if you've used spark you're probably familiar with that and and also the vector object that refers to ml libs vector object not just a standard it's called a object or anything so we get all that training data together and we send that into our logistic regression function and spark performs all those calculations for us it calculates that coefficient and it builds a model and that model is what we later use to spit out a prediction on another piece of text so we may use that right away or we may put it into some sort of storage layer and we call it later depending on what we're doing with it but once we have that model that means that we can make predictions on anything else in the domain we want to keep that in the same domain as the original text that we use for trading so when we get a new piece of text that's where we'll first run that piece of text through the exact same feature ization pipeline that we use before to get our future ization our feature vector and when we have that we go we grab the model that we built and we send it in we ask it for a prediction Emma lip handles all of that it just takes our future vector as an input and gives us a number at the end gives us a prediction and that's where we go back when we transformed that that label into a number we do a lookup to find out what that prediction is and we send that piece of text back to the person who asked for it those three components the feature is a shin model training and prediction those are really the the core features of it EML but really what's the point a lot of libraries do those three things and we could do that in a lot of ways so why do we even need a DML if that's all that we're doing well the point is it helps us operationalize it so if that's all we were doing on a local machine and we wanted to just make simple predictions for ourselves that would be fine we wouldn't need an entire persistence layer to do that but the reason we built a tml the reason we needed it is because we were we had thousands of models for all different customers and if you're maintaining that many models you really need a simpler way to maintain that because your ops team I mean it doesn't make any sense to to do it manually so you want something that can automate that layer so one of the first things that one of the biggest advantages is just that it puts everything into a single object everything you need for retrieving that prediction is all contained within just a simple jar file that means that's all of your that's the index that you used for turning the the text into a double that's all of the that's your future is a Xuan pipeline so that you know you can apply the same features a shin pipeline to your new piece of text as you did to all of your training items and that means that you can store everything you can put everything into a storage layer and when you go to recall it a prediction time if you have to rebuild anything you can retrain models you can do everything over again from just that one object you don't need to store things in different tables in different locations everything is all right there sorry I just the other the other advantage to that is that it adds a lot of flexibility and my flexibility I mean in terms of that that has a couple a couple of aspects your deployment environment right now I think we were we were using docker containers deploying on AWS images but if I don't know let's say the business relationship with Amazon goes south or they kill one of the services that we were depending on we could very easily because this was all self-contained and a separate module and it was separate from our deployment platform we could very easily move to a different cloud provider or do something else entirely use lambda or something else this is also in terms of the device the EML supports generic input streams and output streams which means that we're not even really dependent on having to write to disk so you could do this on a very lightweight piece of hardware somewhere or you could do this really anywhere you want it and also the logging framework is pretty important that insulates us from spark is very reliant on log4j but any Amell uses cell for J so it supports a lot of underlying log frameworks but gives us sort of one interface for integrating into a different application a lot of flexibility also in terms of flexibility what it does you do is use not just ml lip but any other library any other machine learning library that may come along let's say ml live dies in a year people just stopped using it decide it's not great or just migrated to something else we would easily be able to add you know the new flavor of the week in without sort of affecting any of our feature engineers that are used to writing features in this in ml framework it also means that we can take some of the cost of machine learning functions that we had before we started using spark we can plug those in easily as well and using the same feature ization pipeline we can sort of select those as a as a developer as someone building these models I could select any combination of those and combine them and run them in different ways so it insulates both your developers and it also gives you a few more options for combining things so the rest api is separate that that's an example of something that would use a DML Saudi balls is really just those core machine learning components and it's it's built to plug into any other type of system it's very standalone you would just call call that jar file with your larger application we will talk about that that is an excellent question and that's that's exactly why we did it so hyper parameterization we want to be able to automate as much of that as possible and the more options you have the better as long as you have enough computing power to support it all you really just need to add compute power and it's just like if anyone familiar with the cross-validation function in spark it's very similar to that but it sort of it's one level above so it's incorporating that feature pipeline and maybe other machine learning libraries as well absolutely that's a very good point and the question was you may have different implementations of logistic regression and those are sort of one group or different implementations of classification in general binary classification or multi class and then you may have some that do clustering there's maybe some lda and something else in there and so the way that we've implemented it is to to have inheritance and what you what you do when you generate the feature pipeline is you would specify that it has to be a classification okay a multi class or a binary classification and then we would when we implemented a new library we would inherit from that super class and and that's a very good point because that means it can be a lot more flexible and and that's really the only way to to support a lot of those to parameterize the whole pipeline so that you can swap in and out it's a very good point so another thing that it that's really important if you're working in an enterprise environment is you need to be able to roll back your versions we talked about having so models get stale when we're talking about text data languages change very frequently and you need to keep them up-to-date so we we use the wii something we call continuous training and what that refers to is anytime there's new data entering into the system whether that be new annotations coming in so new training data or perhaps some model parameters have changed any time any of those specifications change we would regenerate the model and if you automate that it means you can kind of set it and forget it you don't have to worry about your model too much especially if you're using automated hyper parameter tuning because it will check the accuracy and can adjust parameters accordingly but let's say you add some data into the system maybe our friend here has entered in some annotations that were a bit funny maybe his definition of intent to purchase is different from someone else's who knows but let's say that the model that gets trained is is actually less accurate it's not what we were expecting at all it would it admo allows us the capability to roll back very quickly to to call up the the models that we generated the day before or the week before or the year before and I talked about having that single object that's how we do it because we're storing those objects and assigning a version number it's really easy to recall that single object and replace it with the one that we don't want but version control also refers to not just the model itself and the parameters that go along with that model but also the code that we use to generate that model that object that that jar file and we refer to that as an alloy I should probably use the name and a low just encompasses all of all those parameters if you if you write a feature ization pipeline and and let's say it's using sparks logistic regression with lbf yes so that's one way of implementing but then a month later we find another implementation that that gives us much better accuracies and we want to replace things so maybe we make the decision to to make that the default instead well the feature pipeline doesn't really know about that because of because of the abstraction so we include a version number in the future pipeline and that helps us determine which code to actually run because a feature of pipeline is really just a JSON file that defines a class name and that class name might be a a few layers above the actual implementation of the actual algorithm that we want to run so that version number will help us decide okay in version one we were using this as a default in version 2 we were using this as a default and the cool thing about specifying it as a JSON one of our co-workers Gary wrote this part it is really cool it takes that class name in and reifies the object and that's how it builds the actual byte code to execute but it's really neat the way you can specify the how the model gets trade so versioning is really important especially for enterprise customers they get really really upset when you touch things even if it doesn't influence anything and God forbid if it does influence anything you better be able to roll back to the version that they like so this is kind of what we were talking about before this gentleman had a question about being able to sort of pick and choose different components within your system we want to parameterize as much as possible and give ourselves the opportunity to try out a lot of different combinations of those parameters because that can have a significant effect on the accuracy of the underlying prediction so hyper parameter tuning refers to some of the parameters that you might send in to sparks logistic regression function so some numerical parameters that affect those how those coefficients are discovered or it could also refer to different feature pipelines so you may have a few that you really like that are pretty general purpose that they give have given you good results on other models and you want to try those out you're not really sure which one would work best with this one so you would supply in your JSON not just one feature pipeline but several and a DML supports cycling through all of those and also using sparks own cross-validation feature which cycles through the spark specific parameters algorithm specific ones and what you get at the end is a model that based on your cross-validation results you expect to be much more accurate the flip side of that is that you need to be really confident in your training data because you don't want to over fit your models this is cross-validation and so you can't I mean it's training data test data the difference but it will at least get you pretty far and it automates as much of that as possible so you don't as a data scientist have to build models over and over again supplying different parameters each time that gets really tedious so let's talk about performance one of the really big advantages at least for us compared to our old system and we built this because we needed something that was we needed something that would give us thousands of predictions per second that we could scale up easily and we had that before but it took a lot of compute power so when we refer when we first started building this we knew that we wanted spark to be a part of it something out but we weren't exactly sure what that would look like so we compared we looked at data frames because data frames you know it's the new hotness I guess not anymore but at the time it was a new hotness and a lot of people were encouraging us to use I mean it has a lot of extra functionality data frames are really great and I was sort of used to using data frames in a batch processing environment and they make perfect sense there but our use case was really to take these really small documents and return individual predictions as quickly as possible and the timings between those two between the underlying just the the core vector object and the data frame it was significantly different so to a data frame is is built on top of a vector vector underlies it and so there's a lot of extra things that are going on in order to use that data frame so in so one prediction using a data frame took us took us seven eight hundred microseconds which was so almost eight milliseconds which if we wanted thousands of predictions per second was a non-starter for us it just wasn't an option so what we did is we looked at using the vector object instead and sort of skipping like the core smart framework and going directly to the ML lib functionality and that got us in the range that we were looking for that was much quicker and we're in a couple of benchmarks comparing the two and it was exactly as we expected you know in the in the smaller batch sizes which is the case that we're trying to optimize for there was it there was a significant difference and we saw much much faster performance with vectors as opposed to data frames and obviously the larger your batch size is you the larger batch sizes are the less of a difference there is and and that's very much in line with the idea behind spark it's meant to process large amounts of data very efficiently and and it does that it absolutely does that but for us because our use case with more of the single prediction it made a lot of sense for us to use a vector and that did mean that we had to expose a couple of private functions because Emma live wasn't exactly built with our use case in mind but we also you know we also kind of found a couple of things that we were able to improve and send back into the code base so it's I think as long as they don't go in a different direction entirely then it makes a lot of sense to use the vector instead of the data frame and this is this is like how dramatic the difference was for us so we were using jruby and some custom libraries that were on top of that and so you have the J you have the Ruby layer you have the JVM layer it's a lot more difficult to to run those basic those core algorithmic functions with with all that so with our old code base we had like two dozen servers to to do you know a pretty hefty volume of predictions but that same volume of predictions which was roughly equivalent to the size of liquid or fire hose in terms of word count we did all of that on it wasn't this laptop but it was a macbook pro from like last year I mean a not even a brand new laptop just consumer grade nothing nothing fancy and it didn't even come close to tapping out the CPU so it made a huge difference for us and email was kind of a game changer and yeah performance was was a big deal so what else do we want to do there are a couple things we need to need to generalize it so there are there specific touch points that were your number of touch points that were very specific to the larger platform to that rest api that was calling this and that could use a bit of generalization it is stand alone but it makes some assumptions about the way the annotations look the way that training data looks whenever you import it into it so that could be a bit more generic we also want to add some statistical models right now we're using binary and multi-class classification but we want to add a lot more to that random forest there's I think we have some clustering in there right now but we want to add different types of clustering we really want to expand the choices that we have during our hyper private or tuning we also want to support across the entire pipeline so so instead of specifying three different pipelines in the JSON we really just want to specify like a bag of features and have it generate different combinations of those features into individual pipelines and then compare that with the the parameters we're providing to the logistic regression function and that just gives us a it that gives us a bit more flexibility in the combinations of parameters that we can use and then also adding more features to it one that I was working on this weekend was a Chinese segmentation because I came across a paper that described a really a really interesting approach and that's instead of her Chinese segmentation you normally would look at a lexicon and look at you know the the makeup of certain characters statistically that forms a word but instead of doing it that way it's looking at the boundaries between the words and essentially using a classifier rather than taking Alexa conveys approach so even just trying out new things that come out in academia you want to be able to easily create features to test that and that's what idiom L is really good about a really good with so essentially it's a really flexible framework it's a lot faster than a fast framework that we were using before so it which means that it can support the goal of providing continuous stream processing at really high volumes and its language independent so you do you might need some language specific features but again those are fairly simple to implement and I really like using spark and ml lib it we started there and we saw some great performance I really love Scala as well that was kind of an added bonus so it is in Scala and yeah it's a fantastic framework so if anyone wants me to clarify or go into a bit more detail all over the floor up to questions so I think you may have touched on this a little bit but um do you have anything to sure selection face so you're computing by grants and programs you know you want to throw those out you this is still this life stage of black magic where you know figure out which features are actually the predictable yeah so the question is whether we have anything that automatically selects features that are important to streamline yes so that does happen automatically you that's that's sort of built into the code and I think the way we handle that is once we build those diagrams trigrams we we just have like a cutoff value so anything anything lower than that it's you know it's not too sophisticated it doesn't employ any sort of algorithm in that process but we've seen pretty great results with that in the past and I mean that's that's always an extension that that could fairly simply be added yeah so it uses spark under the hood like spark is a dependency and it just pulls that in it uses Gradle as a build tool it pulls in that dependency and just call it the libraries yes yeah so as long the question is instead of spark could we use other libraries and that's kind of the point we want to bring in as as much flexibility as possible to bring it as many as possible to compare them to to really be able to compare apples to apples and as long as it's supported on the JVM we can pull it in because it's Scala good great recitation of a question first light cream cheese yeah sure oh yeah sorry so so this was before we were using Eddie ml and we needed like dozens of service servers to to have enough computing power this was on our jruby framework with our sort of custom platform and then once we added a tml we were doing essentially the same process the same type of prediction on the exact same data set and we can do that on a single laptop like not even distributed just on a consumer-grade laptop and run the exact same volumes of data through and get the same predictions yeah so so we started with spark ml lib was the first their logistic regression function was the first thing we used so we were using a logistic regression function from within JRuby and and then sparks version of that but we weren't using the distributed that was literally just running spark locally on that computer yeah we were accessing the spark function thank you