Scale By The Bay 2018: Jeff Fenchel, Labels to Inference: A Continuous Sentiment Pipeline
Recording: Scale By The Bay 2018: Jeff Fenchel, Labels to Inference: A Continuous Sentiment Pipeline
penis pipeline cool I'm a senior software engineer at signal absent I'm gonna jump right in here so essentially we're gonna start about talking about what signal Labs does and how we help organizations be more data-driven we're gonna follow that up with why we felt it was necessary for us to rebuild a sentiment pipeline or a sentiment analysis API and then talk about iteration and how essentially we don't get things right first time all the time so iteration is really important to us and how we do this with continuous delivery and then we're going to talk about developing a pipeline from labeling all the way to making inferences so I'll apologize in advance you will see some Python here and we will see Python in production not the greatest thing but it's a little bit of the reality of if you have a small team today and want to get something and working in the real world so what we do at signal Labs essentially we provide our customers with real-time insights across the media spectrum to help them make more informed decisions basically today we provided organizations a platform to do things like see the share voice against their competitors after a major product launch or understand the health of their brand in real-time or even watch news spread across publications after a big announcement or even stop a crisis early through the identification of a trending story essentially we're really here to support these teams whether it's marketing communications executive teams or even investor teams across their entire workflow from understanding the situation to action so when we start to look at what different types of analysis we care about is in the labs we can kind of break it up into these main categories where essentially we start with descriptive analytics and we start to understand essentially what's happening right now and we can say what media outlets captured my brand in the last three weeks or how many mentions did my brand receive in October from there we can go into diagnostic analytics where we try to understand why what actually happened took place and this is where we get questions like what is the sentiment across my earned media or what events drove the conversation to get that many mentions and we can go even step further here and talk about predictive analytics trying to understand things like what is the likelihood this story will go viral or if I don't do any intervention how long will this story dominate the conversation or even how will my content be received and lastly on the final kind of tier of this where we talked about action we can talk about prescriptive analytics where we can start asking questions like how should I craft my messaging to leverage current events to get the maximum impact so we have all sorts of great questions we can solve here why in the world are we talking about sentiment when we start to build out a solution in-house we have to go through a lot of different tasks here from understanding the product to creating kind of the infrastructure and seeing what's out there to leverage open source technologies and then we have to maintain that code it's a lot of work and there's open source api's that will basically do all that work for us and we don't have to worry about it well the answer is we started with open or the kind of pay for api's and we use that in our platform and we got a lot of customer feedback first they said that sentiment was really important to them and it mattered second they said that the solution that we had wasn't working and we needed something better so we went down this path of trying to understand what about our data was so special or what about their perspective was different to where these sentiment solutions that were already available weren't working for us so we'd over into the data and sometimes it's really obvious about what sentiment is like in this case where we've got a great group of people making the world a better place obviously positive no question about it but other times it gets a little bit more complicated there is this notion of sentiment around emotion where sentiment is defined essentially by the emotions that are portrayed and in this case sadness it's a negative emotion and that becomes the sentiment but there's also another perspective here that we can take where we can ask ourselves well what does this say about John McCain as a person and lastly we noticed that a lot of our content was a state's straight statement of facts so when you're looking at this from this perspective it's a kind of news outlet that's just trying to be unbiased and present information but the fact is that when people see information that happened it could kind of taint our perspective about our companies about these companies that were reading about and this is something that our customers wanted reflected so in this example what is this statement say about Lehman Brothers so we kind of took this all in mind and we came up with several objectives that we wanted to meet on a sentiment rebuild essentially first we decided to redefine the problem here where instead of having a generic idea like sentiment where there's lots of different perspectives about how to score it we went for this polarity of reputation essentially trying to understand how a particular statement impacts or what it says about the image of a brand or company the next thing we wanted to tackle was data skew essentially we've got a lot of different customers monitoring a lot of different things in our platform and some of these things have a pretty high volume and some of these things are pretty small and have hardly any mentions about them but in our system we get customers that look at these small streams on their own so if we build a classifier that does well on the aggregate it can completely ignore the case of the minority streams and it'll have awful accuracy on that so to address this we went with equal sampling across all the data streams in our platform and lastly we wanted to do something about this trust and transparency problem that we had we kind of came to the conclusion or we felt that there was these expectations that sentiment was perfect all the time and we wanted to kind of do more to expose the fact that this is a machine learning solution and it's not gonna be perfect all the time so we wanted to offer this ability to kind of next to your historical sentiment charts where you see positive negative neutral and a number of mentions over time you can also see this well here's how accurate our sentiment is whether that's 70 80 or 90 percent accurate so this is kind of the high level solution that we're going to be walking through essentially starting with elasticsearch and starting with labeling and going to Amazon Sage maker we'll be jumping into each of these technologies a little bit more in depth in case you're not all that familiar with it so throughout this whole process essentially we like to take small steps so we can get things out there very quickly and continuously improve so Zimmy labs we do a lot of continuous delivery where essentially we codify everything in our deployment and make it very easy for us to continually improve on solutions moving forward and additionally we have some Python here which is you know kind of the scaling issues when you start having massive Python projects and we would like to keep things small so we have this deployable project pattern which allows us to do small projects basically micro-services and still have the leverage to be able to kind of reuse all this infrastructure code so it's easy for us and a few lines of code to say I want another lambda or I want another elastic container service so where I can deploy my docker image to or I want even another API just to spin up using all these kind of common deployment patterns that we have in addition like I mentioned earlier everything in our pipeline or everything in our deployment is pipelined so there's none of tribal knowledge around how do you deploy this into production or how do you deploy this in the staging or what should I check after I deploy it's all codified which is absolutely fantastic for us to make it easier for us to iterate so essentially we have this method of like developing code for us we have our ideas of how we want to make sentiment better and for all of this the kind of key foundation for machine learning right now is labeled data so we need a massive amount of labeled data and additionally we decided that we want to continuously have this so we can continuously provide feedback for our customers about how accurate our solution is so I work on a team of five people and if we decided to try to label this data ourselves we'd probably spend the next eight months labeling data and then we'd spend the rest of eternity trying to maintain the solution to provide those labels so we decided to leverage a crowd solution where essentially Amazon Mechanical Turk has basically serviced where essentially there's thousands of people out there in the cloud just willing to accept work for us so we have a process every day that wakes up and samples our elasticsearch data store which is basically where we have all our data normalized across many different sources whether it's reddit Twitter music blogs broadcast even and we take those met data and then transform it into easy questions that a single Amazon Mechanical Turk worker could answer for us and publish it into that cloud and then we've got a separate process that wakes up every so often and collects the complete questions where a complete question is one where five different workers have answered the same question and infers kind of a best-fit answer they're trying to kind of guess about worker qualities and we're biases using an expectation maximization algorithm and it publishes these kind of best fit answers into our last exertion coaster so we can have that transparency with our customers and use it to drive our front-end or our platform itself as well as to s3 so we can continuously improve our model going forward so important aspect here is our label data quality essentially our model is never going to be better than the quality of our underlying label data so we spend a lot of time trying to understand how it was going wrong and how we can improve it we care a lot about sources of disagreement between both our internal labels that seeing the labs in the crowd and between the crowd workers themselves and we kind of narrowed down this disagreement into several kind of key tasks here the first is task ambiguity where essentially not everyone kind of understood the idea of reputation polarity upfront or even there were times when arc X our instructions didn't quite align with the data that people would see so we did things like iterating on our instructions to hopefully make things more clear and adding extra options for people to answer for example we added multiple polarities for cases where essentially there was both a strong positive and negative answer offered at the same time so we essentially saw a coin flip about which one people were choosing the next kind of case that we came across was a worker perspective has you noticed we try to get sentiment on statements of facts which some facts can be kind of polarizing depending on people's prior opinions well this is very useful and you probably could dive into this for quite a while is about understanding how people react on kind of a segmented fashion for us we're not there yet so this is kind of a source of air and our models at the moment and lastly there's worker quality here not all workers are going to be as honest about answering questions upfront and they're not going to take the time to actually read through the data that they're answering for so this is something we had to be vigilant and filter out and kind of understand that this is happening and to a certain threshold we're okay with it but I've below a certain threshold we weren't so the great thing about the process that we set up here is we were doing a small amount of kind of labels daily this means that we can kind of start to understand what's going on and then correct the situation going forward after the first 50 or so days of doing this we looked back and saw essentially how many workers were coming back to us and if there was a chance that we could start to work with these workers to make them better and make the whole solution improve for us and we found that roughly 50% of the workers came back ten or more days and it's gotten even better since then so what this meant is we can start investing in workers with feedback we're essentially on a weekly basis we email them back saying here's what you answered for us and here's what our best fit answers are and while we realized that these aren't 100% correct you can kind of understand if there's a pattern here that maybe you should go back and read the instructions and kind of understand how to improve the situation going forward we also give them like an overall accuracy metric saying this is how good you've done over your recent tasks so we have all this label data and a really small team and we need to somehow create a machine learning model right now the state of the world is that there's a lot of really great frameworks written in Python so we picked a few of those off-the-shelf Kerris tensorflow NLT kay if you look at any of these frameworks and the basic examples you'll probably see a sentiment algorithm in there a lot of them use bi-directional long short term memory networks and that's kind of what we picked up for news and blogs we kind of leveraged our knowledge of what our customers were looking at in their data stream to figure out which specific sentences were relevant to our customers and we filtered those sentences out and then sent them through a neural network to get a final sentiment value for Twitter we did something slightly similar where we just used the whole body but we also included the link title because sometimes we found that that was pretty determinative and understanding the overall sentiment on the tweet itself so we have these wonderful Python frameworks they want great locally and we have this day's real-time data enrichment pipelines somehow we need to Union the two currently we have a bunch of data collectors right in my thigh and that goes into a Kafka has many people have seen architectures like this before and our streaming data enrichment is written in storm with Scala and this is kind of where we wanted to integrate we already had this pattern here established where we work a lot of third parties and they provide rest api is for us to integrate with and that's kind of the path we wanted to take here to be kind of a seamless replacement as well as able enable two different teams to kind of work on this project where this real-time data pipeline is managed by different team than kind of the data science II team we're on that so we looked around and found Amazon sage maker to kind of try to handle the Python code that we have into a production environment so we're gonna dive more into each of these but essentially we have lambdas handling each kind of section of the job so every day we take those new labels that we have on our s3 bucket in conjunction with an initial bootstrap labeling set and we train a new model using kind of the parameters that we've best-fit and past the model is all saved in a docker image that comes in and essentially Amazon the sage maker worries about spending off the hardware for us running the model and we save out the final results of the model to Amazon s3 as well where we save essentially how good this model did on its cross-validation the model weights and the encoding from inputs and outputs to vectors the deployment process is somewhat similar so we had to be really careful here because essentially if we change our model or change something significant in our code there's going to be a huge time lag between the time we released that and the time it hits production because every time we make a change we have to centrally train a new model and once that model is trained we can finally put that into a production API so we want it to be extra careful here about how we were doing on deployments and that's why we went with canary deployments where essentially our models are introduced has a 5% variant meaning 5% of the traffic is routed to that endpoint and we monitor this over the first 15 minutes or so of it being introduced and if it passes our health check which the best we can do right now is checking for 500 replies then we promote it to a full hundred percent variant meaning all the traffic is routed to it so the actual API here was just done based on provided architecture recommendations basically it's a flask app no kind of aweful with python but basically two different options here one is for predicting Twitter and the others for predicting long-form and we just go ahead and return those predictions this is hand this is all runs within Sage maker so we have auto scaling rules set up so if we get in huge influx from traffic it worries about scaling it up to the appropriate levels and the underlying implementation here is in Karason tensorflow so it runs pretty quickly because all of our the majority of the code doing work is written in c++ and c so the result of all this work is that we saw a huge improvement essentially in the labs we care about monitoring our solutions pretty tightly so we have understanding of what the previous solution was as well as what the new current solution is and we saw about a 25 to 30 percent accuracy improvement now this isn't to say that these public api's are bad we've done a lot of things that aren't make this not a fair comparison we've changed the problem that we're solving and we have an entirely new data set with new distributions to kind of use in our training model and this is the same thing that we're using to judge the accuracy for so what this really means is that we're 25 to 30 percent better for our customers which is good for us and secondly we've also included a bunch of metrics here around our workers and our work or pool to kind of make sure that or try to make sure to the best of our ability that their work or quality does not degrade over time we still occasionally go back and try to do hand labeling comparisons between their quality and our quality so summary essentially nothing's ever done there's always more work that can be done and for us one of the huge things here is the inclusion of relevance inside of sentiment analysis is still heavily kind of underserved our customers come to us with a certain perspective and they expect a sentiment to be done in a certain way according to their perspective and we have to kind of understand what that is and come up with kind of a best fit for that so well like their solutions like entity based sentiment goes away to that they still just want to see one sentiment value they don't want to break it down to the entity level and lastly and also model exploration we follow typical model for data science where you spend 99% of your time doing data preparation and gathering labels and 1% actually doing a model so there's probably a little bit of room that can be squeezed out of that in terms of improving model performance so the kind of cool things that worked here is that we have this continuous amount of pipeline that will continuously get better over time which is absolutely fantastic for us and it's really hands off as well we don't actually have to go back to it too often or at all to keep it up and running and make it get better we did actually put this through a bunch of customer trials as well before we pushed it into a full release and got a bunch of positive feedback there and now that it's in more of a full release we've seen that essentially the customers that were doing the most overrides in our system have are now doing 60 to 90 percent less overrides for sentiment which is totally awesome that they no longer have to do this no longer have to waste such a massive amount of time hand coding every story and also we got a lot of cost savings compared to the api's that you pay for here enough to hire a few more engineers so on that note we are hiring so if you're interested in working in data pipelines and Scala or machine learning check out a website or email me or talk to me after the talk after this thank you guys actually getting that degree of improvement in customer satisfaction that's shows how generic the the problems actually are we do have time for questions and yeah sure so essentially we understand that sentiment isn't perfect in our platform and we get customers that care a lot about sentiments so they've heard of things like people time tying people's bonuses to sentiment so what this means is that they want to make sure it's really accurate when they do the reporting in our platform so they will actually go through and on an individual story basis or on a rules-based basis they all say that this story and our platform is marked as negative but actually should be positive I'm going to apply that and now it's going to be in all the reports that I generate from now on so that was a good indicator of how satisfied people were kind of feeling with our sentiment solution but at the same time because we had this kind of rules based method of allowing that to be done it was a little bit difficult to turn around and use those for labels themselves because we don't really necessarily know the accuracy or kind of how well that was done especially because many different customers do it in many different ways well thank you guys [Applause]