SF Text: Adam Pingel, Katrin Tomanek, Gabor Melli, Content-Driven Commerce at VigLink
Recording: SF Text: Adam Pingel, Katrin Tomanek, Gabor Melli, Content-Driven Commerce at VigLink
works okay so I'm Adam pingle I'm an engineer here at biglink um this The Talk Amongst the three of us uh here tonight from biglink is going to Encompass the business and architecture I'll cover that uh Katrina is going to talk about insertion in depth and then uh Gabor is going to talk about product categorization so I just want to set this stage for those next two talks really quickly um so Oliver mentioned um moving value from the destination back to the origin and affiliate marketing is the actual mechanism by which we accomplish that um I've a lot of the actual affiliate programs out there are um a little I wanted to alide a lot of the details so I just created this fictionalized uh hypothetical example here there's some Merchant Acme um they they want to they want to encourage people to link to their products so they need to create this web service which will allow people to send them traffic um and uh or and and get the revenue U moving back the other way so in order to do that uh the the source needs to send the encoded ultimate destination URL um as long as an identifier for themselves and um an identifier for the specific click um and this is just a vis visualization of what viglink does you can imagine um some of these links may have already been included by the author uh we have uh this link insertion technology that finds additional product references um just a few bullet points about the business it was founded uh late 2008 uh had a profitable quarter um in 2013 uh raised some money last year um from uh those list of our backers right there we're about 45 employees I think we've been growing a lot recently since the summer U we just moved into this space as Oliver has mentioned um we have just a few metrics to characterize the the business we've have hundreds of thousands of publisher campaigns and about 50,000 Merchants that includes large Merchants like like eBay and Amazon but also aggregators like Commission Junction and uh just to you know prove that we do in fact have some Scola here um we've historically been a Java shop and over the last couple of years more Scholars been working into our core uh runtime uh components uh you can see it coming in there as the number four um programming language of course we all know that any language on the right one line of that counts for for a few of of java so you know got to wait that accordingly um and in in our ad hoc analytics uh scol is actually um over 40% of the lines of code there so that's where it's really taking off um really high level architecture uh we're all on Amazon uh we're in Three Amazon data centers in uh Oregon and in Ireland we just run the API and in Virginia We Run The API as well as all the link insertion technology that you'll hear about later log processing uh uh viglink.com and uh alog together um I I didn't do the aggregate here but I I think we're we're peing uh consistently above 10,000 requests per second to the API um at a really high level um there's some JavaScript running on a Publishers page that we make available um it'll send a a ping and an insert call uh in in reality we actually can batch those together typically um the Ping is just hey I'm here and the insert is uh please give me all of the the links to insert uh Katrina's going to talk about those actual data structures in a minute um and behind that we've got my SQL and Cassandra um and and there's in a couple slides we'll go into much more detail um I think I'll I'll skip over a lot of this stuff except to say that we uh so this is up here the our API producing raw logs which get merged they flow into red shift uh the revenue feeds processors of course are an important part of the business and then from a large set of these uh merchants or Merchant aggregators we get offer feeds uh and we will we put those in elastic search uh that's an important part of the link insertion pipeline and um I think the last slide here is um just a highle schematic of of the link insertion pipeline uh Katrina's going to go into that in in much more detail but at the end of the day um we're we're receiving insert calls from these publisher Pages uh we've got a monitor watching the logs and making sure um that it's it's that the content is either not too old to stale or if it's never been seen before we will we will uh insert request to crawl the the content into this sqsq that will cause a crawl uh after the crawl we will then run the link insertion pipeline uh which uh brings the content of the of the page together with what all the offers that we have in elastic search and produces a set of matches and then those set of matches are available to subsequent insert calls for that page and uh now I'll hand it off to Katrina to talk more about that yeah than what just said okay okay so after a lot of infrastructure and architecture we're going to have a little bit of a deep dive into how thises link insertion actually better this way how link insertion actually works um okay so the goal is for a given page find all the products which are kind of mentioned in text find these intext and actually link them to Offa databases and ideally to the offer from which we or our Publishers can earn the most money when someone actually buys this product so so what we need to do for given page is first you know find the offers it's basically an anty recognition part um task um and then we subsequently for every Mar here yeah for every for every of these products we find um check our elastic search offer database which is huge if we have any offers which could match this phrase and that's kind of a funnel process where we end up having a lot of potential offers which we have to filter for some business logic depending on the on the users um and then do some semantics on top of that to really find offers which perfectly match this phrase well and in the end we might select base on some business rules to yeah to eventually find products from which we tend to earn most money um so that's kind of kind of a view we have we tend to have a lot of offers for every for every potential product and removing some removing accessories and then eventually we find that one could be the most valuable for us um so NLP related tasks we are having going through these it's basically an entty recognition or product recognition on large scale linking then the question how how are we dealing with our offer database what do we have to do to curate it how how are we keeping it how can we search in it um a little bit of understanding our offers since offers are kind of unstructured and we have to do some some Logic on top of that and then on top of that we do have a taxonomy of product categories product terms how this all works together to help us understand um the stuff um all right that's kind of hard to see I guess but just in quick the product recognition for us means to recognize all this different things it could be product terms um we have something which we called branded product categories it's a brand name and a type of thing could be a Samsung smartphone for example it can be product lines so in the whole world of products there's a lot of different kind of words we're actually interested in um and all these should be recognized um and yet so for those of you who are familiar with this is typical named anti recognition problem it's kind of a sequential tagging so have a sequence of text a sequence of words and you want to distinguish for every word is this part of a product name or not um and usually named ENT recognition is pretty hard to do on a dictionary based approach so it's it's kind of impossible to use just a dictionary match it against a text and by this find all the product names um there is a lot of spelling variants um there's new products which you will never have in your database there sub products um um sub numbers there's a lot of ambiguity so words could be product but could also be standard um English word um it's context dependent and so on and so in the end we end up with um hybrid approach where we have statistical models to help us but also dictionaries of course help um some rules also help and all this goes together in kind of an ensemble approach um and the good thing here is that we do have some thresholding which helps us to Define a level of precision and recall um depending on users needs um okay so for those of you maybe familiar conditional random Fields um is a sequence labeling model which which is used a lot for this task it's actually um structured prediction what we're doing so instead of predicting for every word um independently whether it should be a product or not a product um we actually create um a sequence of words and we use the surrounding words as information about um the current decision um and so there's a scheme a tagging scheme which is called the iob so for every word we Mark whether it's the beginning of in this case a product inside a product nothing at all product category and whatnot um and then usually what has to be done here is a lot of feature engineering to find out um what your model should be informed about so what's interesting is usually the word itself the tokens Maybe stem form um a lot of information on orthographical or morphological stuff such as um suffixes prefixes maybe word patterns are there dashes are there a lot of numbers inside is it capitalized and all this stuff um helps the model to learn um for sure context is important so how do the neighboring tokens look like um there's certain indicator Words which which help to know if this word tends to be a product um word modifiers seem to be very helpful pronouns articles quantifi buers Etc and in the end we make use of our taxonomy and dictionary um to inform the model about extra information such as brand names um typical product features we know about so all this goes into the model um and then we have um a test and training set which we use to actually get this model and evaluate our model um in Parts this has been published um in a previous cular competition in 2012 12 right um so we have a lot of different categories and for most import for the most important ones which is Automotive consumer electronics and fashion for us since that's where we have the most products and the most Pages actually um we do have a test set of documents um and it's maybe interesting to see that um in fashion we tend to have more product lines since there's fewer specific products you have a shoe which is kind of align because there's a lot of different sub models um in consu consumer electronics we have more specific products for example so there's certain distributions about that um and for training our model We additionally make use of um click data so we take all our lock files or parts of our lock files and see where people used to click on because we assume that this is kind of correct or at least kind of interesting to get a little bit more training data um so this is a tool we're using for The annotation it's called Brad maybe some of you have heard of that it's kind of a configurable tool where you can set up the type of um of annotations you want to do it's pretty good for sequence labeling but can also be used um for any kind of relation annotations um we added some extensions for um for pre-labeling correcting annotations also inter annotator agre calculations to see how well human annotator actually does um so this allows us to in in accept acceptable time to actually get some data for testing um and then oh all right and then evaluating this um so without going into the details we we have a couple of models as I said before which go into an ensemble and they have different performance values so there's numbers called recall and precision so recall usually tells you the coverage so how how many of the actual entities does your model recognize prision tells you how precisely this this is done um we have models which are Super precise and others which tend to be rather sloppy um so they recogn phrases which are not actually products in our very clear definition but it shows in the end we have to keep this models in since when we do ab test it shows that people like to click on this little bit of sloppy terms um which still lead them to interesting sites um and so we came to the decision that that's actually our the model we want to use in the end we do have this Precision recall curve um where we can still decide for a given publisher who might be more interested in high quality links um we can still say okay let's let's have a lever somewhere here we are very much interested in Precision we lose a couple of product links or potential product links by that but we can ensure that the quality is higher or we can go the other direction and go more for recall and sacrifice a little bit of precision all right so once we once we got this this is one of the potential product links what we got this um the task is to link that to the most correct product offer and this already gives you an idea um that a product offer usually is much more than this little bit of a product mention that we find in text and that's actually the challenge here to to find the correct product offer um so challenges offers are unstructured so this is one of the offers a pulk audio Blackstone TL 1600 and so on so all this is in the offer um the offers usually contain a lot of accessories so you might be interested in an iPhone but in the end what you get here is um a button replacement um okay we have a lot of overloaded alha titles um you know it comes with a cover it's free shipping it's like new and this goes on and on uh this goes on in a way that sometimes offers are just nonsensical just kind of doesn't make sense to link to those at all um we see especially words like iPhone iPad they just sometimes tend to um concatenate all of these just because it makes them likely to be found so we have to make sure that we actually folder those out um linking is also hard because of high diversity so we have about 30 different industry types starting from beauty products going to home and garden over Automotive to consume Electronics so it's a big variety we have a lot of different motions um everybody has feet 20 fet structures and all these could be structured a little bit differently and that's what we have to deal with and in the end it's it's just pure volume I mean it's a couple of hundred million um offers out of which we have to select the best offers like semantically the best ones and also those which which optimize revenue for us and just to give you an idea when we're looking for iPhone 5 we actually find 650,000 offers which contain this and not all of those you actually want to have um so how do we get where we want to get um we usually start with the a kind of semistructured query to elastic search which gives us a lot of offers um we're basically looking for kind of a normalized version of the product mention um we do a couple of restrictions basically by industry type and user specific settings um then there's some business Logic on top of that to to filter even more and then what's more important or more interesting is what we do on the semantic side to get to you know a set of offerss which are actually acceptable out of which we can select um and the question is given them a product mention and an offer candidate is this is this an acceptable offer um and there's different um aspects to which we came which which are interesting to do that the first question is um is this offer actually valid for my given my given mention so say we're looking for jeans and we're finding a jeans jacket well it's about a jacket so it's actually not valid for what we're actually looking for the second question would be centrality so if you order this thing open the box you were looking for an iPhone but you're taking out a replacement button that's also not what you were looking for that's the question of centrality what is this offer actually about well the first one is really about a metrix amp specific one for headphones but still it's a metrix amp the second one well you will not buy Voodoo you will buy the DVD player um and then centrality is kind of related to that uh accessor is kind of related to that because a lot of time non- Central offers are essentially um accessories and all this you want to know about and once you do that um you can have kind of um decision tree which guides you through the process so if an offer is valid is not Val is valid and not Central then we have to decide if it's not an accessory that's kind of a weird thing that's this Voodoo thing where it's not the center and it's still somewhere in the offer but hey what what does it mean actually um and this goes on with some more um category specific logic to actually guide us to the question are we just missing this offer or are we actually keeping it and we have a lot of different juristic and learning based approaches to to come there um again we have some training data um I skip this here and um maybe some statistics on the data which is interesting we see in fashion we hardly have accessories that's because you won't find so many accessories for a sneaker except maybe for shoelaces or stuff but um in automotive we have a lot of accessories since we hardly sell cars or maybe motorcycles on our in our offer catalogs but tons of Parts um so there's different distributions of of these problems um in the end right now we are at a Precision of around 70% um for this um decision tree so there's clearly room for improvement um but it's a good start um okay so I'm going to hand over to Adam for more on the offer feeds yeah I'll just describe offer store really quickly and give these guys a chance to uh swap the the mic um so we consume feeds uh approximately 20 different feeds some of these feeds are composed of uh I think hundreds of files um at Le one case um that uh that Mets us about 400 million offers and we refresh that every single day so the there's a long story behind our offer St we we started off many years ago well before I was here um just embedding Lucine in a tom cat container running it that way that that worked for a while then we moved to solar um early versions of solar 3 still on a single box with lots of memory uh and then in around early 2013 um on top of solar 35 we did our own sharding uh just product ID mod 5 you can imagine that wasn't the most uh flexible Arrangement uh with our with elb load balancing so what we went to about a year ago uh was elastic search um it has handled as high as 10,000 requests per second uh I think we run typically much closer to that two number these days done a lot more tuning a lot more caching um the the we have about 30 nodes in the cluster um depending on the index size that'll be anywhere between 1 to 8 shards um with uh no replicas and we are um we unioning these effectively with an alias so that the the insertion pipeline gets to view all of these different feeds as a single index uh just to make this really concrete this is an example of a feed we get with uh some uh inar headphones some neon pink inar headphones uh so we get gigabytes of this kind of thing every day um and we turn it into something like this so it's normalized uh Json document uh there's a picture had an image URL there um things like a category it's in consumer electronics we've got a fairly a flat uh categorization uh hierarchy at the moment Bo's going to be speaking a lot about that um and yeah that the process is what you might imagine we have to parse these things extract the relevant Fields uh we write we write them to an elastic search index uh suffixed with _ WR um and concurrently we'll be writing that to S3 as Json uh that helps uh uh that's appropriate for some analytics workloads that we run against these offers uh when the right is done we'll swap that uh call Itor read we'll drop the old index and uh these things stay around for three days uh for a lot of business reasons we don't want to have stale offers in the offer catalog and if things are working correctly we're going to be refreshing these things daily anyways um and this has been written uh in a very stream oriented way um Shake our expert elastic SE Arch Guru uh has has done this and we're um also been prototyping AA streams uh lately uh so we can get some back pressure going through the system and now I'll hand it off to um so as Oliver mentioned I was a little frous I arrived here but in a sense it wasn't I uh when I knew that um text was coming then that's when I decided to jump into the space so the timing was somewhat planned and it but it did work out a little bit the focus that I I I took was to understand and to apply it to production systems where that knowledge can make a small but substantial even if it's only cents per um item so one of our main challenges is understanding offer so we have lots of them but here's an example of one this as uh krine mentioned this pulk audio what does it say well um we want to decompose this and maybe now a human can look at this you you you all can but we want a machine to be able to do this and to do it for a wide gamut of um product offers so the way we thought of this is to parse it so instead of treating it directly we if you can see down here and I'll have more examples put some structure to the text and labeled the uh chunks so as um as I mentioned there's a lot of variety so there's over a billion well I know there's more than a billion offers out there um we don't have all of them right our elastic search but uh one day we will um but we have a lot of them we and so there's just a a an incredible number of this and I'll I'll put a small plug for this talk so I already gave this talk at kdd have a paper there and so if you look for my knowledge based uh this paper and talk is is there and so back to our uh our task so uh um who's uh built a maybe a conditional random field model models on text here any machine learning people here some so so this is the takeaway message so as soon as you see that you have to uh build um an a machine Learning System I recommend that you create some annotated data if if if you can't get people have enough resources to annotate some data the problem is probably not that interesting so we we have uh an annotator that annotate some data and and one of those is krina and I um it's good to do the first step bit of anotation yourself so annotate a lot of data so uh no no special uh tool required but you have to think about what you're annotating what you're trying to accomplish so um and Katrina already mentioned this or let's say that we tokenize the string but you'll I repeat it in part because you'll also encounter this you know this is one of the takeaway messages of the state-ofthe-art what you typically do is you put the labels here with this B IO labeling and then you move that over here here because this is your label and now you're going to put some features around here what's the first character you you put a bunch of features this new deep learning method is trying to get away from doing this engineering but uh for now um you have to get your hands dirty and actually it's also good to know what the the space is so that's typical uh as K mentioned we use it for recognition we also are using it here so what do what was the so we did five full cross validations we divided the data into five chunks and then tested at 57 .7 F1 so that's sort of low but you know what you have to look for in uh when you apply machine learning is that your task doesn't require 100% if it requires 100% you're probably U picking the wrong task you want to pick task in machine learning where there's some some room for error and here it was and I'll tell you why but what we did notice is that some categories like books and arts and entertainment did poorly huh uh maybe we can let's let's explore so now we're exploring in the model so we're let's not just take it on its let's find out what this behavior is and this is where the the interesting part of it was for us is that sure enough when you train just on books you you set aside the books and you don't include all the other noise like consumer electronics then it gets great performance so we get that less and ideally this would happen automatically that it would not recognize this division for now we do it manually but um I'll show you in the next slide oh no this past one uh so when we grouped uh each one of these we had several models and lifted from the 57.7% so another nice outcome from this work as as as I sort of require and demand of myself is that there's a a return on investment it's not just a proof of concept toy so here are some example of terms that we didn't know about that weren't in our dictionary and so prior to when we did the work there were hardly any clicks on these terms after we did this work we scoured our elastic search found these terms as being significant added them to the dictionary now we have uh clicks on them so and each of these clicks is pennies but this is just uh eight of them instead of the 100,00 that um were added and um those 100,000 pennies uh paid for the project in a few months and um another little hint is H to to not do it all in one bang to to iterate and to add examples with when there's low confidence focus on those where the edges are and so then your annotator is just fixing remedying some of the problem areas that goes by faster like if if it's if it believes these two are together split them up so that's um another direction we're going now so this is more now future looking is that you notice that these were our categories are fairly shallow we're we're trying to really expand this and so this is going to make it a little more difficult but so far we're having success at even modeling hundreds of nodes thousands will be a problem but that's where we're going so this is where we uh the kind of depth that we want is um to categorize each of those offers at a very very granular level and uh some recent work has been sort of fun uh I don't know if you have you heard of word toac it's this new deep Learning System that Google put out actually I mean they're the ones that got the buzz but there's been a lot of of other work similar to it but it's fun to look at and so we are automatically finding clusters in our offers and so what are these oh these are cell phone batteries so automatically we didn't I didn't have to hire anybody to to discover this cluster oh they needed to do is to tell me that those are that so I I recommend that you that there are some unsupervised ways semi-supervised ways that are are are getting some tracks some good mileage some good discoveries these ones were discovered and um that's speakers so there's some uh nice new work coming along the way in the in the neural network this is not uh deep learning because it's actually just one hidden layer so it's shallow network but it's uh um it is recurrent so uh lots to do I you know I've been at it here now for three years there's too much to say um but um but if you're interested then uh you should definitely uh contact us question so this ontologist your build so you want to grow a deeper ontology yes uh so is this automatic anology going automatically or is it manual or combination so it's combination certainly we gave the the beginning of it and in part we inherit some of the structure from all our Merchants they have our own understanding of a an initial Division and uh then to go deeper we have a human sort of because that's fairly easy you you know the the Returns on that investment diminish as you get more granular and so that's what this opport this looks like a nice opportunity is to be the ability to grow Concepts automatically and then find ways to link them it looks promising but stay tuned any other questions on our technology maybe on elastic search or yeah yes where is the difficulty between making something more generic go after TR what's what's the challenge there well um would you let's say if we go back to this uh so is this generic enough like is that a gamut of like this is about 30 different Industries all and those all of these are in elastic search ask if there's a new indry would you have to Mi specific effort for them first or just they just go into the final and there's no no need going right so one way I think of it is transfer learning you've done all this work does it what you've learned transfer to this new domain and now that we've done all these uh it transferred fairly well so uh we found that adding a new one like you know a big step was going when we had from consumer electronics and automotive and going to Fashion you know we needed to do a little more training but uh now it's getting far more very general so each of for example we don't have models for each of these industries it's one entire one only one model but we we did have to occasionally give it a little more evidence and they recommend that too that you don't just throw it out there and let it sit there for years you continue to give it a few hints um but that's not as painful just to occasionally give it more information so very general one model for all these industries um but we do occasionally add more training data it would be kind of temp if we had just one industry type and say one feed we you know know the structure know this the specifics of this feed to optimize for that feed that's why I was showing this diversity thing because this so many different things um we don't optimize for one of them and what I I showed here is that in this case books yeah books you should have a an individual one but other than that each of these other benefited from having evidence of the others so okay but just to clarify you don't have to create different or specialize in ontology for each domain no you have like one onology that's right for the universe we do yeah so all those features just go in and whether the fashion terms decide to use them or not it's up to the model thought I there back there yes sure so the questions how do we handle with addition and removal of offer so I think Anna might be able to help with uh the processing because in part it's an elastic search problem how do we time put timeouts there and we we just feed them in it's just a all right well thank you again for