Text By the Bay 2015: Katrin Tomanek, Transforming Unstructured Offer Titles
Recording: Text By the Bay 2015: Katrin Tomanek, Transforming Unstructured Offer Titles
foreign as a company is working in the field of of content marketing and so like in a nutshell what we're doing is we're inserting links Affiliated links in the content or in the pages of our Publishers who subscribe to our services and those links usually go to product pages and whenever someone clicks on these links our Publishers earn a commission from us so as a part of this we have to deal with offer records a lot like electronical offer records and those records are only partly structured so we do have structural information in terms of price information links images URLs and stuff but the actual offer title what this what this product what this offer is about it's pretty much unstructured so to give you an idea of this we have titles such as this like a Lucky brand women's Amazon exclusive blah blah blah Gene eventually as humans we easily identify that this is about a gene but we don't have no structured information about this per se similarly for automotive we have this cheaper key replacement fog light and it goes on like that so in an offer title we would usually see general information like how this is shipped how the warranty Works how many pieces you get but then also the actual product so it's pretty important here to understand that if you buy this thing and you open the box you won't have the cheap actually but this is a little fog light for this Jeep so it's important to understand this offer so what is it about what's the actual product what's the brand name where would you apply it and which parts of the offer are just not important in terms of describing the offer itself a little bit more complicated example but that's something we see a lot is this one so I already highlighted the important parts for you first question would be what are we actually buying if we're we're getting this are we having the iPhone or on the left side the speaker well it's pretty important to know that we're actually buying a speaker if we're ordering this and not the iPhone or iPad or whatever we can infer the brand name from this title we can infer some attributes such as it's a wireless one it's rechargeable and stuff but all this is not at all structures in the original Alpha titles and so this talk will go about how we're dealing with this kind of structures how we're how we're automatically inferring the structure and why it is important for us as a company to do so there's basically two products where Glennon has which which are making use of NLP techniques one is called quick link insert where we are scanning product and we're scanning web pages and finding product mentions and then we want to link them to to our product offers and a second product is called backlink convert where we identify already inserted links and want to convert them to other offers which are identical in terms of the actual product but where the actual publisher could earn the most commissions so in both cases we need NLP techniques to understand the website to understand the offers and to match them two together a short introduction to the business we've been founded in late 2008 and just about a year ago we raised some extra funding which helped us growing today we're about 50 50 employees and the service is actively used right now we have about 300 million active Publishers who are using our services on the demand side and on the supply side we have about 40 000 Merchants whose offer feeds we're importing and to whose offers we can actually link to and on a given month we are we have about 500 million links where people click on so that's a very actively used service um in terms of alpha feeds as I said before we have this 40 000 merchants of course there's like very well known allowance like Amazon eBay but also very tiny little Merchants whose office we have to import into our databases then structure and then use for the service um in the end right now this sums up to about something about 300 millions of offers which we store locally in an elasticsearch cluster we can then use for the two products I mentioned before once again to to point out where the challenges of these originally unstructured offers are so you find the term iPhone somewhere in a website and you want to link this to your office right now we find over half a million of offers which you know somewhere mentioned the term iPhone so just linking to a product offer which has some iPhone in the in the offer is really not it's really not enough and will result in very unprecise offer linkings some examples we see here the first one is actually about the case as is the second one so we have a lot of accessories in our offer database and we need to find out is this an accessory or is this actually about the iPhone this one is really an iPhone it's important to identify that and the last one is the example you've already seen before so that's why we want to structure and understand our office um and so over the time what we identified as like important parts that we need or want to know about an office and if we know that we can go towards structured search and to solve this problem so the first thing of course and ideally is a product ID so in this case it would be the svx7 that's the actual product ID identifying this specific product a lot of offers also tell us what an arc what an offer actually is so in this case it's an MP3 player and a portable speaker I would say about 90 of our offers actually have this information so whenever we have say a search which is not by product ID but more General like show me all the smartphones all the cameras you got it's helpful to know what this specific product category of an offer is we got brand names then we have a lot of features or attributes of offers in here we find it's wireless it's rechargeable we find colors for fashion we might find different stuff so this really depends on the verticals but usually offers have a lot of different attributes they are specified and then for us what's interesting also is like which part of this long potentially long after title is actually essential so which does really talk about the offer versus you know like offer information such as shipping warranty so that's what we call the central phrase and then there's an additional one which we call the application context the application area what could you use this thing for this is especially important for accessories because usually they mention they mention the products you could use this for remember the Jeep Cherokee we saw before we have this a lot of time with with mobile phones cameras because just because they have so many different accessories now how we're getting to this structure information we have a three four step approach where starting from the original title we do some shallow pricing at the first level where every token is eventually classified into into a different token class on top of that we can identify different phrases so the two main phrases we're interested in the central part and the application and and then as a building clock in the middle we identify the actual fields that we're interested in such as the product or the product category now let's have a look into how this actually works so the first step it's kind of a shallow passing um basically a segmentation task where every token or like every multiple tokens have to be assigned the labels we identified six different labels which are which actually can describe all the offers from about 20 different verticals we are having so the schema works both for Consumer Electronics as well as for automotive or we can even use it for fashion or Home and Garden Products so categories we've seen them before product product line Etc what's interesting here is that we have something which we call an offering attribute or feature like something is brand new it is stylish it comes for free free shipping warranty Etc so we see this a lot so it's important for us to to recognize these phrases and then we have functional terms which basically have a syntactic role like telling forward something is used it works with Etc so this work has been published previously and we have some training data for that and we basically run this as a machine learning problem using conditional random Fields because that's very similar to standard named anti-recognition tasks for those of you familiar with it we use some pretty standard features there such as word identity there are some tokenization some stemming involved orthographical and morphological features are very helpful since token patterns especially when it comes to the product names itself they tend to have dashes they tend to have combinations of numbers and characters contextual features telling us about the context of the word what is the neighboring tokens to the left to the right word modifiers show to be really helpful product names tend to have pronouns articles quantifiers this helps us to identify all these different different steps and as a last step we do have a lot of lexical resources in-house because we we know and learn more and more about different products and so when you feed in this this dictionary this lexicon into the into the system into the Learning System this increases performance here so brand names it's basically just a lock a list we have a lot of different product features which we already know product categories some brand some some products which we know and this all goes into the model and helps identify these tags the evaluation was done over over about 20 different verticals so it's important to to see that there's a huge variety here so this includes all the different types of products we have starting from book names over fashion products which are very different from Automotive Products for example just the way people specify a brand name to consume electronic products um you see that the performance values actually are pretty different so there's types for example the brand name where we have a pretty high performance it's kind of straightforward to identify brand names even though there's some pitfalls but in general that's more of an easy type to identify product names are way harder as I said before between the different verticals you have very different ways to specify product names and then offer features are even more hard since the variety is even higher and they are completely they're completely unstructured um so before thinking about how to improve this model and whether we have to improve the model we need to see in how far do we have error propagation to The Next Step so for now we keep the model by this and this Improvement to come later um in the next step we want to use this past information to actually see how we can infer our two Central phrases or our two most important phrases from it and then to to extract the actual fields um so to do so we started with annotating a set like a test set about 200 titles and we chose three of our general verticals which is fashion automotive and consumer electronics those just because most of our products actually are from this area and we find that a lot of our Publishers actually work on this on these fields and that's actually the approach is pretty standard to First go with an annotated test set and then study the different phenomena in your problems and to see how you can actually solve this problem formula the Baseline approach to see where it's worth to dig into where the real hard problems are what the what the actual challenges are um so we did this um I started with them pretty straightforward Baseline where we could use the underlying pricing make use of some of the tags especially the ones the functional terms which help us splitting the original phrase into sub phrases and also the offer feature phrases such as if it's about free shipping Etc like in an iterative approach we split the phrase it split the original title into sub phrases and within that we can then extract the relevant fields and it shows that given that we have like perfect offer passes this works pretty well so that's pretty straightforward approach gives us pretty high performance values for identification of products or product categories um and also the identification of the central phrase is at this point pretty high however that's based on perfect passing so when we go down to the imperfect path that we actually get we see that the numbers are dropping and we can see where actually the numbers are dropping so that's now for us a good hint where on which level we have to improve the system and when you remember before we saw that the of type the offer featuring type has a pretty low recognition performance this is one reason why sometimes we do we have problems in recognizing Central phases so it's a good point that after this extrinsic evaluation go back to a step further and see where you want to improve the system um One Challenge which is not covered here is something we call the embedded application so remember our Jeep Cherokee example the cheaper placement it's pretty hard to identify that it's actually a fog light for a cheap hierarchy because like purely syntactically there is no hint for this so the only thing you can actually do is use World Knowledge or some external knowledge to find out about this because in most cases if it's if it's correctly identified we have this kind of a relation we can see that a product is of type product category in this case this is purely wrong so ideally we have to identify cheap Cherokee as the actual application um what we can do here is since we have a huge amount of offers um kind of reformulate the offer into a product category for an application contact context that's what we see here if we reformulated into fog light for Jeep Cherokee and tested against our offer base and if you find evidence for that that's a good hint that actually here we have such an embedded application we do this and this seems to work out pretty well so over time since it's an iterative approach we'll see refinement of the extraction of our office our last step here is then normalization and subcategories subcategorization since what we want to do is for all the recognized product categories since those names can be pretty long we've got some examples here it could be a cake baking kit it could be a fiberglass boat terms can be multi-word Expressions so we want to reduce them to the actual base category um and for that we basically make make use of a dictionary which tells us about valid and invalid categories but since usually not all of the terms can be in the dictionary we have to use some heuristic where we infer the head noun from the categories and then test if a head noun is a valid one so for example if we reduce the cake baking kit to the Head Mount which here would be the kit and we know that kit is a very uninformative term we know that we have to step back and take the little bit longer term to not lose the information that we're interested in so this is one step we need to do and we need to do something similar for the attributes since attributes seem to have a lot of different variation for fashion for example we're basically interested in age groups like is it for women is it for kids for toddlers we want to normalize the values and we want to group the values also here we do have a dictionary which we apply but the question is how how do we actually manage this dictionary how do we grow this dictionary and we started to work with word to back to to collect new synonyms or possible terms which we can then feed in there and I'll show some examples where where this can actually work and where this does not work so what we usually do is we take High frequent terms such as the color red here because we see that a lot of our products are red and then we search the word to vac model trained specifically on fashion products here search for near synonym terms this works pretty well for colors we see all the different colors I think there's a mistake somewhere here but in general this helps us find a lot of a lot of things it also works pretty well for sizes so we have all the different spellings of sizes it starts to become problematic for for example patterns or materials we see there's some interesting stuff Merino and Cashmere lambswool but then it mixes it up with patterns so for right now this is basically a suggestion system where for high frequent terms we can find potential candidates and we use humans to curate the dictionary and decide whether something has to go in or not um another interesting application here is the word analogy which is offered by word2back where you can where you can ask this analogy questions small goes to S such as last large to award and doing this reveals that L and this holds for these these simple cases this seems to work pretty well if you go into more complicated classes like patterns and materials well we've seen a couple of interesting results for word of it yesterday we got the same weird results sometimes it's more funny than actually useful at that point um as I said so distance is helpful to get input for potential terms analogy at this point it's more for playing with it and seeing where the potentials are in the end once we once we did the overall extraction we're actually storing this in elasticsearch as some fields and then this allows us to do a structured search on top of it what proved helpful for us is actually to store confidence scores like from the CF on the token level so we can actually search for instances which is a high which have a high confidence especially given that now we know that the especially the pricing still has some problems but once we restrict it to something about it's like a 90 percent confidence score we usually get pretty good buffer instructions um I'm going to show it to um two applications of this where we can use it um one is offer clustering so as I said before in big link convert we are actually interested in finding similar or ideally identical offers so that we can replace them the problem here is that we got this huge offer catalog of over 300 million offers so like a traditional clustering approach where you just run a clustering over the whole set it's purely impossible and the second aspect is that also text pure text similarity is totally insufficient here we learned that offers contain a lot of uninformative stuff such as shipping warranty Etc so you need to you need to factor this out before you actually do the clustering and the nice thing is once we do have the structure we can actually run the cluster on the different fields we could restrict to product names to attributes to product categories whatever we're interested in um and to make it more feasible we can even easily apply some plucking mechanisms by that we pre-select for example brand names and product categories we can say let's have a look at Panasonic headphones this reduces it almost already to a very limited number of offers in which then we can just run clustering in this showcase we just run a hierarchical clustering pretty standard approach similarity is calculated on a feature level on a feature basis here just using a Chaka overlap measure um and this shows how actually well this already works with a pretty standard approach so what we did is we restricted to Garmin GPS devices that's just something we found in the offer database that we seem to have a lot about them and we see that this nicely clusters our offers in first product lines so there's the original line Vortex Dakota newbies so these are all General product lines of Garmin and then it groups together like different actual products like the 650t is one specific product and when we look into the specific offer phrase we see that some include shipping information warranty some include information on which batteries you could use Etc but working on just the important part of the office helps to helps to do this clustering pretty neatly um and a second application is what we call an analytics dashboard so this helps us basically right now in-house to see what is actually in our offer database because before we would import a new offer feed and just not really know what's in there since these feeds are usually used and huge and unstructured and now we can have a look what is in there what kind of product categories is this about what kind of brand names are we selling Etc we're using the open source tool kibana anybody heard of that tool it's yeah it's basically used for in combination with lock lock data it works pretty well for our offer data and it gives us pretty easy access in an innovation way to our offers it's pretty pretty small probably pretty hard to see it's just a simple case where we are looking for within consumer electronics for a given offer feed what are the main categories and I'm not sure if you can read it but we can see that we have basically batteries then there is cables adapters Chargers cameras and then speakers so this once again shows us how many actually accessories there are in all these products um another application would be let's have a look at our categories the the verticals based on what we can earn from them so usually what we want to see is the EPC the earnings per click what I'm showing here is as a proxy just the price per this product is and this nicely shows for example that this one which is jewelry has on average the highest price which is not surprising this kind of falls down to the EPC as well whereas fashion which is this one has a way lower average price and then once you drill in and select one of these you can have the same overview to the different product categories So within jewelry for example we have a lot of rings which sell for way more for example than same with hard for watches so that's an easy way for us to understand what's in our offer databases and the last example let's kind of clustering perhand so for the same example as we saw before someone can go into the category into the catalog can say okay let's have a look at consumer electronics okay Panasonic seems to be an important brand name they have a lot of headphones let's have a look here and now we see which kind of product names actual headphones they're having and once we select this we see all the single we see all the single product names and seeing how far they are related if they could be used for wickling convert for example to to be interchangeable um yeah so this is how we use NLP techniques to do alpha structuring and it's currently work in progress so I'm happy to take some questions okay so you mean the the parts of a product which has have actually not been used for clustering that could be done yeah so right now this clustering has been done on all the product features that we have um we'll be planning to restrict this to say for fashion we are basically interested in age group maybe material maybe sizes too restrict it to those so but for now we just used all the different features we would be having