Text By the Bay 2015: Mike Tung, Turning the Web into a Structured Database
Recording: Text By the Bay 2015: Mike Tung, Turning the Web into a Structured Database
some more people coming in so i think it's um first of all thanks a lot for coming everyone it's impressive to see so many people here early on a saturday and uh i was also impressed when i looked at the attendees to a lot of different customers and partners here which i mean it's neat but it's also a little bit telling about our field too it's like a kind of a small place um and uh we really need to collaborate work together if we want to grow the the texas analytics community larger um so that we can you know we can see some unicorns you know some bigger companies um um how many of you know what the fight is i've heard of this bot before okay so you looked at boilerplate or readability and stuff you checked out for the article api okay um anyone over there heard of it or i saw a few hands okay um so i'm going to be talking today um not so much about particular techniques for text analytics because you know we do a lot of that at our startup but i was thinking like what i could really share that would be unique and i think at our company you know we use a whole grab bag of different techniques but what's really special about our company is i think the sort of philosophy and the way we think about these problems which allow us to do really well because divot is really about i would say 20 to 40 different classification problems maybe five clustering problems and the um the algorithms that are in production uh perform uh close to human level performance uh and it's you know it's a pure machine learning api so the only reason like companies like microsoft and ebay and these folks use it is because it's something that they can't build in-house um so we really pride ourselves with um the quality of the extraction uh and you know robustness and speed and so forth um i want to start out talking about just computer vision real quick so this is um a screen cap from the movie terminator 2 which hopefully you guys have seen this is when arnold wakes up and he's naked and he gets up and he steals you know he steals his harley so you can see here it's like you can see this bike outlined and there's obviously some sort of like occlusion detection going on here with the dotted line um you can see visual harley-davidson model 958 analysis you know i guess it says vehicle size top speed horsepower power i don't know 600 in this code range there's an estimate of this probably 798 miles from here um so this is the the problem we refer to as object detection right in computer vision it's um given a scene um how do you identify the objects in the scene and give them labels right in this case to label the what we call the the y is harley-davidson model 956 so you know recently there's been a lot of improvements in the state of the art in computer vision um i'll show you this slide here which sort of illustrates i think this is the best sort of capsule of the computer vision object detection improvements so this is um imagenet and this is the classification error in 2010 it was 0.3 right so 30 of um of imagenet was classified wrong and then 2014 it's less than 10 and this is actually out of date so uh um google google net published um at this is i think 6.6 crosswell or tester i think shortly after that baidu computer vision team you know led by andering published a 5.98 and then a few months earlier just like in february some folks at msr published a 4.94 so to give you an idea of what this means a human level performance on this task so this is a task of about a thousand classes and 1.2 million training images human level performance of human labelers is 5 so the latest msr number is beyond human level performance on image classification and i don't think this graph really does justice to how big of a improvement this is because you know when you when you're here you're basically it's kind of like it's pretty much random and sometimes it's right right you can't really trust it when it's at 0.3 uh classification error and now it's performing better than you know an expert that's looking at these images all the time so in terms of how it feels it's like really like a step function up um it's not just like you know linear improvement it's like uh it feels like it's moved from the realm of something that's not useful you know it's pretty much random and slightly better than random to something that's better than human and can be used in certain limited applications right so this is one of the great successes of of ai recently and the reason i bring up computer vision to start out with in this event that's about text analytics is i want to ask you know how are the computer vision guys able to achieve this so i'm my uh seeing more people coming in can you try to move to the front of the room so yeah um so i was in the stanford ai lab so i remember back around this time when this imagenet dataset came out uh the sort of general attitude was um you know you're building a label set of images that's not you know like that's not fundable research like research is working on algorithms right and and improving your method you know writing a paper talking about a technique that outperforms you know an older technique uh and here these guys were building this data set right that's something you would have uh an undergrad be doing right as an ra maybe right but this effort of collecting 1.2 million labeled images is what um ultimately propelled us from not useful to better than human level accuracy right um and so that's uh interesting um but so we've basically sort of solved um part part of the vision problem of object classification in a limited domain right but how do we apply this to these other parts of the brain right like uh the motor cortex i think we've got it pretty nailed we've got robotics we've got control systems um you can see it's getting better and better our ability to manipulate objects in the world um is pretty much as good as human or better than human already you've got robotic arms that can lift more than a human right but audio processing is getting improved we have algorithms like dsp and stuff that can process audio signal right and some of the same kinds of techniques as in speech to decode it but you know what about this frontal lobe what about the getting this fundamental ai problem right so a lot of us got into this field right which is pretty niche field because we want to build systems that are intelligent and that hasn't happened and i want to ask how can we how can we do with natural language understanding and these other ai hard problems the same kind of improvements that have been done in in computer vision and you know that in imagenet right so uh one of the ideas we have at diffbot uh is sort of one of our philosophies is what if what if actually we don't need to what if one of one of the algorithms that we have today already is an algorithm because if you look at imagenet there weren't really any new algorithms created here a new big data set was created and it turned out that convolutional neural networks which is a pretty old technique um worked was the perfect sort of algorithm that we had already invented that worked well with this big data set to produce these kind of gains now what if we already have the the nlp algorithms that we need we just are lacking that same kind of really big data set right so that got me to thinking about how we would go about building this sort of really large data set um how large you might say you might think you know how big would you need it to be so this was 1.2 million you know before this you had image um image label sets that were like in the thousands right or at best tens of thousands um painfully manually labeled and curated what would you need for language what would you need for to teach a system about the concepts of of our worlds so well how many how many different classes are there for imagenet there's only um a thousand classes and they've got 1.2 million labels um but my rough approximation of how many concepts there are you know wikipedia has nowadays around five million entities um it's sort of like this so it's actually slowing down so i think it's going to approach maybe around 10 million entities in wikipedia eventually so like if you had 10 million classes of things uh how much data would you need and assuming um sort of like uh the amount of data has to grow um super linear to the you know i would say quadratic to the number of classes i think um a a good goal is to shoot for like a trillion labels so this is sort of like the sort of a leap of faith or a belief is like how could nlp be changed um if we had a label set that was had a trillion labels in it right tag data um maybe the techniques that we already have you know parsing and grammars and using probabilities would lead to an intelligent system and so this is something that i started thinking about maybe a few years back and i started thinking about how could we how could we build like a machine that could output a trillion labels like what sort of conditions an environment would would have to be there and if we could get a trillion labels just you know just maybe we'd have a shot at building uh truly intelligent systems um so just think about that for a second like as an exercise like how would you get a trillion labels um well there's some things that um obviously wouldn't work if you think about that kind of scale right so human labeling isn't going to get you a trillion labels right so creating and annotating data sets if you look at you know imagenet part of the reason they were able to build this data set you know obviously they had a team that was motivated and cared about it but it was also because there was technology that they leveraged here so they um they used um google image search and google translate right to query these um syn sets and get a whole bunch of matching images and then they use like things like mechanical turk right to go through and make sure that those images were of that concept right and so those things didn't exist before so even if um that's this is the reason sort of like people had built you know label sets or images in the past but they didn't have um google image search and mechanical turk right so there's some enabling technology that needs to be in place um uh so human labeling itself will never scale to this number of labels um and it's it's also i think pretty obvious to me that you know any kind of semantic markup or micro data or other formats they'll never like they'll never get you to that scale either and those have a host of other problems it's not comprehensive enough you'll never be able to have the perfect schema people don't mark up their pages with um the schema they lie sometimes when they're marking up the pages there's all sorts of you know weird interests so if you think about you know trillion labels obviously the the data source that goes in and then trillion label comes out it's um that data source has got to have more information than the output right and if you think about what sorts of corpuses um are of that scale where you could have more information going in than coming out um i think it becomes really obvious that the web is um the best resource we have of human generated information and documents out there but you know now you've got a bunch of web pages right so this goes back to building this machinery again this factory basically what can you do with web pages how do you go from web pages to labels right um and so there's a journey that our company is taking to try to try to get closer and closer to that but we're basically building the the fundamental layers that can take web pages and transform them into cleaner and cleaner versions automatically uh and and output these tuples so what diffbot does is this this is sort of our key insight is web page extraction is a computer vision problem um so this sounds like non-intuitive to people other people that do scraping or webpage attraction you know they see this as regexes and you know writing patterns against the markup of the page but the way we see it is um here's here's a the print version of the new york times here's the the web version of the new york times um these are essentially scenes from the digital world that aren't random they're generated by some underlying model and we as human beings can parse them and make sense of them instantly like even though you can't read the text of this this probably just looks like a blurred line you know from you know 10 feet you know 5 feet or 10 feet back you can still make out what this is instantly you can you can tell that you know this is probably a headline here's the text that goes with it you know you can't see what i'm pointing at but you probably know this is the author right there's just so many conventions in here that date back to thousands of years that to sort of have you know there's a generative model and how these web pages are organized that um that humans can instantly parse right so um how can we teach or teach an algorithm to essentially simulate the same parts of the brain that we all employ when we're looking at web pages for you know hours on end you know in a given day right so you think about how how much brain power and aggregate human beings are using just reading and understanding webpages right now and how much you could save if you had an algorithm that could do that automatically so we use computer vision and we're building the the base layer of this machinery right now from human um documents that are intended for humans to read how do you get um a more structured version and then from there we can get even more structure and even more structure and i'll explain the steps we're taking you know along that more structured path just as an fyi we now have a index of um this sort of what we call layer one lightly structured objects that is in the in the we have about close to half a billion and in terms of the tuples we have around three to four billion right now so we the dipot technology as a backend service powers parts of you know the bing search engine like duckduckgo instapaper whenever you are like insta papering a news article it's basically passing that url into default servers we're converting it into a structured article and that's what shows up in your in your instapaper account and uh various different other companies you know like radiant six use us to monitor um what people are saying on these web pages and we just released a discussion api that can automatically ferret out the discussions so let me uh so we think that the right approach and we've if i use this we've tried every single approach we use whatever works is using the visual features another reason is you know here's like the markup for like this page right there are so many different ways that the html can look um that generate the same look to the human right so why not just work at the layer that human at the human sees right rather than trying to infer right uh based on a whole bunch of hard-coded rules and and things of the html syntax which is unreliable and which go away basically when we move to a new kind of document presentation technology like if you go to go to flash or if you go to like html5 then you know those conventions and programming things all go away right so it's another reason why i think visual is so important um let me explain the overall architecture of how doofbot works so we basically start out the easiest way of using default is you pass this a random url and the first thing that we do is we have developed our own custom render that takes in that url and simulates what the humans sitting in front of their computer in a browser is doing their browsers rendering the pages executing the ajax it's downloading the css we've hacked webkit so that it's about 10 times faster than than chrome we've thrown away like things like memory management uh bookmarks um all those things that are human affordances that bots don't need when they're analyzing web pages and then the the basic point of the rendering is to produce features for the machine learning algorithm so our um machine learning methods take into account everything that the human sees about the page so we have pixel level information x y height with color alpha positioning information layout information as well as everything that the browser has access to excuse me so because we control the rendering process we know um the byte sizes of all the headers we can see that the time of flight how long this http request took we can see the video stream that's being played in that video that's on the page and analyze the frames of that video we can see ajax calls things that are in the headers so and we can also see the markup too so we can also see what's here so why disadvantage your bot right if you're trying to build a bot that's super accurate and people that would people would pay money to use right you want access to everything the human sees up here you also want access to everything that the program sees down here and uh there's information that we can mine from here too so this is also a language that's by the way a human it's intended for humans so um who is it that said that that said source code is mainly for humans to read right secondarily for computers to compile so we built language models for the program the variables in the programming language we built language models for urls right so you can think of the analogy the analog of word back but for urls which have a different generative model than the natural language right but but it's also a human language in the way we format urls and and css classes and things like that um this is what i mean when i say div bot has you know literally like 20 to 40 classification problems it's uh or our company is basically a machine learning algorithm with a light api wrapper around it so then we've got all these features and then we've got label data we've also got a whole ton of unlabeled data from crawling the web and from urls that customers and partners provide us and we use semi-supervised techniques and we use a lot of them so it would take me too long to describe all the different machine learning techniques and hacks and we use whatever we can get our hands on and that works and we have we have methods of um evaluating the um the performance the precision and the recall on on a nightly basis and pushing them to prod and we have methods of seeing a web page and having that update the model and and being pushed to prod and learning about the site structure as it's crawling the structure so there's a um an interface that the easiest interface of using the default as a product is you pass it a url it performs the classification it determines what kind of page it is and it extracts it as structured data another way of using divat is you give it a domain and it'll crawl the whole site so you can say give me all the products on jcrew.com and you get back just this structured database of all their product catalog that's queriable so when we're doing that crawling we can learn about the generative model of of the pages by seeing a lot of pages on the site right so there's a lot in the machinery and and last year we acquired a minor search engine so we have um this really robust crawling infrastructure that's capable of crawling um the web which very few entities do nowadays in the us unfortunately when i so when i said classification um let me describe how we think about that so if we go back you know our goal is to try to basically convert the web into a structured format so we can get a lot of label data that can be used for all the different things that you guys you know are building in this room right all kinds of things that can be done once you have clean text once you have the entities that are in data in a more structured representation so in order to achieve that goal we want basically our bot excuse me when it encounters a page to be able to automatically determine what kind of page that is and extract it into that that pages schema right without any you know human intervention so um this was about two and a half years ago we partnered with twitter and um one of our customers duckduckgo so this is like from the twitter firehose sample so this is a a particular sample of urls it's sort of the shareable web so it's a biased sample of urls but this breakdown shows um what types we found right so 36 of tweets the links in a tweet are to an image page so this was information that nobody really knew before i mean they knew that some percentages instagram right some percentages twitpic but they didn't know that um we sort of created this on you know web page ontology um 16 of the urls that are shared are two article pages nine percent are two video pages eight percent are product pages seven percent are front pages uh nine percent are these um so and the page types that givebot has released in production uh include image article video product front page and as of two weeks ago we also have discussion uh in faq so we can of the shareable web we can process around i think it's around around 75 to 80 percent of the web just and when it's one of the ones we don't support right now we just put it into other so we'll just save it for later um but we think that if we um after a few more months of development we can complete these other page types and be able to get like 98 of the kinds of pages that are on the web and then at that point in time then we can actually start um we've actually already started actively crawling the web and starting to both convert the web into this structured form um let me give um oh uh product api so let me give like a demo so let's see if this works all right that was a little fast but that happened live so um so if you remember back to the first slide uh arnold schwarzenegger saw the harley and he determined it was good so here's a product page this is typical of what our customers do you can just pass this page or any any page you can just paste it there and diffbot will say it's a type product it's a product and the product's title is this right this is this description of the product here's the price of the product it's product id here's you know it's found the image of the product and the image has a caption it's determined that there's an offer on the page in sku i think there's more too there's a json view human language english that's an easy one so you can imagine this this spot right now is crawling the web and automatically doing this for every product page out there and now we have a lot of these on the order of um close to close to a billion um so we still got a lot to do so the next basic steps in our roadmap are we've now got a whole bunch of these json objects you know some of them are from us crawling some of them are from customers and some these high vault higher volume customers that send us urls and they're just all just sort of like there in this big goo right now right we're working on um normalization and entity resolution so this harley-davidson you know that's sold on this one site and this other site right now are two separate edition objects but they are actually you know they're the same product they're just being sold by two different places there's two different offers right so i'm working on methods of fusing those objects together so that we have a coherent entity so that that's an example in the product space but you notice one of the other page types is like profile so that's people right so i have a linkedin page i also have maybe a facebook page right and there's different information on there but it's really the same person and those those fields should really be merged right so how do you do that at large scale across the web reliably so we're building it up in layers but the goal is that at some point as the bots crawling the web and discovering more and more pages the our confidence level about the objects in the in the real world just increases automatically without any um you can imagine you know if you if you see this hardly enough on enough different pages you start becoming increasingly confident that you know that harley has has a certain mileage it's certain dimensions you know it comes in these colors um you start becoming increasingly confident that this chair um you know is is white has four legs um has a certain height and width um it's certain price at different places just from seeing it a lot right so as it's crawling the web without any human input we want to get to a point which we're not at yet where it's just increasingly confident about uh all these entities and it can unify them in a way that you can now query across it and say you know give me all the give me all the objects or or give me all the things that match these criteria and then sort of like an sql kind of query and just get back all the matching things see um where the those are referred to on different pages on the web right so you could actually do stuff with it right like you want to buy that harley right then you eventually need to to go down to the guy that's selling it and um we have thing we're working on things that can actually execute the purchase flow completely visually like click the add to cart button put in your payment credentials um put in your email address and your home address so that the motorcycle arrives at your doorstep right so that's that's the overall vision of the company um so that concludes my presentation um the final thing i want uh so thank you all for listening to it um for everyone that's at this conference um just like as a um uh just like a token of appreciation you know no pun intended um i like to give everyone here actually a free paid api token to our standard plan which um it costs it costs 3 600 a year like you know that you pay for so for personal use yeah so no one outside of this no tweeting or sharing but if you want the way to get it um is just when you generate a token on our site just say that you talked to mike or something at text you know tbtv thanks a lot i love to open it up for the questions discussion yeah um so yes so we do save it but then um obviously uh for things like products the the properties of that object actually change over time so some of our customers want to track the price over time and stuff so they'll recall it again so you'll have to know um when to be able to use the cash and when not is that what you wanted to know okay yeah um so we sort of purposely set up our company in a way where it would be unscalable for us to tweak anything because um people can send us any url right and it's gotta work um it's also a paid api so if it doesn't work we don't get paid so um there's sort of a forcing function that it has to be really good that's very precise um and we're a small team so we're 14 ai people working out of a house in palo alto we're all technical we don't have any dedicated sales or pms or anything like that so we don't have we're not the capability to do customer specific like fixes and things like that in our product we offer a way for the customer to train device so if they find a page where they actually want to add a field that's not in our schema like let's say i don't know it's something about i don't know car site and there's some feeling there that we don't extract there's a wysiwyg interface that allows a pm to just point and teach default you know extract this and then it'll add a fields in real time to their api calls and that also adds a label to our training set that we can then use to refine our model one million pages well it depends on if you do it sequentially or in parallel it depends on the robots.txt policy it determines depends on whether sites block you it depends on whether there's spider traps or other weird crawl loops on the site it depends on how complicated the site is and it's what's needed to render the page right if the site like it's one of those where you have to like scroll down for more stuff to load and there's like this infinite scroll and stuff so diplo handles all those kind of cases um on some new sites like a pop-up will come up right and the fight actually can visually identify that pop-up and close it and uh and follow the next button all the way through to like the ten pages of the article needed to concatenate together that article right so um totally depends on the site it's um but it's roughly you can think about it as um your like how long would it take the site to appear in your chrome browser right because we're using the same core under rendering engine it's just we made it you know slightly faster yeah so well i just gave an example right of the way diva can actually interact with the page to um to end up at the um the state that it wants to analyze right so your what is your question though it's uh how do you how do you do interaction or yeah we see it really as an ai system you know so you've got a renderer here that displays a certain state at a certain point in time right that's like one frame of a video that's playing right so you may see a pop-up come up and visually identify that then you have to send an action back to the render and tell it to close that and then then you have to serialize the rendered snapshot of the page again right in order to get what you want or in the case of the next you have to keep clicking next or keep scrolling to get get everything yeah yeah so i mentioned like i looked at the attendees and there's like like six or seven different customers like that that came to this uh so they're wide ranging um they anywhere from uh just nlp research or getting a ton of articles to news news applications to products um e-commerce to um now with the discussion user review mining discussion mining brand monitoring content marketing um does that give you a sort of flavor of the clusters yeah but we see it as you know these these apps are super critical to us because they provide the training data for the system as as they used to bought and then at some point you know we'd like to you know open up this um this essentially large structured index and allow people to contribute you know even more structure to it right people doing processing on layers on top of default should be able to write that back to the graph so that other people can use it but that's what's i think is going to get us closer to that goal of getting australian labels yeah that's sort of like um one of those like classic nlp questions um whenever you have some sort of fixed ontology um i'll tell you what we do right now which is i told you we have this like big big goo of json objects right and we don't actually try to resolve them into hard clusters at index time what we do actually is that query time we have a really fast clustering algorithm that can query the things that we think um go together and present that back uh with some with some threshold but we're still working on that it's um it's a good it's a fair point whenever you're clustering things um think of it you know that you have to choose like what's your cutoff point like what's k right or if you're doing hierarchical clustering like where's that line for different customers it may be different things for in the new space right is all the news that about is about the same event form a cluster or is it you know this same news article that's um on many different sites a cluster um so um we don't know yet uh maybe make it uh user selectable like that threshold well i mean our goal isn't just to hit some vanity number our goal is to try to produce an intelligent system right so i don't yeah we have um let's see there's um this is like the best study we know of um it's actually on github it's like um some phd student did it as his master's thesis comparing different text extraction algorithms so this is keep in mind this is just text extraction so this is like the article api which is only like one of like the 20 apis we have right so because we don't even know of any other libraries or automated methods of doing product extraction there's just no one to compare to but for article um this guy got a decent data set um and here's the f1 score is comparing diff bot to boilerpipe which is open source uses some machine learning readability which is a bunch of hard-coded things in javascript alchemy and bedley and goose but you can look at that if you want you could download the data set to if you want to validate it yeah it's it's still pretty nascent right now so we'll extract some things like the video metadata so we'll try to get some tagging uh duration um the discussion functionality means we can get the comments of the video um but we don't we don't do a lot of deep analysis of the video itself right now because it's too expensive yeah okay oh okay yeah so um i sort of mentioned before the machine learning algorithm has access to um all the webkit internals so that includes the um so we're not we don't actually have to do ocr on the image to get back the text that's already there right it could but you know it's already there so why do that so we can because um we have hooks deep we've basically instrumented the webkit render we know that you know this rectangle on the screen corresponds to you know this text node and the dom and can just get it directly but many times in the article api the correct output doesn't actually isn't actually even a dom node it could be um what we call a disjoint node list or a whole set of nodes that aren't even continuous on the page all right so all you have time for yeah i'd love to i'll still be here so i'd love to stick around if you guys have um more discussion or questions you have interesting ideas but thanks a lot you