data.bythebay.io: Anand Kulkarni - Building AI that Searches and Sells
Recording: data.bythebay.io: Anand Kulkarni - Building AI that Searches and Sells
[Applause] hi everyone and thanks for the kind introduction um so uh as was mentioned I'm the chief scientist and co-founder over at lead genius uh we're a technology company based in Berkeley um today I'll be telling you a little bit about how some of our internal technology operations work uh in particular how we' built our core engine for data collection data curation and making data predictions um and specifically I'm going to talk about how this works in the context of sales so my own background is in mathematics um and in theoretic computer science so moving into the sales domain was really new for me um but it turns out that a lot of what we're doing today in the sales universe is uh uh a problem space that is really interesting from a technical basis and has a lot of lessons from other as well um so let me start out by making a little bit of an audacious plan um which is that uh I think that in in 10 years um the job of salespeople is going to be largely replaced by artificial intelligence and this is a bold statement um what I mean by this isn't that U we're going to fire all of our salese and put them out of work um but what I mean is that we're going to have a system that is more like the systems that we've seen take place through mechanization of other fields where where uh you have a lot of people who are still in operation but wielding a set of very sophisticated technology tools that make their dots easier so you'll have a few people who are pulling the uh letters in a big operation for uh for sales and they'll be assisted by a whole lot of software um some of that software is software that we're holding at lead genius um but for those of us who aren't in sales uh I don't come from a sales background either um in might VI usful to briefly talk about what sales people do all day um so what do sales people do all day well salese find companies or people who might want to buy something the old school way they have a rodex they flip through it the new school way that they might browse through Linkedin go person by person check each one of these people's profile and see if they want to buy something or not they might think a little bit about each one of those companies and try and decide uh if they want to buy what their company is selling um you know the old school way was a system called B which is see if they have the budget The Authority the need of the time the new school way is you use use some Advanced tools like HubSpot that do a little bit of behavior analysis to try and see if somebody is really engaging in your product or service and then they actually go through and and sell those people and that's through the form of oneon-one uh outbound communication and Outreach old school way was You' call them up You' work we on the list say hey do you want to buy something no sorry hey do you want to buy something yes great um the new school way is people who are sitting on laptops all day um typing away sending emails trying to figure out uh if people want to talk to them or not so uh I think about these is three technical problems um one is a search problem one is an intent problem and one is an actual natural language communication problem um and we call this the search Problem the intent problem the sales problem um over at Le genius so um our goal as a company is to build out software that addresses each one of these points in turn so we're building AI that finds companies um AI that tries to understand buying behavior and an AI that actually talks to people um so uh let's talk about each one of these problems in turn we'll start out with building AI the search to customers um the actual search problem so um the way we think about this problem is this um a customer might come to us and say well um we have a product we want to know every single company in the world who might want to buy this product um can you uh give us uh a list of those companies um that's a very hard problem um we'll start out by solving a slightly different problem which is how we can just come up with the database every company in the US and then we'll look at how we can filter that down to companies that might be a match for a particular customer prod okay so um the easy way is to start out with a data pipeline um the way we've done it is by craing data for about 55 different sources um start out with a whole bunch of business directories online um we've also used a few of the public apis like Angel list um we've plugged into all the Secretary of State databases um some of these are not neatly accessible um some of them are providing you free information but they don't really really make it very simple for you to crawl government data um and sometimes they'll just send you a CD um there's also SEC filings and then there's lots of public records that are easy to access like the IRS nonprofit database um and in our system our pipeline is set up with Scrapy and beautiful soup I'm running on a virtual machine cluster um we can take web pages raw web pages uh that have lots of information in them and take them apart um it takes a human being a little bit of time to set up um uh parts structure and be constructed but once you've done it you can uh run the schema pretty easily we're doing this on a pretty large clustered machines um you can see that we've got uh various parameters that we're pulling straight out of the web pages pretty straightforward process um for information that is a little trickier to get um if sources are not very uh friendly to obtaining information um for example for those Secretary of State web pages um you'll find that they're built in uh Frameworks that are not really amenable to crawling um we've used tools like Ai and selenium to simulate browsers to actually grab the information it's relatively happy uh because it's it's brittle uh depending on the page structure but it actually works very well you can grab information pretty reliably from those systems okay so um here's an example of how efficient this is in practice um we're running a compute cluster virtual machines on uh Windows Azure um to pull about 7 million 7 and a half million companies which is what we found in one particular um State's database um we could grab all the information process it and link it into our directories about half a second per organization um takes about 43 compute days or $50 in virtual machine time um and that pulled out 202 GB of data so pretty efficient pretty effective um the data once it comes out looks like this running a linking process behind the scenes to match these companies up with one another but you can see it's basically firmographic data right things about the company the web domain um the organization size the revenue Etc um and this is a compilation from different sources so we pulled information um that wasn't just in one particular record but found in multiple sources so it's difficult to get say the organization size just by looking at the um say the Secretary of State database record but it would be easy to get the organization size by looking at the companies about us page um so we grab information from both those sources and combine them behind the scenes okay that's a tricky problem um there are lots and lots of companies inside um all the different sources we might look like like that have very similar looking names or otherwise might be similar um in the various parameters that they have so for example you can see here on the right um some of the companies are database that have very similar names um Ortho is a very common term it's used in lots and lots of different uh data sources and data uh excuse me lots of different data sources to refer to very many different companies um it's a tricky question to ask if two companies that have a common name like Family Dental or family Ortho are the same company or different compan so here's the problem um you can think of how you might solve this how do you tell the two companies that are being talked about on different places on the web are the same company or different companies so another problem that comes up often is that companies change their names sometimes those name changes for cosmetic they've just changed the name from one name to another but the company The Entity the legal structure are all the same sometimes companies have establish subsidiary relationships with other companies uh so they're actually forking themselves off into two different companies uh that are no longer the same entity um and that makes things very tricky when you're trying to figure out if web content refers to the same company or different companies you have to figure out very quickly if these companies are the same or different so um here's how we did it um I'll give you two solutions the first is the fancy way the fancy way is to turn a company uh P into a vector of 30 different properties that we know about it and these properties are uh different properties that we pull from the company's record in the particular source that we looked at so for example um from a company's website we want you to pull their name their address their industry the founding year um and the Technologies they're using um from the Secretary of State database we may find information about the organization's name and address their revenue and their industry um we turn all these objects into U vectors in feature space um threedimensional feature space uh you can add or remove features as desired to get a smaller or bigger detector space and then you just run a nearest nebor Alor on these and the probability of two companies described in different places being the same is dependent on this form formula here which is just the uh distance that they are in in this feature space um and that's it it's a straightforward approach although you have run over a very large number of companies the distance between any two companies is the probability that they are the same company okay so uh you can decide if two companies are the Same by drawing a little circle around uh two companies in that feature space and deciding uh if they are in the same Circle in the distance between them is small enough then they're the same company and if the distance is Big then they're probably different companies and the when that value of Epsilon is is tricky and we obtained it experimentally by running this over enough examples and labeling all the mistakes and getting it to be small enough that actually worked okay and so here's what that actually looks like when you run it um you end up with a whole bunch of uh um probabilities and uh outcomes for different companies okay so this works it's pretty slow um it requires us to do a lot of pairwise comparisons around a huge number of data points and data sources you can optimize this uh we did optimize it data falls out of date constantly so we have to rerun it quite often um and when you optimize it you can produce the number of comparisons to being substantially less than n squ um but you still have to do a fairly large amount of compute tensive work there's a shortcut you can apply um if you saw the last talk you might have an idea of what it was um there's an easier way to solve this which is to look at the small number of parameters that are hugely predictive and less likely to change and there are features that do that um the easy way is to use the USPS standardized address for the headquarters of a company this works in a lot of cases um and if you standardize the format to a normalized format uh given by the USPS um that one parameter is actually hugely predictive about the inity of a company companies may change their location but they tend to change it across all of their listed official registrations okay and you can use a simple theistic like this two companies are the same if and only if they have the same official physical address um so so one thing you might think about and this I'll leave this one up as a question for the audience all the answer is not tricky what do you do if this company moves um how do you update your records land back okay great so um of course we've done this craw um we we analyzed this over the entire us um we came up with about 21,700 businesses in the United States um it's not too far off from the official US Census readings um about 6 million of those are what we call interesting companies um and by interesting we mean that they have more more than one person working in there um I put up some comparables here about how other sites have listed the number of companies that they track um yelp's got about 47 million establishments or physical locations some of those are the save legal entities some those are different linkedin's only got about 2 million unique companies front space 700 650k it's gone up since since this talk was put together angelist got about 300K same story it's gone up um we can answer some interesting questions using this data um here's one example which US industry is have the most distinct organizations um when I say industry I mean say retail or Finance um based on the LinkedIn uh self-identified parameter the answer is construction um you can see some of the numbers here number two is real estate those are the largest number of different uh organizations that are reporting themselves other question you can ask how many Fortune 500 companies have websites answer is not 500 it's actually $4.99 and I'll leave that as an exercise to the audience um or come ask me afterwards if you want to find out which one that is one okay there's a bunch of bonus problems I won't go into but I think you're really interesting and I'd love to talk about them uh if we had more time um understanding how long this information changes uh how long it's accurate you can model that with Decay function your data and grab data selectively when it gets better um you can optimize the frequency by which you re information using some expectation maximization algorithms and then we have a great way um to grab information from sites that don't have clean schemas just by watching humans do it and learning what the schema is through an inference method um but we'll come back to that another time okay um I'll talk very briefly about the intent problem building AI that understands buying behavior is a very big problem um we've taken a very small corner of this um and started to address it uh basically the problem is formalized like this you come to me as a customer and say um I have this list of companies who have bought stuff from us in the past which company is going to buy from us in the future okay so um very tricky problem it's not generalizable ac across Industries very well whether somebody's buying something depends very heavily on the specific industry and whether a company needs a product is always changing so it's time dependent as well um of course one way you can solve this is with classification algorithm um this is sort of the naive approach because we have all these companies that are already parameterized um we can pull in some of that data as you've got about 10,000 customers you can uh um make a make a stab at this um you can train a neural network to do it um we've done this to see how well it works you can learn a function to try and classify it um and try and decide if they fall into the category of going to buy now or not going to buy now okay um so you could do this uh to try and match whether companies are inside of buying profile or not it works pretty well to know if a company matches a set of demographic parameters it's hard to use that to actually predict if a sales outcome is going to happen or not um what we've done is this um we pull a probalistic estimate on whether this company is going to be in market for or not it's basically based on how similar that company is to companies that have already bought from the from the company in the in the past if that probability is low we escalate it to a human annotator who takes a call and that call is just based on um a similarity assessment they're using informal branders for they could choose to add features to the model or propose features to the model we review those manually to decide if those are good fits or not and then we add that data to the training set um now this is basically what uh a salesperson might do on a day-to-day basis we've just taken this and given it to an annotator instead um and that boost the likelihood of success to human levels but um only to the level of the human who's doing the labeling so uh there's still more work to be done here um but you can do a little bit more that's more interesting um you can look at behavioral data we've started to incorporate this but um we haven't taken most advantage of it yet um behavioral data might include things like if they've looked at websites that are relevant to the technology that you're selling if they've started hiring for roles that would use the technology that you're selling if they've talked about this in Social settings all this information is in our database um we've yet to incorporate this into our prediction methods but we have a feeling that there will be some significance to it um in terms of deciding uh for decisions okay the last thing I'll talk about um is the last step that sales people do which is communication uh with people which is building um a that talks to people um now actually building communication uh interfaces or chat box is a tricky problem to cover all the bases but if you're talking about uh this problem there a couple advantages that we have um first of all we're doing it over email so we're talking about how you can sell people products or communicate with people about purchase decisions via email so you're not time constrained you can take as much time as you want to come up with an answer within reason the second is that your conversation domain is highly constrained you're only talking about do you want to talk about buying this product or are you not interested in this product so you only have a limited number of things to choose from given that we have a stab at passing it so our challenge here is this how do we write sales messages by email that are going to be as compelling or as good as those sent by a salesperson Okay so there's lots of plausibility arguments um if you look at results from Deep learning there's lots of great conversational systems that are solving way more difficult problems than this right so xiaomi um has done a whole bunch of good work on uh real world chat bot scenarios there's also been good support chat results train. data um you can see lots of these and lots of great results this is a nice support inir from Google um you can see that it's got a really nice natural language conversation that's happening in real time much harder than what we're trying to solve we break this problem down into two pieces the first is writing a message how do you compose the content the second is how do you understand the response classify it appropriately okay so given the topic an author an inbox and a recipient we're trying to write a message that has a maximum likelihood of somebody writing back and saying hey yeah I do want this and it should look really plausible it shouldn't look like it was written by a robot or by a um like a Spam engine it should actually sound like the author and it should only be going to people ideally who actually want to receive this kind of message it shouldn't just be going out to anybody okay and we also have a couple of secret weapons in here because we have access in our systems to a set of data from our customers which is their historical email conversations that they share with us to help food strap this system and then of course we have a network of annotators that we can choose to use if we want to to help anain our ucation okay so our approach is this we craft a template based on analyzing sender's email inbox and looking for things that look like outbound messages we collect data at scale that we use to fill in the blanks and then we customize that data based on all the stuff that we've called on the web okay so here's an example of what that message looks like here's a template that we came up with um it's got a bunch of fill in the blank spots and the fill in the blank spots are populated by information that we do that we grab during our crawl um we've done a bunch of analysis about How likely people are to reply to US based on the length the tone the subject complexity and the word choice we optimized this a little bit um we've gotten some results I'll share here um subject lengths that are longer the subject is longer people are less likely to open your email um if you include variables like your name if you say Hey Joe in the actual title of the email the subject they're much less likely respond um which is a little bit counterintuitive but if you think about it you can understand why it sounds very unnatural um and then of course we published some work on this too that you can use humans to try and edit those templates after youve generated them algorithmically to try and come up with good results and we've got some Publications on this if you want to take a look okay the last problem is how to understand the responses okay um we try and classify responses into eight different types people who are saying yes I'm interested positive replies um people who are saying I'm not the right person but go talk to so and so um and then variations on I'm not interested like I'm not interested but talk to me in six months or I'm not interested and never talk to me again we classify all of those um so we've done this via sentiment analysis sentiment analysis works pretty well um we uh were able to show that we can get pretty good results out of it and then as usual you can escalate it up to a human um as the easy way if your classifier fails right um and then doing this we've gotten a pretty robust label set that we've been able to use um to train our overall system you can take this further and script responses I won't go into that in the interest of time so that's it um we talked about three problems uh searching for companies um companies that understand buying behavior and intent and uh software AI that can actually comp and hold email conversations and uh we'll talk a little B about how you can solve each of these problems in turn that's it um if you'd like we can take some questions and uh please come talk to me we're hiring no great we'll have questions great we'll take questions off live then thank you very [Applause] much a