Learning From the Diner's Experience

Text By the Bay 2015: Sudeep Das, Learning From the Diner's Experience

Recording: Text By the Bay 2015: Sudeep Das, Learning From the Diner's Experience

so let me just start here so I am a data scientist an open table my background is in astrophysics I'm not really an NLP person but I've been poking into a lot of rich a textual data open table for the last year and we see a lot of very interesting things which I am trying to share with you today at sort of a high level so just a little bit about what open table is how many of you have used open table yeah there's a everyone loves open to that's great so we are at about 32,000 restaurants worldwide we are presence in UK besides which is besides the US we have presence in UK Germany Japan Canada Canada and Mexico and about 800 million dinars have been seated through the open table system since its beginnings in 1998 we seed about 16 million dinars every month you know and these diners they go they have the experience at the restaurant and then we followed them up with writing reviews so they produce a lot of textual data for us which are in the term in terms of reviews so basically the goal at opentable used to be just transactional in the sense that you want to go to a restaurant you already know you want to see whether there's availability of the restaurant you would go and reserve the restaurant through our system right but there is a a big change more recently in which we want to move from that transactional side to more of an experiential side where we basically start powering a much better dining experience not only during while the dining is happening but also before in terms of helping you find a good restaurant and also after dining in terms of you know leaving feedback and so and also we our customers are really the restaurant so you want to use a lot of our data science capabilities to give feedback to the restaurateurs as to how their business is going so that's basically a lot of connecting dots between the on the left and the restaurant on the right here and you can make a very nice experience a magical experience happen you know before during and after dining if you understand both the diner and the restaurant very well so on the down your side you would like to from your past interactions from the reviews you have left from your ratings go to build up certain profile of the diner as to who you are when it comes to dining you're that guy who wanna Friday you really wants to go to a cheap Chinese restaurant but maybe you on a special occasion you want to go for the you know upscale French place right and then there is the restaurant on the right-hand side I want to know from mining all the reviews of the restaurant or all the metadata we have of the restaurant what this restaurant is about right so how should I match this restaurant to that user on the left right and and that's where also a lot of the textual context comes in you into mind everything we know of the restaurant that comes through textual data and then there is also a lot of data about past interactions so you saw the users the diner's go to restaurants they leave reviews also they rate and just the fact that you search for a restaurant or you you know transacted a restaurant is a signal and which we use for our collaborative filtering or our recommendations that but today because it's a text you know text focused conference I would basically just focus on what we're doing with the text that opentable text comes in various forms that open table but the lion's share is about 30 million reviews that diners have left after dining then there is also a little bit of text around what you know diners put in when they make a reservation as special requests right and you see these things like peak during say valentine's day because people want a rose on rose petals on there you know tablecloth something like that and I'll get to that later then there are menus which are just you know digitized menus we have access to and there are we also look at external data like social Twitter Facebook to gain insights about you know when you roll out something new how that's being perceived by buyer by the diners so moving on I just start with what we're doing with reviews first our reviews are rich and verified in the sense that the person has to have gone to the restaurant and experienced a meal and the hole and then he's followed up and he writes a review so not anyone can go on our system and write a review and they come in all shapes and sizes there's a you know pretty sort of fat distribution of the reviews as you can see here they can be very small like one word like superb that's it right and then they can be these reviews which are very very verbose and they talk about all every step of the experience from arriving and meeting the Matri to sitting down going through appetizers means desserts and then also you know maybe some after thoughts after dining so and then the and the data is the review data is very rich because you know these are people who are very passionate about their dining experience and and they write at length about it so when we started to look at or wanted to approach this data one thing as sort of clear before hand is that when you look at you no reviews in a certain domain like restaurants they are going to be about certain teams they're not going to be about everything in the world so obviously we expected reviews to talk about these main points which is food and drinks there will be something about the ambience there's service value for money you know whether they got enough value after howard is spent of the restaurant and there will be like special occasion topics in the sense that there are restaurants which are very very good for celebrating something like a birthday or anniversary so this sort of led us to attack the problem first through topic modeling because topics these are sort of topics that we want to learn from the reviews so what we did was we jumped in with a bunch of tools like lda and non-negative matrix factorization but overall this is the sort of direction of how we approach the reviews so instead of taking all the reviews on our system at a time we sort of thought that there might be geographical nuances between you know reviews in the san francisco by a barrier versus reviews in chicago vs reviews in london so we analyzed the corpus of views broken down by geographic regions and then we learn topics and then we and the topics come out to be very nicely categorized sort of you know if you look at them you can buy I categorized emojis the food topic is the drink topic and i'll show you some some of those and then map these topics back to restaurants because many restaurants have thousands of reviews and there's no way you know you can read through all of them to make sense of what this restaurant is about but once you've learned these higher level topics you can go back and say this is a restaurant that's about seafood and views where this is this other restaurant is about you know a meet and live jazz and then because you know the topics are basically learned bottom up from the reviews once you learn the topics for a restaurant for every topic that's important for that restaurant you also can tie it back to the reviews so you can say show me all the reviews that are most relevant to the view topic and these reviews will have a lot of information about what sort of views you get up the dress draw so you know we experiment with a bunch of tools but you know it's pretty easy to actually learn this topic very cleanly with non-negative matrix factorization there's one sort of detail in there is that the algorithm needs to be initialized with the nnd SVD version so there are various version you can you can see the algorithm with but to get interpreted Lou topics through n NN f true n and n MF that initialization is a we found it to be very important so what do you do you have a document and word matrix so in vocabulary could be something like hundred thousand words and we have say 20 you know 10 million reviews or whatever and you break it down into these two matrices with the constraint that you know it has to be non-negative and you can then interpret these things which is suppose we learn 200 dimensional topic space so these this matrix T up here is 200 by 10,000 matrix and every role in that you can interpret it as a topic and this is how the look so basically you can see by looking at the topics here is a fish and a very efficient chips grill tackles the fish centric topic on the Left there's topic we will talk about noise that the restaurant is difficult to do conversation and then there is views of the bay bridge so this is all from San Francisco you can see there is Bay Bridge east bay bay views parking being easy or very difficult in ballet there is sushi the server being helpful Sunday brunch grab and Dungeness is very very San Francisco topic here and then there's the topics about celebrating an anniversary or wedding on the bottom right so so you can see that topics come out to be very very thematic Lee you know condense and and and focused on a very certain specific aspects of dining and so it's not very hard to just go and categorize them into a food topic Syria steak and dessert on the Left drinks wine and bar scene ambient music in and view and then value service and occasions right so people go Valentine's Day Mother's Day Father's a birthday and that's a topic by itself so that's all good so how do you use that right so you now have all these likes a hundred topics that you learn from all the reviews but how do you go back and use it so basically the first thing we did was to collapse the topics for all the reviews within a restaurant so basically you suppose the restaurant has at 500 reviews for every review you have a distribution of this topic weight and then you can squish them down into this one vector which is you know my average topic wait or for this restaurant and you can look at the top topics for update restaurant so suppose you know you're at a restaurant which is by the way and has excellent seafood the seafood topic and a view of the bay topic would be the highest weighted topics in and I'll show you examples here so this is just a different visualization of the topic so every little bubble here is a copy I mean bubble cloud and here is a restaurant espetus churrascaria which is in on your mission and market which is well known for its meat right it's a churrascaria of course but without having to go to the 2000 reviews that this restaurant has you can basically look at the top topics that is that are visiting this restaurant you can say it's not only good for me which you already sort of know but they have a very good salad bar and then you can also say people go to celebrate special occasions here okay and it's a services top-notch it's also very quick kid friendly so these topics basically just pop out without having to actually go through and look for whether any review really like you know mentioned anything about being kid friendly so there's a lot of stuff that's not reflected in our restaurants meta data if you just look at the description of the restaurant none of this is there not much of this is there and by doing this sort of a bottom-up approach we were able to surface the salient features of restaurant and this makes topics very in a local so it's ultra local in the sense that topics let you distinguish between one restaurant and another so if you go to fog harbor fish house which is on pier 39 you see a lot of the topics are about seafood and crab I love their crab fish and chips but then also it boasts a very an excellent view of the bay and if you clicked on that Bayview thing there will be a lot of reviews which say we love the view during the sunset so it's really you know if you're going if you're taking a friend who is visiting you to experience an excellent dinner and also a great view this is a good place to go to so and also we found that you know your own summarizing restaurants which is very very local you can also go a little bit broader in the sense that summarize what a metro area or a town is about so here we learn topic separately for San Francisco chicagoland in New York and you can see just the view topic is very different for the number of topics which has to do with views is much larger in San Francisco because we are very well endowed with restaurants that have views and then you can see that in Chicago the view is mostly about the city the window in our looking out to the lake in New York it's about hudson river or the new york skyline manhattan whereas in london most of the view topics around you know st. Paul and the London Bridge and the tower so these regional nuances really pop out when you compare topics and there is also differences between countries so it's going from restaurants to metro to a country level you can see these very different ways people talk about what what they go for on a Sunday for example in the u.s. Sunday brunch is the main thing but in the UK Sunday roast is more of a thing on a Sunday and if you look at what people say around the topic of wine we always talk about wine pairing here but the chosen word in UK is wine matching so this is something you know it does not look like you know it will have immediate business value but when you look at you know when marketing people go and write copies about emails that they have to send to these kind of small nuances are what matters so topics on certain special days this is really something i like so we wanted to look at you know what polarized people on valentine's day right so valentine's Day is one of the biggest days for us and people who leave a lot of reviews after their experiences and we wanted to know what really generates a one-star review versus a five-star review of course there are many many different reasons for that but what really was striking was one topic which is steak so if you do your steak slightly wrong on Valentine's Day you're screwed Frank so he's he she ended up not even eating it so that's like a you know like a burning bad review for I ordered medium rare ain't came burnt on both sides right a lot of reviews were just you know we could basically see that because one of the biggest topics with highest weights were steak and then again on the other side if you do your steak really well you get an immediate five-star review it's kind of awesome so these kind of insights then can lead to things that you can do with marketing you can also tell the restaurant that you know this was the reason you got a bad review and also we do have a blog that's restaurants facing we have a blog that's tech blog where we write about our data science and other technology things we have a blog which is more like diner facing but you also have a restaurant facing blog called open for business and this story ran there where we highlighted the most polarizing things about Valentine's Day and it was really useful to the restaurants so other things you can do with reviews you can find what are what is this restaurant known for what are the dishes I should try if I go there right and that can be pulled from the reviews so and also the attributes like if this restaurant good for a business meeting right so we use use thing at this thing called dish tags to highlight the popular food and drink items so right now the experience is shown here where if you go to blue suit in New York you see you know chicken tagine is something you want to try it has a picture and name click on that you land on this page on the right which shows a big review with with the chicken tagine kind of highlighted and we started this thing with a curated list of dish tags right but there are so many dishes out there and so many variations of dishes out there so what do you do to get all those dishes and this is sort of you know one of the first projects i did when i landed at open table was to basically find by grams and try grams which are mentioned a lot within the reviews but but if you did it with all the texts in the reviews you will get a lot of like non-food diagrams and ty grams and there is no way and then you have to do like me look for you know which one is food and which one is not but then what is inside that if you looked only within the context of food topics because you already have these two topics which are con which consists mostly of food words and then you do the by graham & trigram exploration with those words then you start getting you know for each restaurant what are the most visited spy grams and ty grams and from that we could basically almost double or actually will more than double our vocabulary of dishes and then basically it's like going back and finding the dish or you know and then there there's a there's a little like subtlety here because the same dish can be mentioned in many various different ways it's a long dish like Dungeness crab cellophane noodles with Dungeness crab some people who say i had the noodles with crab that's safe right so there's a lot of like this ambiguity in a lot of ambiguity in the way people refer to the same dish so there is some smarts that we do about the design be grading these when we attach so whenever you click on say Dungeness crab on our dish tag it may land on a review it just says crab and that sort of mapping is done internally once you have the dish tags interviews with a little bit of you know simple vectorization and linear algebra you can look for the dishes that are trending so basically that have been flat before things that have been flat before and suddenly low rising up in trend and this is how we spotted in our reviews the new york cauliflower trend early in February and march and a lot of these high-end restaurants which are on opentable they started making very very creative dishes which with cauliflower and then we go went back and looked at other places like Zagat and they were writing about these things you know so things like cauliflower steak I want to try that so right now for example you know artichokes this is more seasonal than trending but the restaurants are making many new and creative dishes out of artichokes and this was purely something that came out of this dashboard of change changing dishes and out goes an email to the marketing team and they write this email I'm showing a copy on the left and you may have had this in your inbox which is saying you know go and try this creative artichoke these dishes that restaurants are cooking up right now so we also learn not just dishes but attributes which is good like things like good for groups this restaurant has a patio this restaurant is great for a special occasion and this is done you know this has been done sort of with starting out with a labeled set of restaurants that have that feature and then training a large degree possession on top of you so basically the labeling is turned down by mechanical turk but the thing nice thing is when you do this you see these features which is like patio might actually map to features like these are obvious features which is outdoor but then there is raining signing smoke things you can see while sitting on the patio right so they basically become features for finding restaurants that has a patio it's very interesting the other thing you can do so now that you have learned to associate dish tags and attribute text to restaurants you can look at a user or a diner and this is our CTO I can look at his previous dining experiences right so I mean you can look at all the restaurants you have transacted at in the past and then you can build up sort of a profile of this user because you can aggregate all these attribute tags across the restaurants you can do it also a dish debts so if I only go to steak places steaks will come up as a big thing in my you know tag cloud so to say so you can see here's Joseph SS who is our CTO he goes to a lot of confer a lot of restaurants which are good for business meeting obviously for foodies yes we flew you see that opentable romantic i don't know matter with that but but you can see like this is sort of his profile and then you can also look at what cuisine types he goes to a lot and this kind of things helps create the you know sort of the dual of the information about the restaurant which is the more information or profile of the diner and then this can then feed recommendation algorithms where try to match diners with restaurants so that was all based on topic modeling and you know i'll regression and sort of doing some fuzzy matching but then we started to look into sentiments a bit and the reason for that is we thought you know we show this dish and the photo what if we also show a little snippet of a review around the dish that gives it more you know ground truth that this is something that someone really liked and that you cannot do without doing a little bit of sentiment analysis because you can pick a five-star review and it can pick the sentence that contains a dish but it might just say I tried the fried chicken at foreign cinema there's no value as to whether it is good or bad so that's why we went in and we're kind of in a nice position because when people write reviews they also rate the food rating right so here I am looking for good sentiment around dishes so i looked at five star reviews 5 star food reviews and one star food reviews and thats already gives me a labeled set of training something like a logistic regression on it and you know a very simple Hyper barometer search gives you like ninety-eight percent you know precision on this sort of predicting the rating and these are the figures so these are features for spectacular dining experience excellent recommend create service attentive good you know these are words that highly predict what a good review is about and these are the words slow salty worse disappointed it's like worst manager won't go back those kind of things are our predictive of a bad experience right so what we did with this was you know we go from a dish which is on the left here goat stew this is at coke curry which is a also near embarcadero and there are many many reviews that mention that dish right and so you can pluck the sentence that contains that dish but then you have so many candidates to show so which one do you show so basically now that you've learned a model that predicts sentiment on reviews you can use that model on sentences too and so basically we choose the sentence that is the highest sentiment about this dish and doing that we found really nice snippets that we can now surface in the app it's like Lobster and others were to die force it knows that to-die-for means really good when it comes to food right may be very different in some other of respect perfectly crispy and full of flavor melt in your mouth extremely tender so these are very very you know highly predictive features that say it's something that someone really liked or enjoyed so that was a more what we do sort of at a high level with the reviews beyond the reviews you also have diner notes and requests so this is when you're making a reservation on opentable there is a space where you can put in special requests and we looked at those for you know various occasions so we looked at Valentine's Day and this gives you a lot of comic relief because there are some requests that are pretty awesome like if possible would like a waiter with ponytail longer hair and our accent I don't know where that came from but they're hilarious draw a puppy on a puppy on a piece of paper and leave it on the table so these are kind of like requests we get but then you can do some sort of analysis as to what's the most requested feature and also how has that been trending over time and this gave us some really nice inside you like what's really highly requested these days are boots during Valentine's and you can see they're just had an upward trend from 2004 to 2014 whereas abstract concepts about Valentine's Day like do something for me that's romantic that has gone down because rest I think the diner's have figured out that if you actually don't ask for specific things they'll probably mess it up right so basically things like candles are coming up flowers are coming up there are more like specific materialistic aspects of Valentine's Day but things like do something for me find an intimate table or magnetic table those are going down over time that's like really useful inside for the restaurateurs also so they can now say Oh we'll give you flowers then the other sort of big part that all these inside goes to is building out landing pages and those pages are also very important for SEO because you keep those pages there it gets crawled and what you really want there is not just the keyword matching for what you're searching for us if you're searching for Italian in New York and the page only shows Italian the word Italian it does that does not have much value unless it has long tail value in the sense that he starts talking about all the different Italian dishes and more about the restaurants beyond just the fact that it's Italian and here you know you can sort of like some simple algorithmic approaches I mean simple how much these approaches may not always make sense because if you say oh I'm just going to show one of the recent reviews that's five star so this is lamar which is also near embarcadero it's like a peruvian place and here is a recent five-star you picked randomly it says after living in Lima for three and a half years it's good to be good to have this together ceviche and pisco fix it's okay but it's not super useful it doesn't say much about the restaurant so what we did was because now we had the topic distribution of the restaurant overall we could look for that review within the corpus of reviews that is closest to that overall distribution in the vector space so basically we can say find me the review which is most rep sentative of the overall topic distribution of this restaurant and here is one of those reviews and it's obviously a little more verbose but it said the meeting with extensive they had the sweet players in a blah blah blah presentation was gorgeous the waiter was helpful and it's near embarcadero bart with a nice view of the bay right so this has immense longtail information that can get crawled and lead to a huge SEO value and also it's very useful for the diner when you're going to the if you have a page which has lists of restaurant and reviews shown with that beside them you know this kind of a review which will basically give you a full picture of what this restaurant is about where you are going to so this is something which is relatively new because we just building out these cuisine pages we thought it but it would be nice to have not only the word Italian in New York found in the reviews but also things that relate to Italian in sort of more semantic way and there is you know I think what a better way to go about that then using some work to Vic magic there so we are actually using apache spark for a lot of our recommendation stack building so we decided to throw it into the work Tuvok implementation there but from what I hear from this conference also you know in general Jensen is also very very powerful tool to do this kind of stuff but here is you know one of the example so say you're going to build a page around sushi what do Vic already gives you all the synonyms of sushi basically you know that knows about nigiri shashimi Guardian hockey you know it's just go about it was really excellent and then there's you know sudden desire slightly more obscure term like GOP know which is very you know native to San Francisco it's a seafood soup and you see the synonym that it came up with this bouillabaisse which is another fish soup right it's kind of awesome and it sort of goes through things that is paired with usually all the ingredients that goes into into a GOP no shrimp crab prawns so this is you know we're just scratching the surface here it's very very recent like I just fan it like last weekend or something but we are going to use it to do a bill of our newer features and here is something that I don't know how much of this is you know just pure luck but I asked you know this man woman king queen kind of question in the food context so I would say if Hollywood pairs with Chardonnay watch would I have with lamb and came up with Zinfandel and the next one was Burgundy so it knows sort of that white wine goes with fish and red wine goes with flan and then I flipped it and say okay if Chardonnay goes with halibut what goes with sake and the answer was nigiri so it knew that you know you're sitting down as the sushi restaurant you ordered sake so this is sort of you know like a high-level overview of all the things that we're trying to do with text at open table and beyond text we also have photos we also have all these interactions so data science is very it's in a very exciting stage a talkative right now and we are hiding so I would like to leave you with sort of this taste of you know kind of scratching the surface with our reviews and if you have more questions please ask now or you can keep in touch over Twitter thank you