Text By the Bay 2015: Dave Holtz, Increasing Honesty in Airbnb Reviews
Recording: Text By the Bay 2015: Dave Holtz, Increasing Honesty in Airbnb Reviews
so my name is dave holtz i'm a data scientist at airbnb and today i want to talk to you guys about some work that we've been doing to try and increase the honesty in the reviews on our platform this is work that i've been doing with a couple of my colleagues at airbnb so eleanor greywall and matthew pearson and also a researcher at the national bureau for economic research andre fratkin so the core concept that i sort of want to set up here at the beginning is the notion that trust and reputation are the backbone of airbnb or more generally any peer-to-peer marketplace that you might have or even a non-peer-to-peer marketplace i suppose uh so if you aren't familiar with airbnb although i would guess that most of you people are because you live in san francisco and you all seem cool and hip airbnb is a peer-to-peer marketplace where people can rent out extra space in their homes uh to travelers who want maybe a more authentic non-hotel travel experience when they're traveling and so airbnb is in 190 countries uh in you know basically any city you can think of and there's all sorts of different accommodations on our site so everything from traditional apartments to tree houses or igloos or stadiums occasionally and so the way that the experience works as a guest is that you would come to airbnb.com you'd see this homepage you'd say where do i want to go i want to go to san francisco and you'd hit the search button and you would see this page um the search results page uh and so there's a lot of information on here we have you know pins on the map that sort of indicate where different listings are you will see pictures of the listings information on you know how much they cost per night but the thing i sort of want to zoom in on during this talk is we give people information about how previous guests might have reviewed a listing so you see right here on the search results page we'll say hey you know this private garden artist space has 118 reviews and if you were to say hey that that sounds really appealing i think i'd like to stay in that private garden art space and clicked on it you'd be taken to the listing page itself which has a bunch of additional review information so we show you know an overall rating which in this case is five stars and then also uh subcategory ratings and a number of particular categories so those are accuracy communication location etc you'd also see reviews that guests have left for this particular listing you know we had a wonderful time staying with keppa what a really great place we had a lot of fun and also if that host had traveled on airbnb as a guest you'd be able to click onto their profile page and see reviews that had been left of them as guests when they had traveled on the platform so someone might say you know dave's a really accommodating guy like he talks a lot but we really enjoyed having him in our home uh so uh the thing that's sort of novel about airbnb is we're trying to take this this concept which normally might take many many years which is you make a friend you grow closer and you sort of want to say hey i'm coming to your city is it okay if i crash on your couch or if i stay in your guest room and that usually takes you know years and we want to truncate this so that within hours you feel comfortable letting a stranger into your home or or staying in the home of a stranger in the case of a guest and we want this to happen over you know the course of just a couple of hours uh so how can we sort of achieve that goal uh the way that we we sort of like to i guess position this is that we want to build up a concept of reputation and so you can almost think of reputation as a sort of trust at scale right so if we have uh sarah and bob and sarah and bob transact with each other repeatedly over time if those transactions go well they'll build up trust you can imagine then that there's some third person and if sarah and bob have transacted a lot and bob and you know dave have transacted a lot over time sarah might come to trust dave as well because there's this sort of like you know uh transitive property um and if you sort of take a bunch of these transitive like trust bridges and just aggregate them up eventually you're going to arrive at sort of the concept of reputation which is that you have this persistent level of trustability that is you know somewhat public and that everybody knows and so when you have a a healthy reputation system there are sort of a bunch of you know really great benefits that fall out of it so a healthy reputation system is going to reduce the need for brand you know uh something like yelp makes it makes everyone way more comfortable going to a small business because you don't have to rely on the fact that i've gone to a chipotle somewhere else and i know i like it a lot i can go somewhere new because i can read and learn about the reputation a healthy reputation system will also help new users to intuitively determine the quality of the inventory so if you have a reputation system that isn't working well and say you know all the scores are really homogeneous or there's not that much information a new person is going to show up and they're not really going to know how to interpret those scores or that information and finally you know when your reputation system is functioning well it's going to help people like developers and data scientists that are going to develop features that enhance your platform so in the case of airbnb we want guests to be able to find the right place for them very quickly and also we want to be able to quickly and easily identify hosts who might be struggling or might be in a slump or might be doing something a little bit wrong and sort of reach out to them and give them the tools to succeed in the future a lot of other people a lot of pundits and such sort of agree with this idea so thomas friedman has said that you know airbnb's real innovation is this platform of trust where everyone you know can rate each other and trust each other a lot and wired has written articles about how airbnb and other companies like lyft or uber have finally gotten americans to trust each other but honesty is really important in these reputation systems because they do sort of adhere to this principle of garbage and garbage out right so all these mechanics and all these benefits that i've described thus far really rely on accurate measures of the quality in these individual reviews otherwise this isn't going to work well and the the sort of caveat is that the ratings in most reputation systems are overly positive and they suffer also from some amount of non-response bias there are a few different sources of bias that i sort of just want to roll through really quickly so the first source of bias that we sort of considered when we started thinking about this is the possibility for you know fear of retaliation so uh you know if i had a bad experience and i'm worried that when i write a review the person i review is going to come back at me with a negative review just to retaliate i might sort of omit that negative feedback because i don't want to get into this you know sort of tit-for-tat reviewing pattern there also might be evidence for induced reciprocity so a good example of this is you know if you drive around somewhere and you see a bunch of high school kids throwing a free car wash or something it's nominally free but there's this idea that you will take advantage of the car wash and have the service rendered for you and then you're going to reciprocate and you'll donate some money and so someone might you know leave a positive review in the hopes that the person that they've reviewed will feel like they have to reciprocate that positive feedback and in turn return positive feedback and finally the third thing that we are worried about is just that there might be some amount of discomfort around leaving a review that someone else can read so just you know socially people are not comfortable reviewing someone that they have met because you start thinking about the fact that hey if i leave a negative review of this person what if then it impacts their business and then in the future they have a harder time getting bookings and then they might not be able to pay their bills so there's just this sort of social discomfort as the social distance between two parties decreases uh so we identified this as an issue that we sort of wanted to tackle at airbnb and so our data science team started to think about ways to fix it uh so here's what we've done so far so uh before we go into this i think it's useful to sort of understand how our review flow worked prior to about july 2014 so say that a booking has occurred and it's ended once that happens a 30-day review window will begin at which point either the host or guest could submit a review of each other let's say in this you know example flow the host submits the review first they would submit the review and then it would immediately be public at which point the guest may or may not read the host review sort of ponder the contents decide how they feel about it and then the guests themselves could also submit a review of the host at this point all the information is on the table and both the host and the guest can see the review that was written of them and they can respond to it if they like publicly we wanted to sort of tackle these first two biases that i had outlined and so we designed an experiment that we refer to as the simultaneous reveal experiment which basically says rather than having this mechanism where a guest or host can read the review that was written of them and then you know think like retaliate or you know sort of ingest that information and respond accordingly what if we didn't let you see the review that had been written of you until you yourself had written a review so to sort of go through this in timeline form so it might be simpler let's say a booking ends then a 14 day review window is going to open at this point uh when the host submits the review first that review is hidden and neither the guest nor the general airbnb population can see it but the guests would receive a notification that says hey there's a review waiting for you and you can see it if you yourself write a review at some point in the future the guest might submit the review and then both the guest and the host can see the reviews at this point as was the case prior to the experiment they can both respond to the review publicly you might notice that the review window here is shorter we shorten this because in the case that only one party writes a review here that review is sort of going to be locked in a box until the review period expired and we didn't want reviews to be sort of sitting there invisible to anyone for 30 days so we wanted to try and shorten that window uh this had a pretty significant effect on review rates uh so we saw that in the treatment the rate at which host reviewed guests went up by about six percent from 73 to 79 percent uh we saw a smaller but still significant effect uh uh in the rate at which guests reviewed hosts so we saw this go up by about three percent uh when we looked at the sort of content of the review scores we also saw a little bit of a change so uh the rate at which five star reviews happened went down by about one point four percent and a lot of that weight was absorbed sort of in our four star reviews so like one upshot here is that in general the experiences that people are having on airbnb are like quite positive but we want them to be you know honest so that we do have better granularity between a four-star experience and a five-star experience uh we so we analyzed the scores but we also wanted to sort of dive into the text which is important since we're at a text conference right now uh so our first sort of very naive very crude way to do this is we sort of just looked for for engrams that occurred far more frequently in say a negative review than in a positive review or vice versa to sort of try to get these you know very uh a very rough measure of how much negativity there was in a particular review and so uh this is sort of the distribution of negative phrases and the guest reviews of listings and you see that like as a first order approximation this is working pretty well so sort of up and to the left are things that occur way more in negative reviews than positive reviews and we're seeing things like was dirty or filthy or rude and if you do the same things for host reviewing guests you start seeing things like rude and messy and smoke and you know people sort of talking about time probably because of the the time the guest arrived or something like that and so we sort of used this information in synthesis with the you know data around the scores and the review rates and we started building out these uh regressions to sort of see if the tr what effect the treatment was having on our review platform uh i don't want to like get too much into the weeds of this regression table but there are sort of like big bullet points that i can point out right um so the fact that this host negative sentiment coefficient which is uh significant is sort of canceled out by uh the treatment uh crossed with the the fact that there is you know some negative sentiment sort of gives you the information that when guests see a negative text in a review of themselves they respond in turn with a negative review so this was an effect that used to exist and we sort of were able to neutralize that effect by making these uh these reviews simultaneous reveal um also the fact that there is this sort of uh uptick in sort of you know the guest not recommending the host and the treatment does sort of provide evidence that there is this you know effect of guests being induced to leave positive reviews when they are left a positive review by their host uh we also wanted to think about sort of the hosts uh like strategic considerations when they're reviewing like are they fearful of retaliation by the guests etc and so this coefficient here is giving us pretty strong evidence that hosts are in fact strategic and that they were omitting non-positive feedback from text before we rolled out simultaneous reveal because they didn't want to sort of take the risk of the guest responding in some way uh we wanted to run a different experiment to try and tackle this this third bias which was sort of you know discomfort and so what we did here was uh we launched a separate experiment which we call the incentivized review experiment and so basically if someone had gotten some number of days very deep into our review period and they had not yet left a review we would email them and say hey dave you know we noticed that it's been 20 days and you haven't left a review yet so if you leave a review of this listing that you stayed in we'll give you a 25 certificate for your next trip so the flow here just to go over it really quickly is that a booking would end and this 30-day review window would open the host would submit the review the reviews public the guests could read the review but if the guest hasn't left a review yet after after some amount of time we'd send them this reminder email at which point they might subsequently leave a review and then everyone can you know view the reviews and they can respond we saw a really significant effect here so note that these percentages for the review rates are much lower than uh earlier when i was showing you these like 70 numbers uh that's just sort of a byproduct of the fact that these are the review rates conditional on you having gotten you know 80 percent of the way into the review window and having not said anything yet so the baseline rate is pretty low here uh but when we do offer this incentivize review treatment the rate of review goes up from 23 to 39 and that's like a very significant result and if you look at the distribution of the reviews that we see from people who get this far into the review window and have not yet left a review like the rate at which people are leaving these five star reviews goes down by you know about seven percent uh and a lot of that weight again is absorbed by these four and three star reviews and so again the upshot is that you know most experiences that people are having are positive but you know sort of incentivizing them is bringing out people who are on average have you know slightly worse experiences we saw a bunch of other sort of high level effects uh which are that you know we saw an increase in sort of the negative uh like engrams that we had identified conditional on a particular rating we also saw that guests were leaving more private suggestions for improvement to hosts so they were saying you know hey like if you sort of close the window or got new blinds this place might be better and also people were leaving reviews faster we sort of suspected that this was due to these reviews being locked in a box right you want to find out what someone said about you so you're going to leave the review a lot more quickly so after we did this first pass we sort of said okay this very crude way of sort of measuring sentiment by just counting up engrams that occur way more often is is good but but can we do better and can we learn a little bit more using you know more sophisticated language processing methods and so this is sort of a new phase of this work that's still ongoing and so how could we maybe use nlp to improve on our results uh we sort of had a little bit of a of a sort of pickle here because uh we wanted to develop a classifier that determined whether or not a review was uh negative or positive but we didn't really know what we wanted to use as our labels uh you naively might think that we could use our star ratings but we sort of wanted to be able to test for variations in the sentiment even you know conditional on that rating so you don't just want to classify whether or not something's a five-star review we also thought about the fact you know hey maybe we can use existing uh you know sentiment uh corpora but that's really tough because you know we think that the language in airbnb reviews is pretty specialized people are talking about mattresses or or you know check-in time so we don't want to just use like the movie reviews corpus or something so what we ended up doing is training a logistic regression classifier where we basically used a dictionary of engrams that appeared in the reviews as our features and so we did all the normal things where we stripped out stop words we looked for engrams that occurred you know above some minimum threshold number of times uh and then in order to try to sort of achieve this what is really negative versus positive dynamic we sort of train this only on five star reviews and one in two star reviews for the the pre-treatment period so we wanted to just get the polls and hopefully have a pretty good idea of what was actually positive and negative uh and then we the idea here is to sort of evaluate this model on all of our experiment data and see if we can observe changes in the sentiment and reviews over time and also changes due to our experiment so you know we basically this is looking pretty well we have you know 85 percent accuracy in this model using some cross-fold validation and we see some verification that there are meaningful engrams that um wouldn't really appear in other sentiment corpora so a good example that i think is kind of funny is that the word third comes up as having like a very strong negative weight and at first we were sort of puzzled about why that's true but it looks like this happens because people have these third floor walk-ups and people are complaining about carrying their suitcases up three floors and also we have listings sometimes where it's a three-person capacity listing but that third person is staying on a couch or something and people are upset about that um the performance of our model uh doesn't seem to really depend very highly on all these sort of meta parameters that we're using like regularization weights and and sort of minimum occurrences of the engrams uh and we sort of see neat effects where we can actually see a discernible difference between our control and our treatment uh in sort of the average score that this classifier is returning uh over time so you know in um and these labels are inverted so sorry about that so in five star reviews like the average score is down about two percent in four star reviews it's down about three and a half percent uh so things are looking directionally as we would expect them to be which is that people are a little bit more honest in the treatment when they are not fearful of you know retaliation or they're not trying to you know induce the reciprocity like reciprocate the positive review that's been written of them uh so the next steps here that we're still sort of working on are plugging this new output into these old regression tables that i showed you guys and sort of seeing if that improves the results that we have so yeah the high level learnings here where that there is sort of a bias in the observed distribution of reviews that we see on airbnb hosts and guests do act strategically there are sort of intrinsic reasons that people have for reviewing or not reviewing a transactional partner on a marketplace and these reasons are important to them uh but the the sort of silver lining is that by making changes to your review process and sort of incentivizing people to review or we're sort of making sure that things are revealed simultaneously you can actually reduce the magnitude of these biases so we're continuing the work on improving you know sort of our analysis of the text and sentiment and really trying to quantify very well the magnitude of these biases so that we can get at the true underlying distribution of experiences and we're also just always brainstorming on new ways to make the product even more conducive to honesty in the reviews so i think i'm a little bit overtime so i do not have time for many questions in this public forum but i would encourage you to come you know grab me after this if you have if you'd like to talk about this and as a quick postscript we're always hiring so you know if that's interesting to you let me know as well sweet thanks have a great couple days everybody you