SBTB 2023: Graham McNicoll, Open Source Warehouse Native A/B Testing.

SBTB 2023: Graham McNicoll, Open Source Warehouse Native A/B Testing.

Recording: SBTB 2023: Graham McNicoll, Open Source Warehouse Native A/B Testing.

my name is Graham I'm the co-founder of growth book uh can everyone hear me all right yeah okay um I know there's not a whole lot of you here the is everyone Engineers or software or data scientists data scientists okay um everyone knows what AB testing is anyone not know what AB testing is all right so we'll go quick quickly over the the introductory part so um my goal of today's talk is to give you a quick introduction to AB testing seems like we don't need to spend much time there um then we're Focus most of the talk on why you AB test and then some of the experimentation tooling that's developed in the last couple years um and then we'll go into some best practices with AB testing and some common problems that we've seen at scale um this is a animal based presentation this is my dog Nelly um so if you don't like what I'm talking about at least you can enjoy the pretty pictures of animals um so my experience with AB testing goes back quite a long way this is me when we were born um fun fact about me I'm a twin uh and actually a surprise twin my Mom's doctor did not believe in ultrasounds um and so one of the problems when you're a surprised twin is that your parents have one of everything and including names so when we were born um we were actually called Baby a and baby b anyone want to guess which one's me B yes B is B is me congratulations there we're still waiting on the success criteria on on that test but um so we'll quickly jump into what AB testing is all about so uh the definition I really like B testing is a controlled way of measuring the impact of a change or feature on real users um and so how that works in practice is you first uh have a hypothesis some idea you want to test you then choose an audience you want to assign that test to uh you then randomly split that audience into two or more groups and assign them into the different variations hopefully persistent uh you then show them different experiences of your product or application or whatever it is you're testing you then track how they behave through your application and then use statistics to figure out if those changes are significant um so there's obviously a lot more to it but that's like really high level um so let's take a look at some AB testing examples so this this is an AB test from Airbnb and so Airbnb was trying to improve the booking rate by being more clear about the cancellation policies by giving a timeline of the cancellation window um so raise your hand if you think the new version improved the booking rates raise your hand if you think it decreased the booking rates yes so decrease so well done you guys got that one right um and I I don't work for Airbnb by the way I I never have but this is from a website called good.org which tracks uh AB tests that it detects in the wild um this next example is from Netflix this is their homepage um so on the left is the original version that has a button that says join now and then it goes to a second page that has a form where you can sign up for Netflix uh the new version is uh the version that they tested is basically the same except instead of having just a button it has a an input box where you enter your email address and then click next so who thinks this lost to the control version who thinks it won who doesn't want to vote a lot of people okay um this one actually improve performance um again I don't work for Netflix so I don't know exactly what happened but my hypothesis would be that um getting that email address ahead of time even without context is probably worth more to them in the long run than uh than a completion rate on a form with context um so why do we AB test so let's take a look at that I'm going to ask you a question in terms of um so we launch features every day we launch AB tests every day how often or what's the average success rate and to ask that another way how often is a feature that you launch successful in improving the metrics with which it was intended to do so does anyone think it's like over 70% over 50% over 40% uh you guys 40 okay 30% yeah okay so the the so industrywide success rates for experiments are about 33% so about onethird of the time the features that we launched are successful in improving metrics with which they are intended and about 2third of the time they are unsuccessful or actually hurt your metrics which is kind of wild to think about because no one one launches a feature they don't think will win right everyone's trying to improve their products and yet we're still pretty much bad at it um it actually gets even worse because the more optimized your product is the lower the success rates get so these are some uh success rates from the industry Microsoft being booking Google Etc um and you really can't get the same degree of information just by looking at so squinting at the data right you if you just look before and after testing this is an example from Airbnb where they rolled out a feature uh tracked the metrics over time and then ended up rolling it back because it lost the experiment but if you just looked at the aggregate performance metrics you just can't see uh the effect of that change because there's so much variance and so much other um stuff happening day-to-day so the takeaway here is that without testing you're guessing um and a lot of companies know this so they a lot of companies that have figured out like a high degree of sophistication with um with their experimentation programs so if we uh on the crawl walk run fly excuse me EXC um model there's usually companies start with just adding some basic analytics they'll do event tracking you then start getting kind of into the walking mode where um you're kind of running some manual AB tests maybe you're optimizing one part of your product you're running kind of five tests a month you then start getting into the Run area where your experimentation is quite common um you may even have a growth team or some data teams making sure you do experimentation well um you're running anywhere from like 5 to 100 tests and then you get to the fly stage where you have uous experimentation where you get companies running tens of thousands of experiments every month uh everyone at that scale here where do where do you work do you mind Amazon okay that makes sense yeah you have the traffic to do that um and so you might imagine that like the tools that you build to be able to run 10,000 tests are quite different than an off-the-shelf tool um and so what you tend to get is this sort of curve like this where this really sophisticated experimentation tooling requires a large company to build it um and then you get all these companies up here on this end and all of them self-build and so into this space um I found myself also building my last experimentation platform and wondering like every single platform here is pretty much the same thing um with slight variations on the data and statistics but really very similar um so why can't we just build a tool that unlocks this sort of Ubi ubiquitous experimentation for all companies and doesn't require a team of I don't know tens to 20s of Engineers full-time working on it um and so that's really what we decideed to build with growth book so growth book is an open source uh AB testing and feature flagging platform and the idea is to really be an in-house platform you don't have to build and really unlock that like uus experimentation um so how that works in practice uh is that most people here already have data going somewhere right we probably have data in a data warehouse um and so you're actually really close to getting that experimentation platform setup already uh if you're using any of these event trackers getting your data into an event Warehouse is super simple um but even if you're using something custom um that's totally fine and so you you already have this one box of our experimentation life cycle already built and so uh growth book is a Docker container that you can just um run and it connects to your data warehouse and then gives you a full Suite of experimentation analytics on top of that and we support really any data warehouse that you might have data in we support connecting to it it and then you get experimentation dashboards like this um out of the box completely for free in open source um and it comes with like full dimensional analysis and some really Advanced features there that they're pretty hard to build yourself um we also give full transparency so behind the scenes everything is SQL powered and so we'll actually expose all the SQL for everything that we build um and so if you ever see something weird or some data that looks a little off you can just go and debug it and figure out what's happening um so now you have the reporting and tracking side but then what about the hypothesis and variation assignments um you're with growth book you're welcome to do that yourself we'd have no um allegiance to how you build that but we did build a feature flagging Suite as well which allows you to kind of quickly deploy features and do the variation assignment as well so in that you actually get like a full functioning Suite um really running on top of your own data for very low cost like pretty much no cost um depending on the size of your data if you're obviously if you're Amazon scale you're going to be paying for that data um but then they don't really care about that because they own all the database servers too so um and then you get like a platform like this this is what gbook looks like or at least it did a couple months ago um with the feature flagging platform and then you can toggle things you can also do like experimentation assignment um through a nice gooey um yeah so uh I could give you a quick demo of the growth of platform where we can go into more of the best practices anyone have an opinion demo okay should we try a live demo this always goes well right um all right so here's growth book you can all see that bit bigger so I have growth book set up um but it's it's not really connected to a data set it's just has my sample data set here um but I'm going to add a data source and for this say for this example I'm going to say I'm going to use Google analytics um so then I just have to connect let see if this works um so connect to my data source tell about the project idea I'm working with and here we go so now we're connected to my big query database and I know that the the data warehouse is or the data is coming from uh Google analytics so I can click on The View schema browser and I can see all my events coming in here um but what's kind of neat about growth book is we added um autodiscovery of metrics so we can just go in here and like um create metrics right like uh how many people yeah we'll just I don't know turn off some of these lesser important ones but um we can just create all those metrics and now we have 16 metrics that are tied to this data source um with literally what was that 30 seconds um and so if I go here on the metrics yeah so here are all my metrics so I can click on like let's say page views I can do a quick analysis just to see if it's working and now when I run an experiment I can use that page view metric um I can just create new experiment and add that metric to any of my experiments and I can even add it retroactively too which is kind of neat um and then you get an experiment report that looks a lot like this this is a sample data report um that kind of shows you the impact of this change yeah let me jump back to the presentation okay um so the takeaway here is that in 2023 uh there's really very few cases where you should build your own platform um I would really recommend not doing that there's there's a lot of risk in building your own um in that uh one bug in your statistics and you invalidate all your past uh decisions that you've made so the risks are very high and um the upsides are fairly low in terms of you know using a tool like ours all right so let's jump into some AB testing best practices um so one of the ones that we always like to tell companies they really need to build trust in your experimentation results and to help illustrate this I'm going tell you a little example about OS SEO anyone heard of him no uh yeah not really a household name but OS was a Hungarian physician he was born in sort of the height of the austrial Hungarian Empire and by the 1840s he found himself basically the uh equivalent of a chief resident of the Vienna General Hospital um now medicine around this time was pretty weird pretty primitive by today's standards and that people had no idea how you got sick um they thought maybe you got sick through like bad humors or being immoral or uh you know bad air and they really had no idea but around this time they started applying science to Medicine which was sort of pretty new and um part of that included doing a lot of autopsies and looking at data and Os was no different and when he was the chief resident of this hospital he found that um he had two maternity Awards one of the Ws was staffed by doctors and one of the wards was staffed by midwives and he found a marked difference in the mortality rates of the women giving birth there anyone want to guess which one had a higher mortality rate okay well the one run by doctors had about a four to five times higher mortality rate than the maternity W run by midwives um and so he was quite perplexed by this he was like why would they be so different so so he started looking at maybe the the doctor's Wars only admitting folks who are you know more sick than than the ones that go to the the Midwife Ward well that turned out not to be true because they admitted patients on Alternate days so the sample size is pretty much the same just depending on when you went into labor um the death rate was so high in one of the Ws that they were constantly um priests walking around ringing bells and he thought maybe that scared the women into just dying and so he um uh he banned bells and he had the the priest wear slippers and that had no effect um it wasn't until a friend of his was doing an autopsy and um cut his hand and died of an infection very similar to what the his women in his maternity W were dying that he had an idea that perhaps we should be washing our hands before going to see our patients after doing autopsies um pretty wild thought at the time right so uh he had his doctors wash their hands with chlorine and when he did that he found that the mortality rates fell by 10x right this is this is great news everyone remembers him because he introduced handwashing into the world uh Well turns out not the case um despite overwhelming evidence the doctors just kind of refused to do his idea so while he was chief resident he was able to enforce it but shortly afterwards the doctors forced him out and went back to not washing their hands and the mortality rates went right back up um and it would be another 20 years before his ideas were accepted as part of like the germ theory of medicine uh rather unfortunately for SEO wise it doesn't really end very well because he um started writing letters to all the hospitals to get everyone try to wash their hands and everyone thought he was crazy um to the point where they tried to have him committed to a mental Asylum and in fighting with the guards he ended up getting cut on his hand and rather ironically dying of an infection um so poor semi wise there but um the reason why I tell you this is because there's a con um Concept in AB testing called the sem wise reflex which is like a reflective rejection of new information so quite often when you tell people about the success rates to B testing or um counterintuitive results they're going to want to like um object against the data and uh you really need to guard against that so the best way to do that is to really build trust in your experimentation platform and the best way to do that is by being really transparent and like running a lot of AA tests if you're not familiar it's basically an AB test where there's no no difference and you want to make sure that the uh the sides are the same you should run data quality checks like sample ratio mismatches um make sure that people aren't being exposed to both sides of your experiment and uh being able to be audit the being able to audit the results is really important and then training your team on some basic concepts of experimentation and statistics um this one's a little bit thankless you ever had to explain P values to folks but um there are some other ways around that as well we'll get into some of the common problems in a little bit um another one that we see people get wrong a lot is experimentation frequency um so let's say you you have a winning test and you deployed that to production uh what are the odds that it's the optimal design first try so if you were to plot all possible versions of that product against the performance of those products what are the odds that that's the highest performing version of itself it's pretty low right probably close to zero and in fact you you have no idea right but um you only have one data point so uh maybe test a different version and it performs slightly better um but now you have two data points and you really have no idea what this shape of this graph is going to look like uh you know it could be like this you could be near a localized Maxima or not um could be like this like you have really no idea and so it takes a lot of experimentation to kind of um optimize the product and figure out where your optimal values for the various things things are um and then you should really adopt experimentation as part of your product development process um I often talk to folks about how they make product decisions um sometimes people listen to whoever's like the loudest person gets to win sometimes um we got the hippos which is the highest paid person's opinion um but you know we're not really great at predicting what users want I mean we're still even with this we're still only batting you know onethird of the time success rate or less um and so how this is how you get products like clarol touch of yogurt shampoo um which is was a bit failure people were eating their shampoo um Walmart just or released I think two years ago they released juneth ice cream and then trademarked the term juneth uh which is pretty toned deff then you know more recently we have certain people destroying a beloved brand and I don't know that jury is still out if that one works um but there's so much Pro so much f focus on like the development process that people often forget about kind of the sides of it right like the ideation side and and what is what does done mean um and so you might be using like an agile process or whatever this is this is lean agile scaling I don't Enterprise uh is just scary I hope this is not yours um and this is the agile landscape these are all the different possible project management process but all of them focus on that middle box and really none of them Focus on the on done um so what does done mean is it just when we ship it is it like that it's defect free is it like accepted user stories um maybe the product owner likes it like none of this really matters because the only thing that really matters is like did we improve our business goals when we shipped this thing um and so each product when you define it should really or each product when you come up with the ideation for it should Define like what does success look like for this um and so like what you know what are you trying to do what are the actions that a user would do to demonstrate that hypothesis what are the metrics that you would need to show that would prove that uh that action and then what is the smallest thing we can build to see if we're correct or incorrect with that assumption the last part is actually pretty important because you don't want to overinvestment that's wrong or the hypothesis that's wrong um like to call this ham I don't know I haven't had much success with that but we'll see if that if that catches on um yeah and then only only really ship those features that positively impact your metrics or or your users all right let's finally let's take a look at some common mistakes um so number one is peing this is where you prematurely make a decision based on incomplete data typically this is most common with frequentist Statistics if you're in that space um but there are some ways to guard against this but the biggest one is to tell people when they're peing and when they're not and keep them honest about that um Tan's law if you're not familiar with Tan's law basically it's um well by example let me say that you you just launched a new feature and you doubled Revenue should you celebrate ask for a raise um you might want to wait on that because large improvements from a single experiment are actually extremely rare um and you really should do some further investigating to see if that impact was actually a bug or not um funny story this happened to me at my last job it turned out we were just double billing all of our customers um which is a great hack to make more money but it wasn't wasn't great for long term so time's laws any figure or or figure that looks interesting or different is usually wrong um good Hearts law so when a a measure becomes a goal it ceases to be a good measure um so by way of example here in around the turn of the century in Vietnam in colonial France Vietnam they had a problem with the plague and they decided to try to uh combat that by incentivizing people to kill rats which are primary Vector for for plague fleas I guess um so they decided to uh incentivize killing of rats by having a bounty on rats and they determined if you paid the or killed the rat by handing in a tail of the Rat and um by all accounts this was like a great success in some cases I think they had thousands of rat taals being turned in day um until they started noticing there an awful lot of rats running around Hanoi without tails and it turns out that the Vietnamese were just cutting the tails off without than letting them go and some cases like feeding them to try to create more rats so that they could get more money from the Tails um so good heart's law is when a measure becomes a goal it ceases to be a good measure uh confirmation bias so the goal of AB testing is not to prove that you're right or some genius the goal is to make the right decision for your product um so you're not just you know looking for the metrics that prove exactly what you want to see um similarly there's the multiple testing problem so basically the more metrics you add to an experiment there's a for every metric there's a non-zero chance it might just be randomly better and so if you add you know keep adding metrics until you find one that uh you see goes up and then declare it a winner like that's metric shopping and you shouldn't do that how am I doing on time okay that says five minutes cool um all right so in conclusion um if you're not testing you really should be uh you're basically guessing at this point um AB testing is more accessible than ever there's some really great tools out there that allow you to run it with your existing data for literally zero cost or essentially free um and then you should apply best practices so you should be testing a lot you should be be building trust in your experimentation platform and then make experimentation part of your process and team culture um avoid some common mistakes that I just mentioned and then uh hopefully with all of this you can make better product driven decisions or better decisions and and increase transparency and Trust in decisions that you make um and above all like don't forget poor SEMO wise uh be open to new ideas that challenge your assumptions um finally the if you're interested in AB testing and you haven't already read Ronney kav's book I highly recommend it um he was at uh Microsoft and Airbnb and I think Amazon as well um built a lot of their platforms uh uh growth book we have our own testing book that's open source and based on particularly our platform um yeah thanks