Scale By The Bay 2020: Chris Fregly, Smokey and the Multi-Armed Bandit featuring BERT Reynolds
[Music] yes hey there everyone thanks for sticking around this is the last talk and uh hope you have been enjoying the conference so far so i've got this fun talk um kind of uh i was putting it together and actually smokey and the bandit was on in the background and i hadn't seen that that movie in many years and um so and we are also or sorry i am also featuring bert in this talk um if you were at the previous talk in this track it was about nlp natural language processing uh spoke about um bert and the transformer based so we won't be training burp models here the assumption is that we already do have a trained burp model but we will be training a reinforcement learning model as part of the multi-armed bandit and there's a picture of burt reynolds there the proper spelling of burt by the way is b-u-r-t and okay let's keep going so this is the high level abstract first um we will actually uh so so we have already done this actually the first part we've already done but the the goal is to actually compare two burp models one of them is pytorch one of them is tensorflow we will push them into production i've already done that just for the sake of time and will be training a reinforcement learning model to figure out which model is the better model and then dynamically shift traffic to the better model and in doing that so secondly i'll be describing the differences between a b tests and the multi-arm bandit tests they are similar but very different and we'll be talking about different trade-offs with exploration exploitation we'll be talking about how to maximize rewards how to minimize regret which is kind of leading into some of the reinforcement learning topics um then we will dive deep into how to actually deploy this and of course i work for aws so i'll be showing how to do this with s3 with kinesis for the real-time streaming data coming in we'll be using something called the amazon customer reviews data set this is a specific data set it's a public data set that's been that is a curation of the like amazon customer reviews from 1995 until 2015. so these are product reviews that have been left by customers and we will the the two models that will be like comparing in production have been trained on that specific data set so i'll get more into how we're going to use all this here in a bit and yeah so one thing to point out there there is a book coming out this is actually in the like early release form right now it's also out on amazon for pre-order if you want quick plug there and i'm based in san francisco i have this san francisco-based meetup called the advanced kubeflow meetup we have sort of rebranded over the years we started off we were the spark meet up the advanced spark meetup and then we became the advanced tensorflow meetup and then eventually um we decided to rebrand to kubeflow because this does include a lot of tensorflow um and pi torch and and sort of the whole ecosystem uh formerly netflix um i was part of the original streaming team uh back in 2011 2012. um i joined databricks around 2014 and have quite a bit of spark and machine learning experience and now i'm with aws um this is actually a sort of high level outline of what the book is going to cover we will be focused on section 09 here which is deploy so we'll be doing a b tests looks like it's not even on here but the multi-arm banded stuff which was recently added about uh one month ago so you're getting super fresh material all right so let's actually compare and talk about these i think this is pretty much uh trained to burn language models we already did that um we'll be using vocal wabbits which is a reinforcement learning framework that's built into sagemaker um and you specify the version of vocal rabbit that you want to use uh and that's what we'll use to train reinforcement learning sagemaker also supports um coach and ray as reinforcement learning but we find uh for like this particular use case that the vocal rabbit stuff works really well and it's super fun to say um at the end we will show a like live demo um and we'll we'll show some of the traffic shifting happening and how that all works on the amazon web service platform all right so traditional a b tests uh so here's the use case we've got um a little box for someone to type in a review and so this could be in the wild right this could actually be on someone's website uh a partner of like amazon this is all made up by the way this so this is not what's really happening i should be very clear this is uh based on a blog post written by one of my peers i took that i had adapted it to the burt use case and then deployed it on to aws so what's happening here is we want to actually let the user select a star rating but then also make a prediction and then compare those two and what we'll do is we will use the source of truth as what the user is selecting so we are kind of fabricating this use case a little bit we are assuming that the like ground truth prediction is what the user selects and then we are going to compare that ground truth to uh what our models predict and in a traditional a b test it's static we have to split the traffic um you know either 50 50 uh or you know five percent ninety five percent we have to use something like a customer id and figure out which of the variants um that user should go to the problem with this is if a particular model is doing horrible so there's sort of the incumbent model we'll say model one and then model two is the uh like challenger model now that challenger model if that model is really really bad um we will still be sending traffic and you know if we're doing 50 50 we're still sending 50 traffic to that bad model and it's performing horribly and you know maybe revenues dropping there's really no way to stop these tests the like traditional a b test they're static they're fixed up front um could start to like impact the business um we like pretty much have to run these experiments to completion and this could be two week experiment four week experiment um and all the while we are actually losing money so we can flip over to these multi-arm bandits and the sort of easiest way to like compare these is we can do dynamic shifting so if that model two is really bad we could actually stop sending traffic to it the uh like bandit will actually learn uh that like model two is not doing well and then and send more traffic back to one or or like the other way around so in this diagram here we see model 2 is actually doing really well and the bandit learns then to send more like traffic to the second variance we could actually finish the experiment early we could run longer we can define uh like how aggressive we want to explore new options so um and like we could also add new models that like first bullet point there if we decide that maybe we had learned something from model 2 and now we have a hunch about model 3 we could actually add model 3 and the bandit will start to then send some traffic to like model 3 based on the strategy that we choose and like the aggressiveness so the whole goal here is to minimize the regrets and that's how much um we like would be losing and then to maximize the reward so these so these are all reinforcement like learning based which speaks very much in terms of maximizing the reward um so we've already trained these two models but uh this kind of gives you um so if this was model one and two actually not not a and b um but one and two the first one we trained with tensorflow um we used burp on tensorflow we use a library called transformers by a really cool company called hugging face um and then model b or two is actually trained with pie torch and we are also using transformers um and there's you know tons of craziness around bert right now these like these like last couple years have been very very popular uh for bert we actually base our whole book on natural language processing and bert so that's where a lot of these diagrams come from is actually the book um all kinds of variants my favorite is probably chem embertz which is a french variant uh there's also uh bercha which is dutch but there's also uh these like domain specific variants as well too patent bert and clinical birds and then bio birds and then scientific bert cyber so like more and more variants these variants have been trained from scratch um on the sort of custom data sets so for example linkedin has like their own variant that they have trained um and you sort of internally for like job listings and like resumes and job titles and all that um and what we are actually doing is starting with the original bert and fine-tuning it to this customer reviews data set and specifically each review has text and then a star rating one through five that has been assigned to it by the user so again we're we are assuming that the user is the ground truth and that's just a decision that's what we're doing so there's a lot of cool stuff going on with bert like just know that that we that uh once like my ear pods are doing some weird stuff here okay um we we are using a pre-trained burp model and nothing fancy there i think we're actually using distilberts which is a much lighter weight smaller version of burt but still retains about 97 of the accuracy okay so let's shift over to the reinforcement learning so again this is supported by sagemaker all you do uh and i'll show you the code here in a few minutes um is give it uh some like parameters tell it the type of strategy that you want to use give it the hyper parameters and then vocal lab it takes over and so this is actually built into sagemaker meaning there is this open source like docker container that has been pre-tuned and um and then pre-installed has like lots of like s3 optimizations and there are lots of sagemaker optimizations going on but this is open source vocal rabbits similar to how we use um straight up like open source tensorflow and then open source pi torch um which we had used earlier uh like before the session to train the two models um and so there's different strategies here i think for our demo we actually use the like online cover probably the most popular reinforcement learning strategies for multi-arm bandits would be the epsilon greedy or thompson sapling so we found for for this particular exercise that the online cover works pretty well all right this is the sort of master uh very complicated diagram here um really let's break it down and i have it numbered so you know one dot yes each of of uh like these arrows here kind of shows the flow going through the system so on the left is each reviewer and they are typing in some text and they're selecting a star rating they're clicking submit where we are assuming that that is the ground truth that then flows through the system now before we make a prediction so you see model one dot dot n those are the two uh so that's the like tensorflow trained bert model and the pi torch train burp model would be model two but the application doesn't actually um know which variant that the bandit model is going to serve up so the first call is to ask the the third model here which is the one that that we are actually training which is the multi-armed bandit uh model with vocal rabbit so when we first deploy the multi-arm bandit the like bandit doesn't really know right like what's the better model um but slowly it will start to realize that like one of these models is better or maybe it's not and so actually the worst case scenario for multi-armed bandits is that both variants are 50 50 and there's really no clear winner and then that's a case where you would probably tear down and you know keep the incumbent and not actually replace it with the challenger uh and then go back to the drawing board start to train again and find a like a like potentially better model so step one um and when we actually ask the bandit for uh which variant the the model 1 or model 2 burp model we actually save the information and then also save what the multi-arm bandit chose so we would save the actual review body text we would save the model id and so the reason that we save the actual review body text is because this is a contextual bandit so we are actually giving it context giving it a little bit more information to uh make that decision and so potentially certain reviews maybe model two is better than one uh and so that that will will all be learned and then like continuously trained um if you see in the bottom here just kind of jumping ahead to step six all that data gets fed in um and a new model is then continuously deployed and this is deployed using sagemaker um and the there is no downtime uh these are you know uh like classic deployment strategies where you can do canary releases you do um blue green deployments that kind of thing so that's all supported through sagemaker with just a couple lines of code so let's go back so we've got line two uh or sorry like arrow two that's saving all the data we've got uh the arrow three so three now the like application knows which model to call based on the bandit so so now the application calls predict um on the specific variant and then gets back the prediction and then there's step four which is now the application uh can like um see what the the burp model predicted and then what the user selected and if those match we then reward the bandit for choosing a good variance if they don't match then we just pass zero which is no reward and so all of this data then gets uh step five gets saved in the reward store which is s3 there's actually we are using the amazon kinesis data fire hose which is fully managed streaming all you do is call one api with your data the uh like kinesis stream will then dynamically scale as like more traffic comes in so that you don't have to do any of that and then periodically let's say every 10 minutes every 15 minutes we then will combine the event data that came from step two with the reward data from step five make small little adjustments and then redeploy a new multi-arm bandit and then the application then uh like continues on so that whole area on the bottom is sort of batch or maybe you know um like it's it's it could be every 15 minutes every 30 minutes every hour but all the stuff on top of that like dotted line is all the real time stuff going on so those are the keys there so let's do a demo here so i'm going to show you what we're actually leading to so model one is the uh like tensorflow model like i said the model two is the actual pi torch model so we see a bit of like exploration happening so i said that the multi-room bandit doesn't know up front what is the best model so it's just gonna send a bunch of traffic to one it's gonna send a bunch of traffic to two and then it's gonna start to learn and it looks like this case could be like one of those worst case scenarios where it's 50 50. and i do want to point out that we train the model on the same data set um and the same like amount of time and it's actually pretty much uh the same model so that was somewhat purposeful and sometimes this could be called what would be uh like an aaa test right like for example where we are actually comparing two models that we think should be the same if they're not then maybe there's a bug with our you know pi torch implementation or bug in uh something with tensorflow um we have seen times when tensorflow uh will like outperform pie torch certain versions of tensorflow sometimes performance bugs get in there and you know certain things happen with these weights that are learned uh but i think here we're using tensorflow 2 1 and pi torch 1 5 i believe and it seems here that that things are 50 50. so um and you can kind of tell by the upper right there there seems to be you know towards the end of this experiment there's actually pretty much an even number of blue dots uh which is i believe model one tensorflow as there are orange dots all right so and there's a funny picture of me from a country music show back in chicago let me show you the actual notebook here so this is part of our workshop uh the the github repo is in the next slide which will will have all the resources um this is that same diagram you can go this is all open source code and let's see there's some really really good blog posts that i have linked here uh there was this nice one i think from august or maybe september from a uh like san francisco company here called stitch fix uh they're doing these multi-arm bandits um there's some some intros to contextual bandits there's a vocal wabbit uh band algorithm section as well here's some theory and then here's some of the actual code and links to uh the amazon specific stuff all right this one we've already seen so you know uh this like diagram shows actually model two doing better that's not the case for for this particular sample we actually do end up in sort of a 50 50 case here so here's some you know boilerplate sagemaker code here let me just keep an eye on time i think i got about 10 minutes left okay here's some of the boilerplate sagemaker code this is where you're just setting it up getting a reference to the manage sagemaker service we have already deployed the tensorflow model and we've already deployed the pi torch model and those run as sagemaker endpoints which are basically rest endpoints they do support grpc as well too and let's see we just make sure that the prediction so this is great it's okay that's working uh this is great is also getting a 5 from the the pi torch model so yes actually ran this like right before because the um simulation takes about 10 15 minutes to run to actually get some like usable results here um and this is something to point out here as well too which oh and then here's a link to the actual reinforcement learning container that can be used this is an open source docker file you can crack it open you can extend it and bring your own docker container to sagemaker also here we're just using the stock vocal vocal abbott one from sagemaker it has already been tuned i don't need to do anything except specify this config yemel this is where i can actually determine what are the instance types um here's some you know this particular bandit we use like dynamodb which i'll show those tables here in a bit very very lightweight stuff uh very very cheap we use pretty small instance types too there's nothing really too too fancy going on here um you know c54 excels uh pretty good machines this is where we're we're setting up the exploration policy so this is where we're using cover um and that's like the online cover so if you like do want to use the epsilon greedy um you can specify it here you would then specify epsilon as the hyperparameter we are using cover which um is it which has uh something called number of policies so there's actually three different little sub policies that are um each like separately making predictions and then coming together in quorums that's why we use three here you would typically want to use some some odd number so that like you can reach quorum uh and you know this is a hyper parameter if you wanted to use five you would put five in there seven nine uh like whatever you want um number of arms this is actually number of models so numbs is two because we're comparing model one to model two if you do want to add a third model you would then change this to be gnome arms three uh and those are like the actual arms that you're they're like uh like quote unquote pulling um oh yeah so i should say bandit by the way is a slang term uh for a slot machine if you go to vegas and or you know somewhere where there's gambling you um would uh like typically call those bandits because they're just taking your money basically is the uh joke there all right scrolling down a little bit um this is all code that's open source you can get it either from my repo or find it on the amazon samples and setting up the experiment manager so this is where we shift into uh taking that bandit and then running it on like amazon at scale so that like you can run these experiments at scale so here's where we're essentially yeah there's sometimes errors here that you just get used to uh stop paying attention to them because they're not really errors um let me kind of scroll through here this is just setting up the initial states um we've got a lot of code in here this is actually a relatively new notebook like i said one or two months old deploying that first model so this deploy model will actually push out a third model so there were two burp models and then this third reinforcement learning model and this is the first version going out that really doesn't have a lot of information now you can train the the first version of the model on historical data that's like assuming that you do have the historical data um here we are starting from scratch and um we are then like collecting um both like that that top uh that top s3 bucket the event data and then we are matching it with the reward data uh that comes after making the prediction and then comparing to the ground truth and here's a simulator here we're actually cracking open one of the sample data set files here and simulating as though someone is typing in um the review body and then the star rating and this is where where we are actually comparing what is the bert predicted class which is star rating one two three four five four um yeah so for that text and so one by the way is bad five is good um we are then comparing it to the ground truth label if there's no difference we give a reward of one to that particular prediction all of this then gets saved and put into s3 uh this is where we're actually calling so here's that that burp model map so model one was the tensorflow model two was the pi torch if you had a third um third arm or third model you would put it in there all right and this is where we're actually going to generate random events uh and runs through here so looks like model one was being served up initially which is kind of what we saw that was all those blue dots we should then start to see um let's see predictive model uh oh okay that was just a small sample here but yeah we'll see in a in a sec where um there's both model one and then model two showing up this is the sample json that's being written uh out to the s3 bucket and then being tracked by dynamodb also and [Music] what we're gonna see is there was a reward zero here uh there we go let's see here uh and then that was the action so here the like action that's being served up by the bandit by this reinforcement learning model is which bert model to so to tell the client application to call um with its um star rating prediction okay we've got action probability and sample probability sits on top of the action probability so the like combination of uh so in this case there was a very high probability that they should be calling um one particular variant one or two but then on top of that is this sample probability that then makes adjustments to um for that particular exploration strategy okay and all of this then goes back to s3 excuse me for the scroll i just want to get to the interesting stuff here now we are deploying a second model so this is doing things manually but in a sec i'm going to get to where we're actually doing this continuously right here okay so here we're doing five loops we're doing 100 uh batch so that's 100 predictions per loop all of this then goes in we are calling basically all that code that we have presented earlier and shown manually and then tying this all together so this would run continuously um so in sagemaker there's something called processing jobs which are like essentially free-form jobs um you can put like any python code in there uh there's actually spark support now also so um you can actually run serverless spark on sagemaker which like not a lot of people know about so we have an example in the book and then also in this repo which i'll give you the link here um next so uh this processing job would then be a long running job or you could fire off a batch job that would uh like get scheduled maybe every 15 minutes every hour or something uh which would then gather all the data and then push out the new bandit model and so here's where we could actually look and and check if the evaluation score of the newly trained model is better than the current model we would then push the uh like new model out if it's not then we don't push it out so we are constantly training and then evaluating we have you know separate holdout data set where we can train and then validate uh and now now we see the actual loop here so this sending a bunch to two this is probably going to have a lot in it here let's see that was a batch of 100 and that goes on and on this is what actually takes a few minutes uh it should start shifting over to one here soon this is gonna drive all this crazy if i keep scrolling but let me see if we can see now we start to see one twos ones and twos we see the actual class and the predicted class here so some of these are correct like that one's correct this one's slightly off and so rightly you can actually decide when to reward the bandit sometimes i've actually changed um like this algorithm to treat fours and five similar that's positive sentiment essentially um or uh um and three would be neutral and then one and two would be negative so if you wanted to actually change the reward sometimes you'll see different results better results because you know really we don't necessarily have to match it up directly but if it is close that could be positive reward as well so all this goes on this is just you know taking a look at status this was actually me like during development i was trying to understand what all was going on here so i've got lots of logging um and here we see let's see i just do the last 10 here all of the joined data so i'm just kind of showing you what the actual um the like reinforcement learning model is looking at and um and how it's actually choosing which action to serve up so we see the combination of the action probability sample probability leads to the actual action that was served and then we see the reward here so what we'll see is the total invocations actually so this looks like two like ended up actually getting more um yes overall and this you know tends to go up and down based on how many if you do a thousand of these uh so it looks like this was only 500 but i would say this is maybe slightly skewed um towards the uh the like pie torch model itself um we'll we'll dive in deeper a little bit here and look at some of these charts so let me get so here's an invocation chart now this shows something very interesting here so this looks like so bert model one and the invocations actually halted and i started looking into this right before the talk i think either the tensorflow model died or something happened here so there's actually more going on than just the bandit but this kind of shows that at the beginning uh bert model 2 was getting all the like invocations and slowly we were exploring one and we were starting to send more traffic and i've only seen this once or twice where that tensorflow model just stops taking traffic and then the bandit figures this out and um so there's something strange going on there but logically so let me keep going here and then this is a chart of the action probability and so we see that first it was burp model one then it was burp model two um and then it starts to go in and out here this one's actually more interesting this is the actual sample probability um on top of that like action probability so we see blue then we see a bunch of um orange for the burp model 2 a little bit of blue and then a whole bunch of orange so actually looking at this again i would say that uh bert model 2 does look a little bit um like things are are heading more in that direction but see here sample probability for 2 is about 0.5 and here it's 0.49 so there's a couple different angles that you would want to uh look at here now this is a really interesting chart this is called the rolling mean reward now what i didn't uh like tell you is that these models were trained very very short amount of time and the tensorflow model and the pi torch model were showing about 40 42 accuracy which is not that good we trained it for about um i believe 30 minutes if you start to train longer and and start to go 45 minutes or no sorry these were actually trained in about eight minutes and each model got about 40 because like we didn't show up that much data if you go to 30 minutes suddenly like you start seeing 90 accuracy bert's like wonderful but burt needs data um and we are actually fine-tuning bert again so we're starting with the uh with the uh like base pre-trained bert that's been trained on millions and millions of documents uh it's been trained on wikipedia it's been trained all over the place uh right like google corpus book corpus um and but here we are fine-tuning it to our specific amazon reviews customer data set and what we see is it's kind of starting to head up so at the beginning we oh so the top line like accuracy here or the like top line reward would be about 40 because that's the best we can do with uh with two models that have only 40 accuracy and we see that it's sort of even you know maybe around 20 25 uh it starts to explore things don't go well then it shifts back and and um so this is overall this is our reward regret sort of trade-off and then we see it's doing well but we still want to explore that like other model right uh so maybe now we're back to exploring one or you know something happens um yeah there's a couple dimensions we have to look at here uh with those invocations stopping up above but um and let's see it it starts to get close to 0.4 and then goes back down oh yeah are we i just wanted to i don't know the constraints of the organizers but just want to let you know we're running a little over and so okay here's yeah here's the dynamodb table that's tracking all these experiments um just uh these three here that's one from another project i'm working on and then this is the s3 buckets that have all of that data so let me go back to all right here's all the resources that's the github repo right here um if you wanted to pick up the book or find an early release copy click on that link there's the reinforcement learning container and thank you so much for your time [Music]