ai.bythebay.io: Tom Brown, Measuring AI Capabilities with OpenAI Gym & Universe
Recording: ai.bythebay.io: Tom Brown, Measuring AI Capabilities with OpenAI Gym & Universe
Today we're going to talk about measuring AI progress with OpenAI Gym and Universe. Um And so those are two different products that OpenAI has built out. Uh I'm going to explain why they are, kind of how to use them, um why we should care about them, and we're going to do a little collaborative coding uh adventure together where we're going to implement a superhuman agent in about 10 minutes. So we got a lot to cover. So first, um what is OpenAI? OpenAI is a nonprofit research company that is focused on building out uh AI in a way that's good for everybody. So it's about 80% researchers right now and about 20% engineers. And um our mission, oh and also plus Jack, uh who runs our comms and has helped me put together this talk. So thanks, Jack
Uh and uh OpenAI's mission is to build safe AI and to ensure that the benefits of AI are as widely and evenly distributed as possible. And since we're about 80% research and 20% engineering, uh that basically reflects how we tackle this problem, which is by doing fundamental research ourselves to push on the capabilities of AI, uh as well as building out tools to make everybody in the global research community more capable. Um And I'm Tom Brown. And similarly, I have a personal mission, which is to find allies to push on safe AI and to make those allies as capable as possible. Um, I'm one of the 20% of engineers at OpenAI, so most of my time is spent trying to figure out how we can build tools and products that will make the global research community uh more capable, um more unified, uh and more knowledgeable about what's going on. Um, and so today we're going to talk about some of the different products that the team has built out. So, first I'm going to talk about the power of metrics. Uh, you all are here at an AI conference, so you are very analytical, so I assume that you're already sold on metrics
I'm going to argue that you should be even more sold on metrics. And uh I'm going to talk about that um in three different contexts, uh for individual tasks, for uh organizations, and for uh entire research fields. So, first tasks, um I'm going to talk through uh basically uh gym and universe are ways of defining tasks. I'm going to talk through them more later, um but uh you can look at this learning curve here, which is some task that you have a reinforcement learning agent trying to achieve. Um, and you can see it sucks at the beginning, then it gets better. Um, so here's a video, and you can see that it wants to walk. So, the metric that it's trying to get to is how far to the right in this infinite plane it has walked, and you can see it can like walk like two steps. And it like isn't getting better very fast, but if we wait for another 40,000 steps, um it gets like pretty good
It like it could work on its running form a little bit, but it's like it's it's moving very quickly. So, let's look another uh uh environment. So, this is a racing game. So, similarly, you have this boat, and what you're trying to do is race faster than these other boats. Um you can't see here because it's cropped, but uh instead of having uh a distance metric, we only have score. So, what we measured here was score. Um and you can see if you look at this graph, it seems like our score goes up over time, right? Uh and it seems like something interesting happens at around 80K. And so, whenever something like that happens, we'll get really excited, and we'll go and look at what actually is happening by um watching a video that's been made
So, here's the video that we found. All right. So, it's driving. You can see here it's got a thousand score, so it's happy. Um still going. And it's not driving great, but somehow it's getting this high score. Up, and so it hit those buoys, and now it scores 3,500. 60 60,000
And so, what it has learned is that trying to like win at this race isn't the way to get a score at all. What you should really do is spin around and hit these buoys a lot. Um but, the bummer here is that like this boat's on fire, and like it's spinning around. Like this is definitely not safe AI. I would not want to be in this boat. So, something has gone wrong here. Um and the particular thing that's gone wrong is that we were measuring a proxy for reward, um and we've misspecified what the actual metric is that is uh supposed to be optimized. Um so, this is the type of things that you discover when you end up having a bunch of different environments that you're training agents on
Um and I'm also going to argue that this same type of thing can happen uh not just for individual tasks, but also for um uh organizations. So, uh I came from a startup background. Um I went through Y Combinator a few years back. And one of the um kind of a main motto that they drilled into us was that you measure uh you make what you measure, which basically means that anything that you put up as your like number one metric, if you take that metric, put it up on the wall, um it will tend to go up, and you'll just constantly be thinking of ways to make it go up. Um And that you'll also be vulnerable to misspecified rewards there. Uh and that actually was something that I found in um my company, we basically ended up optimizing for getting the most users, uh but didn't measure how happy we were making those users. Um and that ended up, as you would predict. Um so, I think this uh so, it this affects organizations
It's incredibly important to figure out what it is that you're measuring. Um And it also affects entire fields. So, uh you're all familiar with ImageNet. Um ImageNet was launched by Fei-Fei Li in 2010. And uh when it was launched, it basically seemed like it was uh too difficult for people. There were already good uh benchmarks that were out there. Um and you can see that over from 2010 to 2011, uh there was only a few percentage uh points of dropping. Um but what mattered was that it was something that was already out there, um and people were all working together to try to improve it, because Fei-Fei knew that this would be an important thing uh that eventually would happen
She didn't know how long it would be, but it seemed like we uh would be able to solve or to work uh to get much more progress on vision. Um so, in 2012, AlexNet came out. Um and because we already had this benchmark, it was something that people couldn't ignore. This was a like night and day shift, and basically launched the entire deep learning revolution that we're seeing now. Um plus as we've continued over the years, this has just been pulled down uh bit by bit by people collaborating and working to make this number um uh decrease. Uh and now you can see as of 2015 that uh the human who actually is uh we know because of OpenAI's very own Andrej Karpathy, um who spent about a hundred hours training his own brain on how to label all these images that you have to like label for ImageNet, um we know that now these uh our image classification is um superhuman for ImageNet and basically solved if like labeling it sorry, supervised learning is basically solved if you can get um enough labeled data. You are you can be confident that you'll be able to uh train an agent that will perform as well as a human. Um which is huge
So I I think the next question then is what comes what metric comes next if we want to look at a field as a field at um where the most important changes are going to come from. Uh and I would argue that reinforcement learning is one uh is a good candidate for that. Um and so I'm going to talk a little bit about reinforcement learning uh and then I'm going to code up a reinforcement learner um or code up an agent that works in reinforcement learning environments uh afterwards. So RL um many of you are familiar with this. Basically, the idea is you have an agent that's interacting with an environment. Um the agent is uh if you the agent is the actor that you're coding up, the environment is everything that's not the agent. Um and the agent can take some actions on the environment. Uh Uh the environment will have stuff happen
If you imagine this being a robot, the robot can move its arms. Um uh it can change its actuators and then the environment will give it back some sort of observations to its sensors. Like if it has cameras or if it has um ways of feeling the distance that it's moved, those will be the observations. Um and you also need a reward which lets the agent know how well it's performing the task that it's trying to do. Uh so this is the standard agent environment setup. Basically, gym just implements the environment. So, you call gym.make with a string that defines this large set or that defines a single environment. You'll always get back exactly the same environment when you call gym.make
Um and if you go to gym.openai.com, you can see a list of all the different environments that have been added by the community. Um So, for example, you could do Ms. Pac-Man and when you call gym.make, you get this little environment. Um and so next when you call step, you'll get your observation and reward back. Um and that action will come from the agent that you write. So, you'll have to figure out some action to give it. Uh the other thing it needs to have is reset which brings you back to the state. Um so gym, make, step, reset, very tiny API, um but you can fit a lot of stuff into it
Uh so, the community has added uh all the different Atari games. Um we've added a bunch of different flash racing games and I'll talk more about these later. Uh these are interesting because basically all racing games are kind of the same. You're like trying to drive and you want to drive fast and if you drive fast, you'll basically do well. Um excluding that boat mix-up. Uh we've got some uh a bunch of different robot control tasks, um which uh basically instead of having uh so sorry, for all of those previous ones, the input the observation will always be screen pixels. For this instead, you have uh different sensors on these robots' joints that um become your observations. Um and we've even added some safety things like uh this predict action cart pole
Cart pole is a classic um reinforcement learning task where you have to balance this stick uh and predict action cart pole tries to balance the stick while at the same time letting you know what actions it's going to take over the next like few seconds. Um which would be a useful thing to do if you have like a car, you'd want to be able to know what the car is planning to do. Um so there's a bunch of different types of safety problems. This is just one example. Um and then most recently we've added a bunch of browser tasks. So this is very fast, but if you pay attention to what's going on here, uh this is an agent choosing a bunch of different things to do, like browsing a file system, moving filling in this form, uh trying to stay in this black box, um basically anything that you can do on a browser, um which copy-pasting, selecting from list, and I guess like banning people on Twitter or something. So anyway, um so a question then is like what doesn't fit in this API? And it's basically just environments that can't be reset. So you can fit almost anything in there, um except for things where you have like an actual robot interacting with the world
Uh the robot then you would want to be able to have it practice a bunch of times, um but you probably won't be able to set up the environment exactly the same way that you had it originally. Um unless you had like a time machine. Uh so, uh I think that that's okay. It seems like most tasks actually fit within this. So, next I'm going to tell you about Universe, which is our giant collection of different gym environments, and I'll tell you why that's cool. Uh but first, as promised, we are going to implement this agent. So, let me grab a drink of water. Ah
Okay, 10 minutes. So, uh we are in Python world. We're going to write uh a simple agent. Um we're going to do our env.gym.make. Let's do Breakout. Um and again, if you're looking for a list of environments, you can find that on the gym website. Um we're going to import gym. Uh we need to get our first observation, which is env.reset
Um and then we're going to take a bunch of steps. So, let's do like for we don't care in range 500. Um we're going to take a step. Uh and our action here is just going to be one. Um and I know that the one is the action that I want here because um I've like done a lot of these Atari things, but you can look it up here in uh the gym repo. And generally, uh the gym code is quite well documented. So, here, uh one is fire, which is also like the okay button. So, we've got to hit okay to like get started
Uh so, we'll call env.step, and then just so that we can see it here, we'll env.render. So, let's see if this works. There we go. So, this is our wonderful agent. It plays Breakout. It didn't do great. But, all right, let's let's make it better. Um so, problems with this agent
Uh one thing is that we don't do anything with m.step. So, m.step returns obs, reward, done, uh info. And what we want is to if it's done, then we're going to restart the environment. Um and let's say if it has a reward, uh let's do let's print the reward. Um and right now our agent's not getting any reward. So, let's fix that. Let's do good old random choice of 1 2 3. So, now our agent is going to uh move randomly and it should get some reward
Come on. There it goes. Reward one. All right, so still not superhuman. Let's see what we can do. Um So, one thing we can do here is uh let's measure the total reward, too. So, here we go. Total reward uh is going to be zero
And then uh when we print the reward, we're also going to add it to the total reward. Very interesting. And then at the end of this, we're going to print the total reward. Great. Um and uh let's actually look at some other environments, too. Um So, what's cool about this is that this uh works for breakout. It um will also work for uh Let's do like Gravitar. Basically, any of the different Atari games have the same input
Um So, here it is. It's doing about as well as I would do at Gravitar. Uh let's see. The um So, they all have the same input, which random choice of 1 2 3 just returns a single number, either 1 2 or 3. Um So, uh any of the different Atari games, like Ms. Pac-Man, etc., will work there. Uh let's do um some of the robotic control games. So, we could do Ant
Uh for Ant, this like 1 2 3 action won't work because in Ant, instead of having like a little controller that you're using, you have a uh these robotic limbs. So, instead, we need a bunch of zeros. 1 2 3 4 5 6 7. Hey, this isn't what my code looks like in production, but if we give this agent this Ant-v1 if we give it this input, here we have our ant, and it sits there with no strength in its limbs. Um and you can see here it's giving it's getting a reward of 0.99 um each second. This is basically just the reward that it gets for just being alive. Um the ant actually wants to get to the right side, but uh this ant just sitting there is okay. Um Okay, so this is our ant
Let's upgrade our ant. Uh let's say action equals um we have like a helper for doing the sample thing. The random choice would be uh um a little bit more annoying with like this eight-dimensional space. So, we can do um env.action_space.sample, which will just give us a random action. Um and here we go. Okay. This is not This is not the best. Still not Still not superhuman
I think that uh we've actually had people in the office try to play as Ant. Um, it's incredibly difficult. Have you guys ever played like QWOP? It's basically like this game here where you're like trying to move all of your muscles, but like your brain doesn't know how to move them at all. So, it's it's it's this. So, uh we can do slightly better than that random agent um as a human, but the score that we end up with uh is uh like maybe we've gone like three steps to the side before our time runs out. Um, so uh let's do better. Uh All right, I said we'd do this in 10 minutes. Um, I wanted to give you guys a quick overview of policy radiance, but I uh Jack said that we did not have time for that
So, I'm going to import my trained agent. Um, I've a MuJoCo agent, and we are going to cheat and use that. So, uh I've got my MuJoCo agent. I'll also need TensorFlow. Um, so we're going to start up a little TensorFlow model that has been trained. So, we're going to have with Tensor TensorFlow session. Take all of these things. Import TensorFlow, and now we want our policy, which is our MuJoCo agent, which basically has just a very tiny neural net in it
Um, let's do So, basically uh here in my directory that we're looking at, I've got this trained models folder. Um, and inside of it we've got a bunch of different uh dot pkl is a dot pickle file. So, it's basically just a dumped Python object um that contains all the weights for uh an agent that we trained last night that knows how to do these various different environments. So, since we want ant, we can use this little uh tiny little 148k ant pickle file um and pull those weights out and put them in our TensorFlow object. So, that's what policy does. Um So, let's do it. Uh our new action is going to be this policy, and the policy takes an observation, and it gives us the best action that this model has learned that will in the long run give it a high score. Uh so, if all works well, we should start this, and we should see an agent that really knows how to get a high score in this environment going and getting a high score
Let's see. Or, it'll give us a bug. All right. So, I have to find where this thing lives. It's this pickle file. Doop doop doop. All right, take two. Oh, wow
All right. So, this is our trained agent. It knows how to run really fast. Um and so, this is how you get an agent working in gym. Total reward is four, which was much higher than the like zero that we got before. Actually, I did a typo. Total reward should be like 4,000. This should be total reward
But, that's okay. Typos. All right, cool. So, um this is like a basic primer on gym. Um what you can do is after you run this uh you can upload your trained results. I'll show you a picture of that here. And I'll explain what universe is. All right
Um universe It's an entire universe of gym environments. It's got lots of gym environments in it. So, we're going to talk about why that's cool. Um to start with the like imagine uh if you have Yeah, so we have about 100 racing environments, each represented by a little pink dot here. Um Here's some agents playing them. They're clustered together because they're similar. So, these racing environments are like Yeah, like I said, they're you do basically the same thing. I played a lot of racing environments now, and uh there is not a ton of diversity
So, these dots um are uh I just like drew them in this PowerPoint, but the idea is that they are in a space of all possible environments. And they're grouped together in location here by how similar they are. Um and we don't have a great idea of like what that actually means beyond the intuitive idea that like racing environments seem similar. Um the robot environments seem much different. Um And what we expect is that as we get this large collection of different environments and see what agents work on different ones, we'll have a better idea of what uh similarity of environments actually means and what the different axes are that are collapsed into this 2D space. Uh And one way that we want to look at that is through transfer learning. And so, the basic setup for transfer is got this 100 different We have 100 different racing environments. Let's pick 20 of them and train on the first 80, set the 20 aside, um and then after training on those 80 environments, see how much better we are at working on these new these 20 like holdout environments, these 20 test environments, um to uh compared to an agent that has never had any experience with racing environments before
And if you can show that having had experience with all these other racing environments made you better equipped to deal with these test environments, then you've illustrated transfer. Um And there's many people who think that that is uh one of the big differences now between our AI systems and humans is just that humans have tons and tons of experience going into any new task, whereas these agents are taking their very first steps when they um are placed into it. So, you can imagine we have this cluster of racing environments that we think are uh that are close together. Um You can look at all the web browser environments that um we showed before. Uh and to me they feel like they're probably more spread out that having played like 30 racing games would make me more prepared for another racing game than having like done 30 tasks on the internet would have prepared me for doing the 31st. Um so, that's why these are kind of spread out. Um But even more uh those two groups are very spread out in terms of space, where having done racing games won't feel like prepare you for filling out your taxes at all, basically. So, Universe um is basically that space, and what we're trying to do is have uh is to first show transfer within these um individual clusters, uh and then expand that to have a single cluster that encompasses every task that uh a human can do on a computer
So, the idea here is that um basically you can measure intelligence or think of intelligence in general as your ability to achieve your goals in a wide variety of tasks. That's the best uh definition that I've heard of it. Um and Universe fills up a large portion of the total tasks that a human being can do on a computer. Um there's a little bit around the edge which uh it would be difficult to um have that reset constraint, but almost any task a human can do would fit in this. Um and what's exciting is that we already have some solved environments in this. Uh for example, playing that little ant game, doing that as a human is much harder than QWOP um whereas an agent can learn how to do it without having any experience with anything else um in about 20 minutes. So, that's an example of something that's already solved. It's like a tiny fraction of the total possible space
Um but we think it's growing or it it clearly is growing and uh we expect this to be something that will eventually fill up this entire circle. Um and you can see some of this progress now on gym.openai.com where people can upload their results um and uh track the progress of various different agents on various different tasks. So, things that I'd like from you all is um if this seems interesting to you, I'd love to uh get you using gym um telling us where it's great, telling us where it sucks um and submitting your agents to this uh to the scoreboard so that we can see what the state of the art is with AI. Um and if you think that uh building these types of products or pursuing on this type of research is interesting. Please check out uh openai.com/jobs or come talk with me after this or send me an email. Thanks. And we have time for questions. So, raise your hand and I'll bring the mic back to you
Anyone have a question? And the back. Always in the back. Um I was just interested in what you found at Open AI in terms of um using transfer learning to like solve all of the racing games. What what are some of the techniques that you were found successful or that you're excited about that kind of let you learn from one area and apply it to others without forgetting what you've already learned. Yeah, so um the question I think was um uh in terms of transfer, this is a uh new Yeah, in terms of transfer, what has been effective effective at Open AI particularly in racing games. Um so, uh we have tried basically the things that you would guess for um transfer within racing. Uh so, we've done um behavioral cloning. So, looking at human demonstrations of uh people doing racing games um as a quick way of getting an agent that um is fairly capable
Um and then uh trying that on uh very similar environments. So, um one thing we did is we just like unlocked a ton of different levels of the same game to make kind of like a baby step version of transfer from uh uh one racing game to another. Um And so, uh and then another thing you can do is then after you've trained with that, kind of um use that uh agent that you have bootstrapped off a human as the starting weights for doing more reinforcement learning. Cool. Any other questions? All right. Oh. Are you allowed to ask questions, Alexi? Thank you, Donald. This is great for developers to play
Could you mention what does it take for an average developer to start, right? So, you have GitHub, you have a readme file. What is the process like of engagement with the gym? Okay, so the question is what for an average developer, what is the process to get involved with gym? Um Yeah, so I I actually got into AI only like a year and a half ago. So, I was asking that exact same question then. Um and uh uh basically uh Greg Brockman has a post on Quora that kind of I followed the instructions for, which were basically first taking Andrew Ng's course online to give you a basic background in ML, then taking Karpathy's course on deep neural nets, and then basically like playing around with some of the simple gym environments. Uh I basically followed John Schulman's deep RL course and started out making like a kind of like throw everything in a dictionary Q-learner without any deep learning. And there's simple enough tasks where if you're doing an environment that only has a few thousand states, that that will actually work fine. And then when you scale up, you can take out the part that's looking stuff up in a dictionary out and replace it with a deep neural net. So, I think just like basically some combination of getting your hands dirty and seeing where you're stuck plus like online courses can get you there quite quickly
Cool. Oh, one more over here. So for the games the the notion of a reward is pretty um straightforward. It's you know, it's like the score whether you get first place and so forth. But I'm I'm curious to to know like for the web browser tasks, what what do you define as a reward and how do you see that um extrapolating across other tasks that uh are more complex that people are doing on computers? So the question there is um with uh a racing game for example, the score is very straightforward. There's just like a score on it um whereas for if you're asking someone to like book a flight or something on a web browser, that's a much harder task to get a reward for. Uh um I think that's definitely true. Uh So I think I think one thing that you can look at there is the um density of your rewards
So with uh a racing game, you have a very dense reward where every second or less you get feedback on how you're doing. Um whereas if I booked I need someone to book a flight uh you only know whether or not they've booked the flight at the end. So that's I that's one problem. Um Many of these tasks that we've built out are uh kind of uh simple form filling tasks. Uh so um we can uh I guess two things there is one thing is that um the just denseness of rewards thing is uh something that there's various different techniques that you can get around. It's definitely easier to have um to make to quickly train an agent that is an environment with dense rewards. Um we've also had success with training agents in environments that have less reward. Uh where you basically can evaluate how well it's done, but it's harder to like know how well it's doing second to second
Um and then for these uh particular web browser tasks, uh if you you can check them out um if you Google for uh miniWOB, miniWOB, um Andre Karpathy has a post about um how they work. Uh they were hand-built, so we like wrote a bunch of JavaScripts to make these different environments where they generally have um clearer rewards than uh just like a random like do this thing on the internet task that you'd find in the wild. Can I also ask a question? Yeah. Uh would it be possible to use this exciting thing gym in the real world like for racing cars? Yeah. So Building an environment ourselves, observing the world with a camera, and then implementing the the algorithm. Right. So the question is, can we have gym environments that are interacting with the real world? Um and I I think broadly the uh the problem there is it's difficult to reset the real world. Um so if you can set up an environment in the real world where it's you're capable of resetting it, then that's fine
There would be a human who would There'd be a human who would reset it. Yeah. Um so that's actually something that I don't think is that crazy. Uh the a particular thing What's that? The reward would be complicated Yeah. Well, so so the the particular way that people do that is um let's imagine that you're trying to train uh an agent to drive a car from like point A to point B, um or let's say you're trying to yeah, get an agent to like stack a block on top of the other one. If you run it in the simulator, uh you can make a gym environment that has a very close to reality simulation. Um, and if you just run it in the simulator, you can train it to be quite good at this. And, uh, if you then are, um, have something that works with transfer, you can basically, uh, have it learn in the simulation
And then, if all goes well, hopefully it has learned how to act in the real world as well. Sure. Okay. Thank you. Awesome. Thanks so much.