ai.bythebay.io: Stuart Russell, The Future of (Artificial) Intelligence
Recording: ai.bythebay.io: Stuart Russell, The Future of (Artificial) Intelligence
you [Music] [Applause] so I'm not going to talk about the present very much but I do want to mention of course that as we know AI is making a lot of progress I'm sure many of you have seen this graph showing the rate of improvement in object recognition this is on the image net task where machine learning systems have now in many practical senses exceeded human capabilities for recognizing a large number of fine grained object categories and images you've probably also seen Lisa doll worrying about losing his game against alphago so these are very exciting times I just want to show you a little bit from Petera Beals lab as my colleague Peter and this is our robot Brett and Brett has to do the laundry and just to show you how fast things are moving Brett will be actually in a museum next year at the Victoria and Albert Museum in London there the three-month exhibition on the future and Brett will be one of the centerpieces of their exhibition and hopefully he'll be folding laundry for three months if we can if we get him to work for that long and people can just give him laundry to fold at the Museum and he'll fold it so let me talk about something else so I'm sure you're all up to the eyeballs in deep learning I want to talk about another approach to AI which is also making a lot of progress that's probabilistic programming so a pair allistic programming program is if you like a Bayes Nets on steroids so it's a way of writing power ability models using not a circuit language which Bayes Nets and and deep learning are these are circuit languages this is a Turing equivalent language so it gives you the full expressive power of first-order logic or a regular programming language but you're writing a probability model directly and in our language was called Bayesian logic or blog for short you can show that every well-formed program programs that don't in some sense have infinite loops in them defines a proper distribution over the variables of interest and then we can write general-purpose inference algorithms that that do inference correctly and converge in the limit for any program that you can write in the language so for those of you who do deep learning you can think of this as as a dynamically generated deep learning generative model and so in the example I'm about to show you the the programming system generates on-the-fly networks with up to several million variables but the structure of the network can change as the inference proceeds and the number of variables can change as the inference proceeds so it's a much more flexible and general way of building machine learning systems so here's the application that we developed over the last few years this is a this is the result of a small nuclear test that took place in Nevada a few decades ago the biggest nuclear test was about 500 times the size of this and just to give you a quantitative number this test ejected 12 million tons of Earth and rock into the atmosphere so nuclear testing has been going on since 1945 there have been over 2,000 nuclear tests and the tests themselves not the not the ones in Hiroshima or Nagasaki but just the tests have killed over a hundred thousand people because the amount of fallout that they produced and they also of course facilitate having a nuclear arms race so there's an organization the United Nations has the organization called the comprehensive nuclear-test-ban Treaty Organization or the CTBT oh and there they're the guardians of a treaty and one of their jobs is to make sure that nobody is testing nuclear weapons anywhere on the earth so if you think about that problem right and in the standard Bayesian way you have to have evidence you have to have a model and then you have to have some inference happening so the evidence is collected from about a hundred and fifty seismic station which is scattered all over the world called the international monitoring system and that looks like this so those typical Wiggly lines that seismograph sounds the size seismograms describe the minut vibrations of the earth when seismic events take place the query is what happened so every 24 hours we have to produce a bulletin saying these are all the seismic events that took place anywhere in the world these are the ones that are suspicious because they took place sufficiently close to the surface and they're not explained by natural seismicity and then the model is everything we know about geophysics so where do natural events occur how are signals transmitted from natural events how do the detectors work what's the cut what's the noise that's corrupting the signal and so on so so that's the seismic monitoring problem this is the blog program that describes that problem so this contains all the relevant to your physics in order to to build the system this is called the net visa model for uninteresting reasons but the point is that this is quite short so it's relatively straightforward in this language to write models that generate millions around of variables using a fairly short piece of code and just to show you some results when we run this model so we take all that seismic data and we combine it with this model we run inference using MCMC this is the current performance of the the previous United Nations system so this is the on the y-axis is the fraction of the events that the system fails to detect so obviously higher is worse and you can see that a different magnitude ranges the failure rates are between 30 and 50% and this is the results from net visa so we've reduced the error rate by a factor of 2 to 3 this shows that we're also getting better locations so this is the 2013 test in North Korea the black the black cross near the bottom is where where the tunnel is which we can find from satellite so the actual explosion took place near there the Blue Square is the location that our system produced and the green triangle on the top left is that's the combined expertise of all of the human geophysical experts and and detection systems in the world and our system we're completely unaided got a better location and just recently we've shown that with a somewhat more sophisticated model we can actually increase the detection rate by a factor of 10 compared to the best human experts combined with the with their software so this is the same raw input data with a factor of 10 improvement on the detection rate so just for this is serve Isuzu as a reminder that there are things going on an AI besides deep learning and that the ability to express knowledge and combine knowledge with data is actually much more effective than just using tabula rasa machine learning in many cases so just to take stock of where we are now and ruing from Baidu used to be in my group has I think a reasonable characterization of what we can do anything that takes a human being one second or less is a pretty good candidate for machine learning to be doing approximately as well or better than a human we have some really cool toys we have lagged robots we have flying robots we have self-driving cars we have perception navigation working pretty well so when I compare the situation today to even ten years ago things are looking really much further along than most people would have expected there's a lot still to do so despite many claims that we can do captioning and and that we can do machine translation and so on there certainly is progress there but I wouldn't say at the at the moment that we have real understanding of language in terms of the ability to extract information and then reason with it combining information for example from multiple different documents to answer complex questions and so on that that capability is simply not there as I mentioned the ability to combine learning with knowledge doesn't really exist in the deep learning connection and if you think about it right if you're if let's say you're you're doing deep learning to recognize objects in images well the output of the deep learning system while an image is a discrete logical category this is a cat right this is a Doberman Pinscher well if you don't have any way of using that information then what's the point of generating it right so there's actually a mismatch between what deep learning systems can take is input and what they produce is output and so they're still I think a lot of work to be done on understanding the integration of these machine learning systems with systems for reasoning and combining information from multiple sources probably the biggest thing that's missing is cumulative discovery whether it's a concepts or theories or in particular high-level actions so the ability that humans have to reason at many scales of abstraction is what lets you come to this meeting right in this meeting over the course of three days you will do several billion primitive physical actions and yet you're able to reason at the scale of oh I'm gonna go to AI by the bay right even though that's several billion actions poor little alphago can only look ahead twenty-five or thirty actions right so there's a again a huge mismatch between our understanding of how to do look-ahead planning and what humans are able to do because we have the ability to operate at multiple scales of abstraction and that's probably to me the biggest missing thing is how do we get machines to create those levels of abstractions so that they can reason over long timescales and be effective in the real world if we solve these problems and I think we are a long way to having human-level AI so let's assume that we solve those problems right let's assume that we don't just fail what does that mean it means that eventually we'll have systems that are better than us and better means that for almost any decision in the real world they'll be able to use more information look further ahead and produce a better decision what does that mean well it means a lot it doesn't just mean you know cause having fewer accidents or medical systems making fewer errors or be a better fraud detection those things are all cool but if you remember everything our entire civilization is the result of what's up here right we don't have big claws or scary teeth we have brains and our civilization as a result of the amount of intelligence that we have so if you have access to a lot more than that has to be a step change in our civilization so this is not just small incremental improvements in our standard of living or the efficiency of industrial and commercial processes this is something that we will look back on in the future as a major turning point in the history of the human race so we have to make sure that turning-point goes well and we can already see some downsides killer robots are already a serious problem there are countries who are already developing and deploying robots that can choose who to kill where where to go and when to kill these are not science fiction imagination this is this is real and the United Nations has a process underway to develop a treaty but at the moment the United States is not supporting that process so we have to be considering whether it is a good idea another topic that I'm not going to talk about is the predictions that we are seeing of massive disruptions in employment which could have huge social consequences but here's what I want to talk about since it's my sunny morning we'll have a nice cheerful topic the end the end of the human race and so we've heard a lot of speculation from various people about the end of the human race and many of you probably think this right that the people who are making these speculations well they don't know anything about AI they're not even real scientists right they're just people who like to spout off in the press so here's a little quote if a machine can think it might think more intelligently that we do and then where should we be even if we could keep the machines in a subservient position for instance by turning off the power as strategic moments we should as a species feel greatly humbled and this new danger is certainly something which can give us anxiety so anyone know who said this is Alan Turing so this this idea is actually probably not that unfamiliar to you the idea that if you make something more intelligent than you are that there's a certain unease that you might be in trouble as a species and you could ask the gorillas the gorillas are having a meeting to discuss whether they should have created the human race 7 million years ago or whenever it was we branched off from the evolutionary tree right was it a good idea and they're having this meeting you can see they're pretty unhappy about it and they decided no it was a really bad idea for them as a species to create the humans because now the gorillas have no control over their future whatsoever and they really only survived because some of us are reasonably generous towards them and some aren't well this is a bit of an inchoate fear it's hard to really do anything practical to ward off this future without understanding more precisely what the problem is so you have to say what is really bad about better AI right has all this great upside what what is the reason why we might have a problem so here's another quote if we use to achieve our purposes of mechanical agency with whose operation we cannot interfere effectively we better be quite sure that the purpose put into the machine is the purpose which we really desire anyone who said this so this is Norbert Wiener who's the father of modern control theory and automation professor at MIT in wrote this in 1960 actually having just seen Arthur Samuels checkered playing program beat its creator and beat several other fairly competent checker playing people and but you could say the same thing could have been written by by King Midas right so so this idea that getting exactly what you say you want can be a terrible thing right goes back a long way in human history and it actually appears in many different cultures with legends and myths you know the whole idea of the genie is always that the third wish is please undo the first two wishes right but with AI you may not have the ability to just reverse what you asked for in the first place so technically this is now called value misalignment so it's a misalignment between what you really care about and the value of the objective that you put into the machine to achieve on your behalf and the problem is we don't have a good we don't have a scientific discipline whose job it is to figure out what the objective should be for the AI system so all of the disciplines that deal with optimization of objectives assume that the objective just comes from someone else right it's just plugged in exogenously and then the job of our discipline is to figure out how to achieve it how to optimize it and they're like therein lies the problem right that if that's the wrong objective then you have a mess now how big a mess was pointed out by Steve Omohundro back around 2000 and a paper called the basic AI drives so he said look if you take any goal even something as simple as fetch the coffee then any sufficiently intelligent machine realizes that if someone switches it off it will not be able to fetch the coffee so as a sub-goal of any objective you give to a machine self-preservation is required so this is not something that you build in to the machine this is something that automatically follows from giving the machine any objective whatsoever similarly the need to acquire more resources so you can increase the power ability of achieving the objective if you acquire more resources so think of the objective of curing cancer if you tell the machine could you please come up with a cure for cancer then it has an incentive now to acquire all the financial resources on the earth and actually to use all the human beings on the earth as guinea pigs to to get that you know slightly increased probability of finding a cure and doing so more quickly so if you take that right take these these natural consequences of an objective and then you have an objective which is misaligned with what you really want right now you're setting up a kind of a chess match or a go match between the human race and the machine which you think is doing your bidding but in fact it's doing something that you don't quite want and we may not win that chess match so that's that's the premise of 2001 a Space Odyssey right where in fact the objective of how land the objective of the of the to humans is mismatched in fact the humans don't really know what health true objective is and that's why early on in the movie they have how playing a chess match with Dave and it's pretty clear that how can you know out think Dave you know without without even opening one of his eyes so so this is the issue that we face and yet oddly enough there are a lot of people in AI who prefer simply not to face the issue and they will trot out all kinds of reasons why we shouldn't pay attention so I've come up with a list of about 20 and they all share this flavor that they can't possibly really believe these reasons because they just don't hold water so I suspect there's some kind of cognitive process going on of a sort of self defense look I I do AI you know if you say anything bad about AI then you're attacking me and my research and and so it must be wrong right all right but this is not anti AI any more than pointing out that you know nuclear weapons are dangerous is anti physics it's not anti physics it's actually a compliment to physics that they can actually have an impact on the real world because because the physics is right right and the same with AI we've been wrong or at least ineffective for most of the history of the field so no one cared because our AI systems are too stupid now they're starting to show signs that they might be capable of impacting the world on a larger scale that we should take this as a compliment but we should also take it seriously so what are some of these reasons I'm not gonna go through all twenty but one of the ones you see oddly enough coming from AI people is well I know I've been saying for 60 years that we will of course achieve human-level AI but now that it's starting to happen no we won't right this is a very strange response but you see it quite a lot including that recent AI 100 report which is supposed to represent the sort of accumulated wisdom of the field about how things are going to go in the future they literally claim that achieving human-level AI is impossible as a way of deflecting any notion that it might present a risk let me give you a little history lesson anyone know who this is Ernest Rutherford so Ernest Rutherford was probably the most famous nuclear scientist of his day he was the man who split the atom back in the early part of the 20th century and on September 11th 1933 he made a very very definitive prediction that we would never ever be able to extract the nuclear energy that we knew to exist within the atom that was September 11th this is Leo Szilard he read that in The Times the next morning and he went out and invented the neutron induced nuclear chain reaction so that was less than 24 hours later the key problem that was preventing us from creating nuclear power and nuclear weapons was solved and he patented within a year he patented the nuclear reactor had an idea about how you could create a nuclear weapon and the first patent on a nuclear weapon was 1939 by the French oddly enough not by the Americans or the British but by the French so things can change so so predicting that humans are too stupid to make the progress that we need to reach human-level AI is a really bad idea there's a bunch of other reasons I just want to mention one more you can read these as they go by here's another one you might see don't mention these risks because it might be bad for funding so this is also a bad strategy right the nuclear power industry adopted this strategy right there are no risks will have clean free electrical energy for the rest of time you know we can get rid of all ads coal and oil and everything else and then they had Chernobyl alright and that was the end of the nuclear power industry when when the nuclear reactor really did melt down and explode so avoiding risks or trying to pretend they don't exist is a bad strategy for your own industry for your own research area face up to them and see if we can solve them okay so let's assuming though you agree with me what are we going to do about it so we have a new center called the center for human compatible AI which is a deliberately rude title to the rest of the field and our goal is to actually change the way we think about AI away from this idea that we should simply optimize objectives and towards the idea that we should make the system's provably beneficial so whatever the human's objective really is it better be the case that the machine that you give to the human race is actually going to be beneficial to them there's a lot of organizations currently a work on something like this agenda including many of the funding agencies and professional societies so what we're going to try to do is shift as I said from optimizing objective to making sure humans are happy with the results and there are three simple ideas for how we're going to do this the first one is that so think of this think of a human and a robot all right you can think of all robots and all humans but just think of this as simple there's one human one robot the robots objective is to make the human happy right but as we point out the human can't easily tell the robot the full objective the human can't list out all the things that they could ever possibly care about so that means the robot does not really know what the human objective is and this this kind of humility is crucial to having systems that are proved will be beneficial the third point is that there is a source of information about human values and that's human behavior that our actions our choices reveal what we really want at least to the extent that our limited cognitive architectures allow us to to realize those of those objectives so the process of value alignment then can be related to a field called inverse reinforcement learning which is about 20 years old it's the inverse of reinforcement learning so reinforcement learning you have you have an objective which you give to the machine in the form of rewards and then the machine has to learn the behavior that optimizes the rewards inverse reinforcement learning is the other way around you observe a behavior and you have to figure out what reward system is optimized by this behavior okay so that's the that's the basic task that we face is how do we do inverse reinforcement learning with the human race as the source of the behavior and the goal is for the machines to understand enough about our values to behave in ways that make us happy there's a slight variant on this which we actually need because it's not a passive process of observation it's an active process of cooperation so humans and robots actually are involved in a multiplayer game and this is inevitable actually that we need this as a theoretical foundation and in this game the human knows the value function in the sense if they approximately behave according to it but the robot doesn't know what the value function is and has to maximize it so that's the game and when you solve this game you find out that indeed the human and the robot do cooperate the human will actually teach the robot so that the robot can be useful to the human and everything works out as you hope and you do in fact get provably beneficial systems let me give you a simple example so I mentioned this point that you can't fetch the coffee if you're dead ok so that means that a robot that's told to fetch the coffee will for example eliminate all the other people in Starbucks just in case they might switch it off so you wouldn't you wouldn't want this right you'd want to be able to switch the robot off if you felt that the robot was doing something wrong so we seem to be facing a sort of a contradiction right we we've just argued that the robots not going to let you switch it off but you want to have the ability to switch it off so that in case it's doing something you don't like so how do we how do we fix this problem the answer is if the robot is uncertain about the true objective so it may know that you want coffee but it doesn't it knows that it doesn't know all the trade-offs involved how much you might be willing to pay you know what it costs to eliminate other people in Starbucks and so on so there's a lot that the robot knows that it doesn't know about the objective and it's precisely this uncertainty about the objective that allows you to have control over the robot because the robot reasons to itself ok why would the human switch me off well because I must be doing something it doesn't like but my objective is only to do things that the human likes so by allowing the human to switch me off I am actually benefiting the human ok if the robot knows for sure what the objective is then it doesn't let you switch it off so it's uncertainty that provides this kind of safety margin and you can this is a mathematical theorem that it's in the robot it's interest to allow you to switch it off and then when you have that you have a provably beneficial robot so this notion of uncertainty in objectives is really I think central to how we should move the field forward in the future to build systems that are provably beneficial and I'm getting the evil eye from my timekeeper so I'm gonna skip over some of these slides and yeah I'm not going to talk about why a heading which is where people who have wires directly stimulating their reward center in their brain will actually continually stimulate the reward center until they die of starvation so and then we show how to how to actually solve that problem so that robots don't do that to you and okay you skipping over this so this is a big project right this is not something that we you know we we can just set up you know a tensor flow system and collect a bunch of human behavior data and shove it all in there's a lot of complications to this the good on the good side there's a huge amount of information right we have big data about human behavior because everything the human race has ever written down every movie we've ever made every television program is really about people doing stuff so there's a huge amount of data or about people doing stuff that's good there's a very strong incentive to get it right because if you get it wrong you have these very unpleasant consequences in the near term right we don't have to wait until we have super intelligence to see the downside of getting this wrong so Google got it wrong because they specified in the loss matrix for their photo classification algorithms they specified uniform costs for every Mis classification and that's an incorrect specification of the cost of misclassifying a human as a gorilla so when you get the cost wrong you make mistakes so we can have a little story about the robot that has to feed the kids and there's no food in the fridge and it sees the cat and then this is the headline and then that's the end of the domestic robot industry right so so one one even module you know moderately serious mistake about what the human value function is could lead to the end of your industry okay so there's a strong incentive to get it right unfortunately there's a lot of difficulties which mean we have to work hard and those difficulties have to do with humans right that we are much more complicated than just a simple optimizer if we were simple optimizers we all be fine right but we're not we're very far short of behaving optimally in any reasonable sense right we have all these complicated characteristics and we have to invert the human behavior that we see we have to invert it through the human cognitive architecture to get at the underlying motivations for our behavior so that makes the problem difficult ok so there's a lot of things we could do I just want to end on some questions can we change AI the way we define it certainly I'll do my best in the next edition of the textbook I want it to be the case that at some point no one talks about about AI safety as a separate discipline that every one of you takes seriously just like a bridge designer takes as intrinsic to the meaning of the word bridge that it doesn't fall down right it should be intrinsic to the meaning of the word AI that is beneficial to people we're not building AI to benefit cockroaches or bacteria we're building AI for us and that's an important thing thank you [Applause] if anybody's the question raise your hand I'll bring the mic back to you and you can ask it so give me a second it's good workout for me hi I wanted to get your thoughts and super intelligence so there's a theory that we can get there in one of two ways first from human brain emulation or also by just purely improving AI with either cheaper cheaper chips or more time and more learning so I wanted to see kind of where where you stand on that good question so I don't agree with either of those theories of how we're going to get there I think whole brain emulation I mean technically it's possible but it's incredibly expensive and it's much less clear what the real benefits are because you still may not understand the system that you build and we have plenty of easy ways of building brains that we don't understand already right which is the usual biological way of building brains I absolutely don't agree that making chips faster and collecting more data is the way we're going to achieve human-level AI I mean somewhat tongue-in-cheek the the faster your chips the faster you get the wrong answer so we really need some conceptual breakthroughs as I as I mentioned earlier in the talk there are several areas where we know that current approaches will not work so we need these conceptual breakthroughs as I pointed out in the case of nuclear physics those breakthroughs are unpredictable and can occur at any time so given the amount of brain power being put into this problem right now the main conceptual breakthroughs could happen in fairly rapid succession so I think the sooner we prepare for how we are going to deal with super intelligent machines the better but I can't put a timeline on it because it's very hard to predict those breakthroughs Thanks so when you're talking about being able to use uncertainty in the goal for an AI to be able to you know performance tasks without harming humans well humans themselves are incredibly uncertain about what the goal is and what other people want out of your own actions you know it's like dunning-kruger effect you know you're incompetent but unaware of that incompetency so there's no reason to assume that AI is put and suffer from a similar problem and just decide that hey even though I'm uncertain I'm certain that this is going to be my action and the action can be detrimental so how can uncertainty really be a true safety net for beneficial AI okay so well that's it that's a it's a great question and it's a complicated set of issues around this probably the first issue is what if humans really don't have objectives at all so it can easily be the case that a human might be uncertain about what is my immediate goal right but do the following for experiment right suppose I had a way of showing you in some compressed speeded up form to future lives for you for yourself and you know for the rest of the world that you care about and you could watch these two movies and then decide what you know which is the one you you like best right I think roughly speaking it's a reasonable question to say yes I could express a preference or I could say well they're about the same right and so that's all we require for this theory to be correct right is that that you that you have some way of preferring one life over another right so even though you might not be sure which is which is the sub goal I should work on now so human goals you should always really think of them the sub goals alright there's a complicated process of reasoning that you have to go through to arrive at a sub goal and that process may operate against a background that you can't make explicit about what your true long-term objectives really are but as long as you have a sense that there's there's a life for yourself and the rest of the world that you prefer or don't prefer then everything goes through correctly yes in in pursuit of the value alignment as well as the understanding of these odd humans has anybody taken to trying to represent some of the findings of behavioral economics the Kahneman and Tversky types of rules about the biases and so forth of human beings as a way to give a view on human beings and their values versus their actions yeah good question so not really in a formal sense and there's there's multiple reasons why it's it's a little problematic so one is that there's a whole branch of psychology mainly the evolutionary psychologists who don't believe any of the Tversky and Kahneman claims about human irrationality and they argue that if yes of course if you present data in the form of text saying 60% of bank tellers are female right and then you ask people to make power ballistic judgments that doesn't work because text with numbers you know 6 and a 0 and a % doesn't engage your built-in power ballistic hardware right but if you present data if you present data showing lots of bank tellers of whom 60% of female then people make the judgments correctly so evolutionarily realistic data and decision problems seem to show that humans are much more rational than 2% Kahneman would have you believe also so Danny Kahneman went through a period when he was trying to figure out you know what is the human reward function right and he he ran experiments so typically in experiments you either pay people and that's the that's the reward that you give them to make decisions in experiments so that they're basically betting and you have to pay them if they bet right or you can inflict punishments and you know punishments with electric shocks are a little bit passe so he decided to use plunging your hand into ice water and that that's kind of works it's definitely a negative reward but but after a while when he tried to fit all the data that he had he found that in fact people prefer longer periods of immersion in ice water than shorter periods and at that point he kind of gave up he said I you know I give a humans are so freakin weird that that it's really impossible to fit what they really do to to the classical you know additive reward models so so I think the real problem is it's a nativity in rewards is not correct that we have some kind of memory thing that makes make some max the map there's a max term in there which is really important so there are other experiments on on colonoscopies so in a colonoscopy is it can be quite unpleasant so I'm told so you can have you know so you can measure how much pain someone is feeling so they tell you do you know click give me the clicker for how much pain you're feeling right now so you you measure the pain and it goes sort of up and up and up and up and then down and down and down and down over a course of about an hour I think the whole procedure is so if you go up on up and up on and then stop immediately that turns out to be worse than going up and up and up and up and then down and down and down and down to down right so there's strictly more total pain in the up and up and up down and down and down it down right but in fact you prefer that to the one where you just go up and then stop because I think what happens is you remember that when you stop immediately you remember the amount of pain you were experiencing at the end of the procedure whereas when you tale off gradually you kind of forgotten the really bad pain so it's it's a very complicated thing but we have to face the fact that you know since we're trying to make humans happy and not fungi and bacteria that we have to get it right we have to actually actually reflect real human preferences yeah hi so you mentioned that we should always be able to turn off the Machine and obviously the same doesn't apply for humans just because I dislike someone's actions I'm not gonna kill them do you think there's a line at which point our control over machines might become a form of slavery oh yes good question well ami you're sitting on a chair the chair is a very simple machine I don't think you you worry about the chairs feelings so the the real question is is about subjective experience you know at what point do we say the machine is having subjective experience and I know there are people for example and think that that doing reinforcement learning where you give negative rewards at any point is is inflicting suffering on the on the machine that's running the reinforcement learning algorithm right there's something wrong with that because you if if that was true right you could simply shift all the rewards up so they're all positive you get exactly the same behavior right in fact it's the same program and now there's no suffering happening so this I think there's something fishy going on and we have really no theory of what consciousness is or whether how they distinguish a conscious from a completely you know a conscious machine from a chair so until we do I think we have to just I'm afraid punt on that question Sonia talked about human values he implicitly assumed that the entire humanity has the same set of values I mean there was a there was a slide that said that we're all different yeah there was this might about hit heterogeneous Genie T of humans but so we live in this class based society where certain subsets of society as much more powers compared to the rest of the society and what is good for them is not necessarily good for the rest of society yep so is there any way to deal with that and address this problem yes so the real issue and so in economists talk about social choice theory and so on the real issue is the fact that when you have a robot or multiple robots that are serving multiple people every robot has no obligation to the whole human race so no one can tell a robot to go and do something that makes someone else unhappy and we have we have laws that further that's supposed to prevent human from doing that with each other but how you trade off the Preferences of one person with the Preferences of another a priori I think the right answer is that everyone counts the same I can't think of a form of legislation that would say anything different from that the the problem that comes up when you do that we've called it the Somalia problem so you come home after a long day and you look out the window and you know the grass is really long but you have to feed the kids so you tell the robot to go mow the lawn and the robot says we know there are people in Somalia who are suffering I'm gonna go to Somalia and help them and you can mow your own lawn right that so there's a little problem with that but I completely agree that the the issue of how we deal with the fact that there's more than one person in the world and that there are not infinite resources to make everyone infinitely happy that's a fundamental one and a I doesn't have a solution to that and so we're bringing in a lot of social scientists to join the center and try and come up with solutions and because it's a Berkeley you can be sure that the social scientists at Berkeley have the best interest of humanity at heart [Applause] you