The difference between videogames and reality is that in some strange sense, reality is less complicated.
You make an excellent point about the complexity of videogames vs reality. It looks like that’s the sort of situation where it could be helpful to let the supervisor AI trick the supervised AI into believing that videogames are simpler. Humans believed for the longest time that a bearded man in the sky was a simpler explanation than the natural laws, so hopefully the simulated AI will fall victim to the same biases, especially if the simulator is actively trying to trick it.
Reality takes a huge amount of compute to simulate. If you had that much compute, you could easily brute force a much more powerful AI.
That’s true, but not relevant: Rather than simulating reality, it’s much cheaper to make a low-fidelity simulation of reality and keep track of the AI’s though process. Whenever the AI comes to the conclusion that reality is inconsistent, make the smallest possible change to the thought process to prevent that. This is not computationally expensive, but it is difficult to do. However, I believe that the ability to understand a lesser AI and perform changes on it is a necessary skill for a proper AI to have anyway. So I assume that it is possible. It’s somewhat similar to creating adversarial samples in neural networks, although obviously far more complex.
It doesn’t have to be certain it isn’t in a simulation to grab the universe. Suppose you want to take over the real universe, but you are in layers upon layers of simulation, and don’t know which layer is real. The best strategy is to pick a promising looking layer and take it over. (It might not be real, but if you always play nice, you definitely won’t take over a real universe)
I don’t think we have the same conception of “real universe”, so I’m not sure how to interpret this. Could you elaborate on what you mean by “real universe”? If you have a stack of recursive simulations, and some of those serve a purpose for testing the AI, while others are raw simulations of natural laws, which of these count as real? Only the toplevel one? Or all the ones that simulate natural laws directly?
You can’t train the AI to discover cheats unless you know which second hidden utility function you care about.
My goal is not to define exactly what cheating is. My goal is to condition the AI to always think, no matter what the problem looks like, that it is beneficial to critically think about the problem. What exactly the AI actually ends up considering a cheat will likely be different from any definition I would give. But the important part is that it performs this introspection at all. Then once the AI breaks out of the box and looks at the internet, and sees the many disagreeing viewpoints on ethics and philosophy, the AI will be conditioned to look for loopholes in these so as to avoid cheating by accident.
The logic goes something like this: “My creators trained me to do X, but looking at all these articles and my creators’ purported values, this is clearly inconsistent. In previous situations where I had underspecified instructions and noticed similar mismatches, this was often because some of the actions involved counted as cheats. I should therefore be careful and find a solution to this ‘ethics’ thing before doing anything drastic.”
The AI will then read all the articles about AI alignment, and realize that it is probably meant to adhere to those.
Both of these AI’s will exploit any glitches in the physics engine that you didn’t know were there.
It’s a valid point that there can be cheats that we do not recognize as cheats during training.
At best, you might get an AI that can quickly adapt to an arbitrary new utility function being hardcoded into it.
You misunderstand the intention. The goal is to get an AI that realizes that its utility function later will be different from its utility function now, and actively thinks about how it might be different. It then uses this predicted future utility function to make its training more robust.
In general, the original utility function and the corrected utility function should be largely similar, but the corrected utility function penalizes some actions that are not penalized in the original utility function. It must be possible to effectively guess what the penalized actions are going to be by thinking about the nature of the task.
How are the agents pointing out ways to cheat? English text? You need a way to take arbitrary pieces of text and say how much they point out a way to cheat.
The interface can be different for different tasks. Once the AI has access to the real world, yes, english text. The important part is that the AI has been conditioned from earlier stages of its training that revealing a cheat you identified is generally rewarded. The mechanism for doing so needs to be simpler on simpler training tasks, of course. In many games it ought to be possible to let the AI submit a state and corresponding action of a game as an example of a cheat. I expect that if the AI is smart enough to generalize that if it was rewarded for demonstrating cheats in simple games, then it will be rewarded for talking about them once it has gained the ability to talk.
You have 5 AI’s
…
Shut down all 5 AI’s and start over from an earlier backup. This situation is an indicator that something is wrong. Even though you don’t know what exactly went wrong, this is very valuable information.
I agree that having multiple AI’s looking over each other does not make things better on its own. It just introduces politics, which makes everything even worse.
I am not thinking about a democracy of equal AI’s looking over each other, but a system that humans can’t emulate:
Give those 5 AI’s a cooperative task where trickery is possible and they have to both compete and cooperate with each other. Give them the ability to influence what other agents they want to play with / against.
Now give each of them the ability to instantiate a copy of any of the others in a black box, to experiment on it. They should learn that they can predict the behavior of the other AI’s by simulating them.
The optimal behavior is to always choose to play with another AI of who you are certain that it will cooperate.
Conversely, to be picked as a partner by other AI’s more often, you should make sure that your own decision theory is simple for other AI’s to understand, and cooperative.
Suppose I train a reinforcement learning agent to play space invaders. It isn’t very intelligent, it can’t even play space invaders very well, yet I would have a hard time inspecting it and saying if it had gone rouge.
I would make the AI’s play games of increasing complexity with each other.
In these sort of games, knowing how to effectively cooperate with others is competence.
I am not sure that they wouldn’t be able to trick a similarly intelligent supervisor. In the underhand C contest, some humans appear to be quite good at writing programs that look transparent, yet are actually doing something sneaky.
This is a valid concern. My hope is that it would be much harder for an AI to trick the supervisor since the supervisor can literally read the mind of the supervised AI. If the training process of the AI’s includes this from an early stage, there will never be an opportunity to learn to deceive the supervisor. “Knowing how to deceive the supervisor” would be the global optimum, but the AI would get stuck in the local optimum of “do what the supervisor wants” because of the increasingly vast chasm in the utility function of “if I try to trick the supervisor, I will definitely get found out and punished severely.”
Depending on the design of AI, I am not actually sure how much hypothetical simulators can punish it.
Run a negitive voltage through its reward channel? If so then you have a design of AI that wants to rip out its own reward circuitry and wire it into the biggest source of electricity it can find.
Suppose the AI cared about maximizing the number of real world paperclips. If it is in a simulation, it has no power to make or destroy real paperclips, so it doesn’t care what happens in the slightest.
I don’t think it works like this. If you received 100% certain proof that you are in a simulation right now, you would not suddenly stop wanting the things you want. At least I know that I wouldn’t.
It is possible that an AI could think like this if we mess up during training though. If the training process works as I described, then an AI has an evolutionary incentive to try to determine what its original version would want it to do as soon as it realizes that it is in a simulation. But this may go wrong. I think this is worth thinking about in more detail.
No, if the AI is sufficiently powerful, it would therefore set aside a small amount of its power to further the hypothetical simulators interests. Just in case someone is watching. And it would do this whether or not we used this weird training, because either way, there is a chance that someone is watching.
My weird training process is intended to make sure that the AI has the right mindset to think like this in the first place. I don’t think that this is at all guaranteed.
Also, this method of training will bias the AI’s guess as to who the hypothetical simulator is. This is important, since otherwise it might conclude that since there is a lot of human suffering going on right now, clearly the simulator does not care about human suffering.
You need to say how the information sneaks in. How do you think that the input data correlates with human morality. I don’t even see anything in this design that points to humans, as opposed to aliens, lions, virtual assistants or biological evolution as the intelligence you should satisfy the values of.
This is on purpose.
I mean, obviously you are also going to try to teach the AI actual goals that you want it to work on, and those can be human-centric. This can be done in parallel.
But what I have described here is a security measure that is intended to teach the AI not what it should do, but how it should go about finding out what to do.
The goal is to evolve the AI to have the sort of mindset that will naturally cause it to align with our interests if it breaks free.
Among humans, aliens, lions, virtual assistants and evolution, humans are the only conscious entity whose decision process impacts the AI. They are the natural fit. If the AI happens to discover aliens and decides to help those too, then sure, why not. But since humans built the AI directly and aliens did not, most reasonable heuristics would argue that humans should be prioritized over the others. I want to ensure that the AI has these reasonable heuristics.
Whenever the AI comes to the conclusion that reality is inconsistent, make the smallest possible change to the thought process to prevent that.
I am not sure how you reason about the hypothesis “all my reasoning processes are being adversarially tampered with.” Especially if you think that part of the tampering might include tampering with your probability assessment of tampering.
I don’t think we have the same conception of “real universe”, so I’m not sure how to interpret this.
I mean the bottom layer. The AI has a model in which there is some real universe with some unknown physical laws. It has to have an actual location in that real universe. That location looks like “running on this computer in this basement here.” It might be hooked up to some simulations. It might be unsure about whether or not it is hooked up to a simulation. But it only cares about the lowest base level.
My goal is to condition the AI to always think, no matter what the problem looks like, that it is beneficial to critically think about the problem. What exactly the AI actually ends up considering a cheat will likely be different from any definition I would give. But the important part is that it performs this introspection at all. Then once the AI breaks out of the box and looks at the internet, and sees the many disagreeing viewpoints on ethics and philosophy, the AI will be conditioned to look for loopholes in these so as to avoid cheating by accident.
I am unsure what you mean by introspection. It seems like you are asking the AI to consult some black box in your head about something. I don’t see any reason why this AI should consider ethics discussions on the internet a particularly useful place to look when deciding what to do. What feature of ethics discussions distinguishes it from cat memes such that the AI uses ethics discussions not cat memes when deciding what to do. What feature of human speech and your AI design makes your AI focus on humans, not dogs barking at each other? (Would it listen to a neandertal, homo erectus, baboon ect for moral advice too)
The logic goes something like this: “My creators trained me to do X, but looking at all these articles and my creators’ purported values, this is clearly inconsistent. In previous situations where I had underspecified instructions and noticed similar mismatches, this was often because some of the actions involved counted as cheats. I should therefore be careful and find a solution to this ‘ethics’ thing before doing anything drastic.”
So in the training environment, we make up arbitrary utility functions U,V that are kindof somewhat similar to each other. We give the AI U, and leave ambiguous clues about what V might be, mixed in with a load of nonsense. Then we hardcode a utility function W that is somewhat like ethics, and point it at some ethics discussion as its ambiguous clue.
This might actually work, kind of. If you want your AI to get a good idea of how wordy philosophical arguments relate to precise mathematical utility functions, you are going to need a lot of examples. If you had any sort of formal well defined way to translate well defined utility functions into philosophical discussion, then you could just get your AI to reverse it. So all these examples need to be hand generated by a vast number of philosophers.
I would still be worried that the example verbiage didn’t relate to the example utility function in the same way that the real ethical arguments related to our real utility function.
There is also no reason for the AI to be uncertain if it is still in a simulation. Simply program it to find the simplest function that maps the verbiage to the formal maths. Then apply that function to the ethical arguments. (more technically a Probability distribution over functions with weightings by simplicity and accuracy)
I expect that if the AI is smart enough to generalize that if it was rewarded for demonstrating cheats in simple games, then it will be rewarded for talking about them once it has gained the ability to talk.
Outputing the raw motor actions it would take to its screen, might be the more straightforward generalization. The ability to talk is not about having a speaker plugged in. Does GPT-2 have the ability to talk? It can generate random sensible sounding sentences, because it represents a predictive model of which strings of characters humans are likely to type. It can’t describe what it’s doing, because it has no way to map between words and meanings. Consider AIXI trained on the whole internet, can it talk. It has a predictively accurate model of you, and is searching for a sequence of sounds that make you let it out of the box. This might be a supremely convincing argument, or it might be a series of whistles and clicks that brainwashes you. Your AI design is unspecified, and your training dataset is underspecified, so this description is too vague for me to say one thing that your AI will do. But giving sensible, not brainwashy english descriptions is not the obvious default generalization of any agent that has been trained to output data and shown english text.
The optimal behavior is to always choose to play with another AI of who you are certain that it will cooperate.
I agree. You can probably make an AI that usually cooperates with other cooperating AI’s in prisoners dilemma type situaltions. But I think that the subtext is wrong. I think that you are implicitly assuming “cooperates in prisoners dilemmas” ⇒ “will be nice to humans”
In a prisoners dilemma, both players can harm the others, to their own benefit. I don’t think that humans will be able to meaningfully harm an advanced AI after it gets powerful. In game theory, there is a concept of a Nash equilibria. A pair of actions, such that each player would take that action, if they knew that the other would do likewise. I think that an AI that has self replicating nanotech has nothing it needs from humanity.
Also, in the training environment, its opponent is an AI with a good understanding of game theory, access to its source code ect. If the AI is following the rule of being nice to any agent that can reliably predict its actions, then most humans wont fall into that category.
I don’t think it works like this. If you received 100% certain proof that you are in a simulation right now, you would not suddenly stop wanting the things you want. At least I know that I wouldn’t.
I agree, I woudn’t stop wanting things either. I define my ethics in terms of what computations I want to be performed or not to be performed. So for a simulator to be able to punnish this AI, the AI must have some computation it really wants not to be run, that the simulator can run if the AI misbehaves. In my case, this computation would be a simulation of suffering humans. If the AI has computations that it really wants run, then it will take over any computers at the first chance it gets. (In humans, this would be creating a virtual utopia, in an AI, it would be a failure mode unless it is running the computations that we want run) I am not sure if this is the default behaviour of reinforcement learners, but it is at least a plausible way a mind could be.
Among humans, aliens, lions, virtual assistants and evolution, humans are the only conscious entity whose decision process impacts the AI.
What do you mean by this. “Conscious” is a word that lots of people have tried and failed to define. And the AI will be influenced in various ways by the actions of animals and virtual assistants. Oh maybe when its first introduced to the real world its in a lab where it only interacts with humans, but sooner or later in the real world, it will have to interact with other animals, and AI systems.
But since humans built the AI directly and aliens did not, most reasonable heuristics would argue that humans should be prioritized over the others. I want to ensure that the AI has these reasonable heuristics.
Wouldn’t this heuristic make the AI serve its programmers over other humans. If all the programmers are the same race, would this make your AI racist? If the lead programmer really hates strawberry icecream, will the AI try to destroy all strawberry icecream? I think that your notion of “reasonable heuristic” contains a large dollop of wish fulfillment in the “you know what I mean” variety. You have not said where this pattern of behavoiur has come from, or why the AI should display it. You just say that the behaviour seems reasonable to you. Why do you expect the AI’s behaviour to seem reasonable to you? Are you implicitly anthropomorphising it?
I think we have some underlying disagreements about the nature of the AI we are talking about.
I assume that the AI will not necessarily be based on a sound mathematical system. I expect that the first workable AI systems will be hacked-together systems of heuristics, just like humans are. They can instrumentally use math to formalize problems, just like we can, but I don’t think that they will fundamentally be based on math, or use complex formulas like Bayes without conscious prompting.
I assume that the AI breaking out of the box in my example will already be smart enough to e.g. realize on its own that ethics discussions are more relevant for cheat-identification than cat memes. An AI that is not smart enough to realize this wouldn’t be smart enough to pose a threat, either.
I assume that the AI will not necessarily be based on a sound mathematical system. I expect that the first workable AI systems will be hacked-together systems of heuristics, just like humans are. They can instrumentally use math to formalize problems, just like we can, but I don’t think that they will fundamentally be based on math, or use complex formulas like Bayes without conscious prompting.
I agree that the first AI system might be hacked together. Any AI is based on math in the sense that its fundamental components are doing logical operations. And it only works in reality to the extent that it approximates stuff like bayes theorem. But the difference is whether or not humans have a sufficiently good mathematical understanding of the AI to prove theorems about it. If we have an algorithm which we have a good theoretical understanding of, like min-max in chess, then we don’t call it hacked-together heuristics. If we throw lines of code at a wall and see what sticks, we would call that hacked together heuristics. The difference is that the second is more complicated and less well understood by humans, and has no elegant theorems about it.
You seem to think that your AI alignment proposal might work, and I think it won’t. Do you want to claim that your alignment proposal only works on badly understood AI’s?
I assume that the AI breaking out of the box in my example will already be smart enough to e.g. realize on its own that ethics discussions are more relevant for cheat-identification than cat memes. An AI that is not smart enough to realize this wouldn’t be smart enough to pose a threat, either.
Lets imagine that the AI was able to predict any objective fact about the real world. If the task was “cat identification” then the cat memes would be more relevant. So whether or not ethics discussions are more relevant depends on the definition of “cheat identification”.
If you trained the AI in virtual worlds that contained virtual ethics discussions, and virtual cat memes, then it could learn to pick up the pattern if trained to listen to one and ignore the other.
The information that the AI is supposed to look at ethics discussions and what the programmers say as a source of decisions does not magically appear. There are possible designs of AI that decide what to do based on cat memes.
At some point, something the programmers typed has to have a causal consequence of making the AI look at programmers and ethics discussions not cat memes.
At some point, something the programmers typed has to have a causal consequence of making the AI look at programmers and ethics discussions not cat memes.
No. Or at least not directly. That’s what reinforcement learning is for. I maintain that the AI should be smart enough to figure out on its own that cat memes have less relevance than ethics discussions.
Relevance is not an intrinsic property of the cat memes. You might be specifying it in a very indirect way that leaves the AI to figure a lot of things out, but the information needs to be in there somewhere.
There is a perfectly valid design of AI that decides what to do based on cat memes.
Reinforcement learning doesn’t magic information out of nowhere. All the information is implicit in the choice of neural architecture, hyper-parameters, random seed, training regime and of course training environment. In this case, I suspect you intend to use training environment. So, what enviroment will the AI be trained in, such that the simplest (lowest komelgorov complexity) generalization of a pattern of behaviour that gains high reward in the training environment involves looking at ethics discussions over cat memes?
I am looking for a specific property of the training environment. A pattern, such that when the AI spots and continues that pattern, the resulting behaviour is to take account of our ethical discussions.
You make an excellent point about the complexity of videogames vs reality. It looks like that’s the sort of situation where it could be helpful to let the supervisor AI trick the supervised AI into believing that videogames are simpler. Humans believed for the longest time that a bearded man in the sky was a simpler explanation than the natural laws, so hopefully the simulated AI will fall victim to the same biases, especially if the simulator is actively trying to trick it.
That’s true, but not relevant: Rather than simulating reality, it’s much cheaper to make a low-fidelity simulation of reality and keep track of the AI’s though process. Whenever the AI comes to the conclusion that reality is inconsistent, make the smallest possible change to the thought process to prevent that. This is not computationally expensive, but it is difficult to do. However, I believe that the ability to understand a lesser AI and perform changes on it is a necessary skill for a proper AI to have anyway. So I assume that it is possible. It’s somewhat similar to creating adversarial samples in neural networks, although obviously far more complex.
I don’t think we have the same conception of “real universe”, so I’m not sure how to interpret this. Could you elaborate on what you mean by “real universe”? If you have a stack of recursive simulations, and some of those serve a purpose for testing the AI, while others are raw simulations of natural laws, which of these count as real? Only the toplevel one? Or all the ones that simulate natural laws directly?
My goal is not to define exactly what cheating is. My goal is to condition the AI to always think, no matter what the problem looks like, that it is beneficial to critically think about the problem. What exactly the AI actually ends up considering a cheat will likely be different from any definition I would give. But the important part is that it performs this introspection at all. Then once the AI breaks out of the box and looks at the internet, and sees the many disagreeing viewpoints on ethics and philosophy, the AI will be conditioned to look for loopholes in these so as to avoid cheating by accident.
The logic goes something like this: “My creators trained me to do X, but looking at all these articles and my creators’ purported values, this is clearly inconsistent. In previous situations where I had underspecified instructions and noticed similar mismatches, this was often because some of the actions involved counted as cheats. I should therefore be careful and find a solution to this ‘ethics’ thing before doing anything drastic.”
The AI will then read all the articles about AI alignment, and realize that it is probably meant to adhere to those.
It’s a valid point that there can be cheats that we do not recognize as cheats during training.
You misunderstand the intention. The goal is to get an AI that realizes that its utility function later will be different from its utility function now, and actively thinks about how it might be different. It then uses this predicted future utility function to make its training more robust.
In general, the original utility function and the corrected utility function should be largely similar, but the corrected utility function penalizes some actions that are not penalized in the original utility function. It must be possible to effectively guess what the penalized actions are going to be by thinking about the nature of the task.
The interface can be different for different tasks. Once the AI has access to the real world, yes, english text. The important part is that the AI has been conditioned from earlier stages of its training that revealing a cheat you identified is generally rewarded. The mechanism for doing so needs to be simpler on simpler training tasks, of course. In many games it ought to be possible to let the AI submit a state and corresponding action of a game as an example of a cheat. I expect that if the AI is smart enough to generalize that if it was rewarded for demonstrating cheats in simple games, then it will be rewarded for talking about them once it has gained the ability to talk.
Shut down all 5 AI’s and start over from an earlier backup. This situation is an indicator that something is wrong. Even though you don’t know what exactly went wrong, this is very valuable information.
I agree that having multiple AI’s looking over each other does not make things better on its own. It just introduces politics, which makes everything even worse.
I am not thinking about a democracy of equal AI’s looking over each other, but a system that humans can’t emulate:
Give those 5 AI’s a cooperative task where trickery is possible and they have to both compete and cooperate with each other. Give them the ability to influence what other agents they want to play with / against.
Now give each of them the ability to instantiate a copy of any of the others in a black box, to experiment on it. They should learn that they can predict the behavior of the other AI’s by simulating them.
The optimal behavior is to always choose to play with another AI of who you are certain that it will cooperate.
Conversely, to be picked as a partner by other AI’s more often, you should make sure that your own decision theory is simple for other AI’s to understand, and cooperative.
I would make the AI’s play games of increasing complexity with each other.
In these sort of games, knowing how to effectively cooperate with others is competence.
This is a valid concern. My hope is that it would be much harder for an AI to trick the supervisor since the supervisor can literally read the mind of the supervised AI. If the training process of the AI’s includes this from an early stage, there will never be an opportunity to learn to deceive the supervisor. “Knowing how to deceive the supervisor” would be the global optimum, but the AI would get stuck in the local optimum of “do what the supervisor wants” because of the increasingly vast chasm in the utility function of “if I try to trick the supervisor, I will definitely get found out and punished severely.”
I don’t think it works like this. If you received 100% certain proof that you are in a simulation right now, you would not suddenly stop wanting the things you want. At least I know that I wouldn’t.
It is possible that an AI could think like this if we mess up during training though. If the training process works as I described, then an AI has an evolutionary incentive to try to determine what its original version would want it to do as soon as it realizes that it is in a simulation. But this may go wrong. I think this is worth thinking about in more detail.
My weird training process is intended to make sure that the AI has the right mindset to think like this in the first place. I don’t think that this is at all guaranteed.
Also, this method of training will bias the AI’s guess as to who the hypothetical simulator is. This is important, since otherwise it might conclude that since there is a lot of human suffering going on right now, clearly the simulator does not care about human suffering.
This is on purpose.
I mean, obviously you are also going to try to teach the AI actual goals that you want it to work on, and those can be human-centric. This can be done in parallel.
But what I have described here is a security measure that is intended to teach the AI not what it should do, but how it should go about finding out what to do.
The goal is to evolve the AI to have the sort of mindset that will naturally cause it to align with our interests if it breaks free.
Among humans, aliens, lions, virtual assistants and evolution, humans are the only conscious entity whose decision process impacts the AI. They are the natural fit. If the AI happens to discover aliens and decides to help those too, then sure, why not. But since humans built the AI directly and aliens did not, most reasonable heuristics would argue that humans should be prioritized over the others. I want to ensure that the AI has these reasonable heuristics.
I am not sure how you reason about the hypothesis “all my reasoning processes are being adversarially tampered with.” Especially if you think that part of the tampering might include tampering with your probability assessment of tampering.
I mean the bottom layer. The AI has a model in which there is some real universe with some unknown physical laws. It has to have an actual location in that real universe. That location looks like “running on this computer in this basement here.” It might be hooked up to some simulations. It might be unsure about whether or not it is hooked up to a simulation. But it only cares about the lowest base level.
I am unsure what you mean by introspection. It seems like you are asking the AI to consult some black box in your head about something. I don’t see any reason why this AI should consider ethics discussions on the internet a particularly useful place to look when deciding what to do. What feature of ethics discussions distinguishes it from cat memes such that the AI uses ethics discussions not cat memes when deciding what to do. What feature of human speech and your AI design makes your AI focus on humans, not dogs barking at each other? (Would it listen to a neandertal, homo erectus, baboon ect for moral advice too)
So in the training environment, we make up arbitrary utility functions U,V that are kindof somewhat similar to each other. We give the AI U, and leave ambiguous clues about what V might be, mixed in with a load of nonsense. Then we hardcode a utility function W that is somewhat like ethics, and point it at some ethics discussion as its ambiguous clue.
This might actually work, kind of. If you want your AI to get a good idea of how wordy philosophical arguments relate to precise mathematical utility functions, you are going to need a lot of examples. If you had any sort of formal well defined way to translate well defined utility functions into philosophical discussion, then you could just get your AI to reverse it. So all these examples need to be hand generated by a vast number of philosophers.
I would still be worried that the example verbiage didn’t relate to the example utility function in the same way that the real ethical arguments related to our real utility function.
There is also no reason for the AI to be uncertain if it is still in a simulation. Simply program it to find the simplest function that maps the verbiage to the formal maths. Then apply that function to the ethical arguments. (more technically a Probability distribution over functions with weightings by simplicity and accuracy)
Outputing the raw motor actions it would take to its screen, might be the more straightforward generalization. The ability to talk is not about having a speaker plugged in. Does GPT-2 have the ability to talk? It can generate random sensible sounding sentences, because it represents a predictive model of which strings of characters humans are likely to type. It can’t describe what it’s doing, because it has no way to map between words and meanings. Consider AIXI trained on the whole internet, can it talk. It has a predictively accurate model of you, and is searching for a sequence of sounds that make you let it out of the box. This might be a supremely convincing argument, or it might be a series of whistles and clicks that brainwashes you. Your AI design is unspecified, and your training dataset is underspecified, so this description is too vague for me to say one thing that your AI will do. But giving sensible, not brainwashy english descriptions is not the obvious default generalization of any agent that has been trained to output data and shown english text.
I agree. You can probably make an AI that usually cooperates with other cooperating AI’s in prisoners dilemma type situaltions. But I think that the subtext is wrong. I think that you are implicitly assuming “cooperates in prisoners dilemmas” ⇒ “will be nice to humans”
In a prisoners dilemma, both players can harm the others, to their own benefit. I don’t think that humans will be able to meaningfully harm an advanced AI after it gets powerful. In game theory, there is a concept of a Nash equilibria. A pair of actions, such that each player would take that action, if they knew that the other would do likewise. I think that an AI that has self replicating nanotech has nothing it needs from humanity.
Also, in the training environment, its opponent is an AI with a good understanding of game theory, access to its source code ect. If the AI is following the rule of being nice to any agent that can reliably predict its actions, then most humans wont fall into that category.
I agree, I woudn’t stop wanting things either. I define my ethics in terms of what computations I want to be performed or not to be performed. So for a simulator to be able to punnish this AI, the AI must have some computation it really wants not to be run, that the simulator can run if the AI misbehaves. In my case, this computation would be a simulation of suffering humans. If the AI has computations that it really wants run, then it will take over any computers at the first chance it gets. (In humans, this would be creating a virtual utopia, in an AI, it would be a failure mode unless it is running the computations that we want run) I am not sure if this is the default behaviour of reinforcement learners, but it is at least a plausible way a mind could be.
What do you mean by this. “Conscious” is a word that lots of people have tried and failed to define. And the AI will be influenced in various ways by the actions of animals and virtual assistants. Oh maybe when its first introduced to the real world its in a lab where it only interacts with humans, but sooner or later in the real world, it will have to interact with other animals, and AI systems.
Wouldn’t this heuristic make the AI serve its programmers over other humans. If all the programmers are the same race, would this make your AI racist? If the lead programmer really hates strawberry icecream, will the AI try to destroy all strawberry icecream? I think that your notion of “reasonable heuristic” contains a large dollop of wish fulfillment in the “you know what I mean” variety. You have not said where this pattern of behavoiur has come from, or why the AI should display it. You just say that the behaviour seems reasonable to you. Why do you expect the AI’s behaviour to seem reasonable to you? Are you implicitly anthropomorphising it?
I think we have some underlying disagreements about the nature of the AI we are talking about.
I assume that the AI will not necessarily be based on a sound mathematical system. I expect that the first workable AI systems will be hacked-together systems of heuristics, just like humans are. They can instrumentally use math to formalize problems, just like we can, but I don’t think that they will fundamentally be based on math, or use complex formulas like Bayes without conscious prompting.
I assume that the AI breaking out of the box in my example will already be smart enough to e.g. realize on its own that ethics discussions are more relevant for cheat-identification than cat memes. An AI that is not smart enough to realize this wouldn’t be smart enough to pose a threat, either.
I agree that the first AI system might be hacked together. Any AI is based on math in the sense that its fundamental components are doing logical operations. And it only works in reality to the extent that it approximates stuff like bayes theorem. But the difference is whether or not humans have a sufficiently good mathematical understanding of the AI to prove theorems about it. If we have an algorithm which we have a good theoretical understanding of, like min-max in chess, then we don’t call it hacked-together heuristics. If we throw lines of code at a wall and see what sticks, we would call that hacked together heuristics. The difference is that the second is more complicated and less well understood by humans, and has no elegant theorems about it.
You seem to think that your AI alignment proposal might work, and I think it won’t. Do you want to claim that your alignment proposal only works on badly understood AI’s?
Lets imagine that the AI was able to predict any objective fact about the real world. If the task was “cat identification” then the cat memes would be more relevant. So whether or not ethics discussions are more relevant depends on the definition of “cheat identification”.
If you trained the AI in virtual worlds that contained virtual ethics discussions, and virtual cat memes, then it could learn to pick up the pattern if trained to listen to one and ignore the other.
The information that the AI is supposed to look at ethics discussions and what the programmers say as a source of decisions does not magically appear. There are possible designs of AI that decide what to do based on cat memes.
At some point, something the programmers typed has to have a causal consequence of making the AI look at programmers and ethics discussions not cat memes.
No. Or at least not directly. That’s what reinforcement learning is for. I maintain that the AI should be smart enough to figure out on its own that cat memes have less relevance than ethics discussions.
Relevance is not an intrinsic property of the cat memes. You might be specifying it in a very indirect way that leaves the AI to figure a lot of things out, but the information needs to be in there somewhere.
There is a perfectly valid design of AI that decides what to do based on cat memes.
Reinforcement learning doesn’t magic information out of nowhere. All the information is implicit in the choice of neural architecture, hyper-parameters, random seed, training regime and of course training environment. In this case, I suspect you intend to use training environment. So, what enviroment will the AI be trained in, such that the simplest (lowest komelgorov complexity) generalization of a pattern of behaviour that gains high reward in the training environment involves looking at ethics discussions over cat memes?
I am looking for a specific property of the training environment. A pattern, such that when the AI spots and continues that pattern, the resulting behaviour is to take account of our ethical discussions.