Yes, that ‘poetry’ explains what extrapolation is, but not why we need to risk it. To my mind, this is the most dangerous aspect of the whole FAI enterprise. Yet we don’t have anything approaching an analysis of a requirements document—instead we get a poetic description of what Eliezer wants, a clarification of what the poetry means, but no explanation of why we should want that. It is presumed to be obvious that extrapolating can only improve things. Well, lets look more closely.
… if we knew more, …
An AI is going to tell us what we would want, if only we knew more. Apparently, there is an assumption here that the AI knows things we don’t. Personally, I worry a bit that an AI will come to believe things that are not true. In fact, I worry about it most when the AI claims to know something that mankind does not know—something dealing with human values. Why do I worry about that? Something someone wrote somewhere presumably. But maybe that is not the kind of superior AI ‘knowledge’ that Eliezer is talking about here.
Knew more: Fred may believe that box A contains the diamond, and say, “I want box A.” Actually, box B contains the diamond, and if Fred knew this fact, he would predictably say, “I want box B.”
And instead of extrapolating, why not just inform Fred where the diamond is? At this point, the explanation becomes bizarre.
If Fred would adamantly refuse to even consider the possibility that box B contains a diamond, while also adamantly refusing to discuss what should happen in the event that he is wrong in this sort of case, and yet Fred would still be indignant and bewildered on finding that box A is empty, Fred’s volition on this problem is muddled.
Am I alone in preferring, in this situation, that the AI not diagnose a ‘muddle’, and instead give Fred box A after offering him the relevant knowledge?
Thought faster: Suppose that your current self wants to use an elaborate system of ropes and sticks to obtain a tasty banana, but if you spent an extra week thinking about the problem, you would predictably see, and prefer, a simple and elegant way to get the banana using only three ropes and a teddy bear.
Again, if the faster thinking allows the AI to serve as an oracle, making suggestions that even our limited minds can appreciate once we hear them, then why should we take the risk of promoting the AI from oracle to king? The AI should tell us things rather than speaking for us.
Were more the people we wished we were: Any given human is inconsistent under reflection. We all have parts of ourselves that we would change if we had the choice, whether minor or major.
When we have a contradiction between a moral intuition and a maxim codifying our system of moral standards there are two ways we can go—we can revise the intuition or we can revise the maxim. It makes me nervous having an AI make the decisions leading to ‘reflective equilibrium’ rather than making those decisions myself. Instead of an extrapolation, I would prefer a dialog leading me to my own choice of equilibrium rather than having a machine pick one for me. Again, my slogan is “Speak to us, don’t speak for us.”
Where our wishes cohere rather than interfere
I’m not sure what to make of this one. Is there a claim here that extrapolation automatically leads to coherence? If so, could we have an argument justifying that claim? Or, is the point that the extrapolation specification has enough ‘free play’ to allow the AI to guide the extrapolation to coherence? Coherence is certainly an important issue. A desideratum? Certainly. A requirement? Maybe. But there are other ways of achieving accommodation without trying to create an unnatural coherence in our diverse species.
These are topics that really need to be discussed in a format other than poetry.
An AI is going to tell us what we would want, if only we knew more. Apparently, there is an assumption here that the AI knows things we don’t. Personally, I worry a bit that an AI will come to believe things that are not true. In fact, I worry about it most when the AI claims to know something that mankind does not know—something dealing with human values. Why do I worry about that? Something someone wrote somewhere presumably. But maybe that is not the kind of superior AI ‘knowledge’ that Eliezer is talking about here.
Rebuttal: Most people in the world believe in a religion that is wrong. (This conclusion holds regardless of which, if any, world religion happens to be true.) Would we want an A.I. that enforces the laws of a false religion because people want the laws of their religion enforced? (Assume that people would agree that the AI shouldn’t enforce the laws of false religions.)
If Fred would adamantly refuse to even consider the possibility that box B contains a diamond, while also adamantly refusing to discuss what should happen in the event that he is wrong in this sort of case, and yet Fred would still be indignant and bewildered on finding that box A is empty, Fred’s volition on this problem is muddled.
Am I alone in preferring, in this situation, that the AI not diagnose a ‘muddle’, and instead give Fred box A after offering him the relevant knowledge?
What if box A actually contains a bomb that explodes when Fred opens it? Should the AI still give Fred the box?
This is the Knew more … Thought faster … Were more the people we wished we were … section of CEV.
Yes, that ‘poetry’ explains what extrapolation is, but not why we need to risk it. To my mind, this is the most dangerous aspect of the whole FAI enterprise. Yet we don’t have anything approaching an analysis of a requirements document—instead we get a poetic description of what Eliezer wants, a clarification of what the poetry means, but no explanation of why we should want that. It is presumed to be obvious that extrapolating can only improve things. Well, lets look more closely.
An AI is going to tell us what we would want, if only we knew more. Apparently, there is an assumption here that the AI knows things we don’t. Personally, I worry a bit that an AI will come to believe things that are not true. In fact, I worry about it most when the AI claims to know something that mankind does not know—something dealing with human values. Why do I worry about that? Something someone wrote somewhere presumably. But maybe that is not the kind of superior AI ‘knowledge’ that Eliezer is talking about here.
And instead of extrapolating, why not just inform Fred where the diamond is? At this point, the explanation becomes bizarre.
Am I alone in preferring, in this situation, that the AI not diagnose a ‘muddle’, and instead give Fred box A after offering him the relevant knowledge?
Again, if the faster thinking allows the AI to serve as an oracle, making suggestions that even our limited minds can appreciate once we hear them, then why should we take the risk of promoting the AI from oracle to king? The AI should tell us things rather than speaking for us.
When we have a contradiction between a moral intuition and a maxim codifying our system of moral standards there are two ways we can go—we can revise the intuition or we can revise the maxim. It makes me nervous having an AI make the decisions leading to ‘reflective equilibrium’ rather than making those decisions myself. Instead of an extrapolation, I would prefer a dialog leading me to my own choice of equilibrium rather than having a machine pick one for me. Again, my slogan is “Speak to us, don’t speak for us.”
I’m not sure what to make of this one. Is there a claim here that extrapolation automatically leads to coherence? If so, could we have an argument justifying that claim? Or, is the point that the extrapolation specification has enough ‘free play’ to allow the AI to guide the extrapolation to coherence? Coherence is certainly an important issue. A desideratum? Certainly. A requirement? Maybe. But there are other ways of achieving accommodation without trying to create an unnatural coherence in our diverse species.
These are topics that really need to be discussed in a format other than poetry.
Rebuttal: Most people in the world believe in a religion that is wrong. (This conclusion holds regardless of which, if any, world religion happens to be true.) Would we want an A.I. that enforces the laws of a false religion because people want the laws of their religion enforced? (Assume that people would agree that the AI shouldn’t enforce the laws of false religions.)
What if box A actually contains a bomb that explodes when Fred opens it? Should the AI still give Fred the box?
As I understand CEV, the hope is that it will, and if it doesn’t, CEV is said to fail. Humanity may not have a CEV.