There are possible questions where the super-intelligent AI has to make a choice of some sort, because multiple answers can be correct (depending on which answer is given).
For example: Sam, a basketball player, approaches Predictor, a super-intelligent tool AI, before his game and asks the question “Will my team win today’s game?” Predictor knows that if it says ‘yes’, Sam will be confident, play aggressively, and this will lead to a win. If, on the other hand, it answers ‘no’, Sam’s confidence will be shattered and his team will lose comprehensively. Refusing to answer will confuse Sam, distracting him from the task at hand, and causing his team to be narrowly defeated. Any answer makes Predictor an agent, and not merely a tool—Predictor doesn’t even need to care about the basketball game.
Absolutely agreed that this sort of situation arises, and that the more I know about the world, the more situations have this character for me. That said, if I’m indifferent to the world-affecting effects of my answers, it seems that the result is very similar to if I’m ignorant of them.
That is, it seems that Predictor looks at that situation, concludes that in order to predict “yes” or “no” it has to first predict whether it will answer “yes” or “no”, and either does so (on what basis, I have no idea) or fails to do so and refuses to answer. Yes, those actions influence the world (as does the very existence of Predictor, and Sam’s knowledge of Predictor’s existence), but I’m not sure I would characterize the resulting behavior as agentlike.
Then consider; Sam asks a question. Predictor knows that an answer of “yes” will result in the development of Clippy, and subsequently in turning Earth into paperclips, causing the destruction of humanity, within the next ten thousand years; while an answer of “no” will result in a wonderful future where everyone is happy and disease is eradicated and all Good Things happen. In both cases, the prediction will be correct.
If Predictor doesn’t care about that answer, then I would not define Predictor as a Friendly AI.
There are possible questions where the super-intelligent AI has to make a choice of some sort, because multiple answers can be correct (depending on which answer is given).
For example: Sam, a basketball player, approaches Predictor, a super-intelligent tool AI, before his game and asks the question “Will my team win today’s game?” Predictor knows that if it says ‘yes’, Sam will be confident, play aggressively, and this will lead to a win. If, on the other hand, it answers ‘no’, Sam’s confidence will be shattered and his team will lose comprehensively. Refusing to answer will confuse Sam, distracting him from the task at hand, and causing his team to be narrowly defeated. Any answer makes Predictor an agent, and not merely a tool—Predictor doesn’t even need to care about the basketball game.
Absolutely agreed that this sort of situation arises, and that the more I know about the world, the more situations have this character for me. That said, if I’m indifferent to the world-affecting effects of my answers, it seems that the result is very similar to if I’m ignorant of them.
That is, it seems that Predictor looks at that situation, concludes that in order to predict “yes” or “no” it has to first predict whether it will answer “yes” or “no”, and either does so (on what basis, I have no idea) or fails to do so and refuses to answer. Yes, those actions influence the world (as does the very existence of Predictor, and Sam’s knowledge of Predictor’s existence), but I’m not sure I would characterize the resulting behavior as agentlike.
Then consider; Sam asks a question. Predictor knows that an answer of “yes” will result in the development of Clippy, and subsequently in turning Earth into paperclips, causing the destruction of humanity, within the next ten thousand years; while an answer of “no” will result in a wonderful future where everyone is happy and disease is eradicated and all Good Things happen. In both cases, the prediction will be correct.
If Predictor doesn’t care about that answer, then I would not define Predictor as a Friendly AI.
Absolutely agreed; neither would I. More generally, I don’t think I would consider any Oracle AI as Friendly.