If you play taboo with the word “goals” I think the argument may be dissolved.
My laptop doesn’t have a “goal” of satisfying my desire to read LessWrong. I simply open the web browser and type in the URL, initiating a basically deterministic process which the computer merely executes. No need to imbue it with goals at all.
Except now my browser is smart enough to auto-fill the LessWrong URL after just a couple of letters. Is that goal-directed behavior? I think we’re already at the point of hairsplitting semantic distinctions and we’re talking about web browsers, not advanced AI.
Likewise, it isn’t material whether an advanced predictor/optimizer has goals, what is relevant is that it will follow its programming when that programming tells it to “tell me the answer.” If it needs more information to tell you the answer, it will get it, and it won’t worry about how it gets it.
I think your taboo wasn’t strong enough and you allowed some leftover essence of anthropomorphic “goaliness” to pollute your argument.
When you talk about an “advanced optimizer” that “needs more information” to do something and goes out there to “get it”, that presupposes a model of AIs that I consider wrong (or maybe too early to talk about). If the AI’s code consists of navigating chess position trees, it won’t smash you in the face with a rook in order to win, no matter how strongly it “wants” to win or how much “optimization power” it possesses. If an AI believes with 100% probability that its Game of Life universe is the only one that exists, it won’t set out to conquer ours. AIXI is the only rigorously formulated dangerous AI that I know of, its close cousin Solomonoff Induction is safe, both these conclusions are easy, and neither requires CEV.
ETA: if someone gets a bright new idea for a general AI, of course they’re still obliged to ensure safety. I’m just saying that it may be easy to demonstrate for some AI designs.
If you play taboo with the word “goals” I think the argument may be dissolved.
My laptop doesn’t have a “goal” of satisfying my desire to read LessWrong. I simply open the web browser and type in the URL, initiating a basically deterministic process which the computer merely executes. No need to imbue it with goals at all.
Except now my browser is smart enough to auto-fill the LessWrong URL after just a couple of letters. Is that goal-directed behavior? I think we’re already at the point of hairsplitting semantic distinctions and we’re talking about web browsers, not advanced AI.
Likewise, it isn’t material whether an advanced predictor/optimizer has goals, what is relevant is that it will follow its programming when that programming tells it to “tell me the answer.” If it needs more information to tell you the answer, it will get it, and it won’t worry about how it gets it.
I think your taboo wasn’t strong enough and you allowed some leftover essence of anthropomorphic “goaliness” to pollute your argument.
When you talk about an “advanced optimizer” that “needs more information” to do something and goes out there to “get it”, that presupposes a model of AIs that I consider wrong (or maybe too early to talk about). If the AI’s code consists of navigating chess position trees, it won’t smash you in the face with a rook in order to win, no matter how strongly it “wants” to win or how much “optimization power” it possesses. If an AI believes with 100% probability that its Game of Life universe is the only one that exists, it won’t set out to conquer ours. AIXI is the only rigorously formulated dangerous AI that I know of, its close cousin Solomonoff Induction is safe, both these conclusions are easy, and neither requires CEV.
ETA: if someone gets a bright new idea for a general AI, of course they’re still obliged to ensure safety. I’m just saying that it may be easy to demonstrate for some AI designs.