I think your taboo wasn’t strong enough and you allowed some leftover essence of anthropomorphic “goaliness” to pollute your argument.
When you talk about an “advanced optimizer” that “needs more information” to do something and goes out there to “get it”, that presupposes a model of AIs that I consider wrong (or maybe too early to talk about). If the AI’s code consists of navigating chess position trees, it won’t smash you in the face with a rook in order to win, no matter how strongly it “wants” to win or how much “optimization power” it possesses. If an AI believes with 100% probability that its Game of Life universe is the only one that exists, it won’t set out to conquer ours. AIXI is the only rigorously formulated dangerous AI that I know of, its close cousin Solomonoff Induction is safe, both these conclusions are easy, and neither requires CEV.
ETA: if someone gets a bright new idea for a general AI, of course they’re still obliged to ensure safety. I’m just saying that it may be easy to demonstrate for some AI designs.
I think your taboo wasn’t strong enough and you allowed some leftover essence of anthropomorphic “goaliness” to pollute your argument.
When you talk about an “advanced optimizer” that “needs more information” to do something and goes out there to “get it”, that presupposes a model of AIs that I consider wrong (or maybe too early to talk about). If the AI’s code consists of navigating chess position trees, it won’t smash you in the face with a rook in order to win, no matter how strongly it “wants” to win or how much “optimization power” it possesses. If an AI believes with 100% probability that its Game of Life universe is the only one that exists, it won’t set out to conquer ours. AIXI is the only rigorously formulated dangerous AI that I know of, its close cousin Solomonoff Induction is safe, both these conclusions are easy, and neither requires CEV.
ETA: if someone gets a bright new idea for a general AI, of course they’re still obliged to ensure safety. I’m just saying that it may be easy to demonstrate for some AI designs.