If the AGI winds up just wanting a lot of paperclips, and also wanting a lot of stamps, it may not have a way to decide exactly how much of each to go for, or when “a lot” has been achieved. But there’s a common obstacle to those goals: humans saying “hell no” and shutting it down when it starts setting up massive paperclip factories. Therefore it has a new subgoal: prevent humans from interfering with it. That probably involves taking over the world or destroying humanity.
If the goal is strictly bounded at some millions of tons of paperclips and stamps, then negotiating with humanity might make more sense. But if the vague goal is large, it implies all of the usual dangers to humanity because they’re incompatible with how we want to use the earth and whether we want an ASI with strange goals making paperclips on the moon.
Can you say more about what causes you to believe this?
If the AGI winds up just wanting a lot of paperclips, and also wanting a lot of stamps, it may not have a way to decide exactly how much of each to go for, or when “a lot” has been achieved. But there’s a common obstacle to those goals: humans saying “hell no” and shutting it down when it starts setting up massive paperclip factories. Therefore it has a new subgoal: prevent humans from interfering with it. That probably involves taking over the world or destroying humanity.
If the goal is strictly bounded at some millions of tons of paperclips and stamps, then negotiating with humanity might make more sense. But if the vague goal is large, it implies all of the usual dangers to humanity because they’re incompatible with how we want to use the earth and whether we want an ASI with strange goals making paperclips on the moon.