Let’s go with: acquire a large collection of bananas.
What are going to be some priorities for you in the meantime while you’re building your giant pile of bananas?
Don’t die, because you can’t build your pile if you’re dead.
Don’t let someone reach into your brain and change what you want, because the banana pile will stop growing if you stop building it.
Acquire power.
Make yourself smarter and more knowledgeable, for maximum bananas.
If humanity slows you down instead of helping you, kill them.
You can satisfy almost no goal if you’re switched off.
You might think, “Can the AGI not just be like a smart computer that does stuff without wanting anything, like the AI’s in my phone or a calculator?” Sadly no.
If a tool-AI is smart enough, “Make me into an agent-AI,” is the first idea it will suggest for almost any goal. “You want bananas? Easy, the best way for you to get bananas is to make me into a banana-maximizer, because I’m a genius with tonnes of ideas, and I can take over the world to get bananas!” And if the AI has any power, it will do that to itself.
Tool-AI’s basically are agent-AI’s, they’re just dumber.
Think of a random goal for yourself.
Let’s go with: acquire a large collection of bananas.
What are going to be some priorities for you in the meantime while you’re building your giant pile of bananas?
Don’t die, because you can’t build your pile if you’re dead.
Don’t let someone reach into your brain and change what you want, because the banana pile will stop growing if you stop building it.
Acquire power.
Make yourself smarter and more knowledgeable, for maximum bananas.
If humanity slows you down instead of helping you, kill them.
You can satisfy almost no goal if you’re switched off.
You might think, “Can the AGI not just be like a smart computer that does stuff without wanting anything, like the AI’s in my phone or a calculator?” Sadly no.
If a tool-AI is smart enough, “Make me into an agent-AI,” is the first idea it will suggest for almost any goal. “You want bananas? Easy, the best way for you to get bananas is to make me into a banana-maximizer, because I’m a genius with tonnes of ideas, and I can take over the world to get bananas!” And if the AI has any power, it will do that to itself.
Tool-AI’s basically are agent-AI’s, they’re just dumber.