What about superintelligent agents that don’t have the “wrapper structure”? Should we try not to create any of those, either? Well, maybe.
But the ones with the wrapper structure are worse. Way, way worse.
“The wrapper structure” just means a fixed constant utility function that doesn’t depend on its inputs.
An AGI “like me” might be morally uncertain like I am, persuadable through dialogue like I am, etc.
The vast vast majority of AI’s with value uncertainty, or AI’s that can have their utility function modified by their inputs are not like that. Think a paperclip maximizer, but if it ever sees a walrus wearing a hat, it will start maximizing staples instead. There is a tiny subset on “non-wrapper” AI’s that use the same meta level procedure to get from observations and moral arguments to values as humans. There are a tiny subset of wrapper AI’s that have the right value function.
As I see it, AI’s with the wrapper structure are like nukes with the spherically symmetric structure. Probably about as dangerous as the (non-wrapper AI/ asymmetric nuke) but much easier to mathematically analyse. If a nuke is sufficiently misshapen, sometimes it will just fizzle rather than exploding outright. If an AI is sufficiently self defeating, sometimes it will just fizzle.
Except of course that the green section is a tiny subset you aren’t going to hit by chance.
“The wrapper structure” just means a fixed constant utility function that doesn’t depend on its inputs.
The vast vast majority of AI’s with value uncertainty, or AI’s that can have their utility function modified by their inputs are not like that. Think a paperclip maximizer, but if it ever sees a walrus wearing a hat, it will start maximizing staples instead. There is a tiny subset on “non-wrapper” AI’s that use the same meta level procedure to get from observations and moral arguments to values as humans. There are a tiny subset of wrapper AI’s that have the right value function.
As I see it, AI’s with the wrapper structure are like nukes with the spherically symmetric structure. Probably about as dangerous as the (non-wrapper AI/ asymmetric nuke) but much easier to mathematically analyse. If a nuke is sufficiently misshapen, sometimes it will just fizzle rather than exploding outright. If an AI is sufficiently self defeating, sometimes it will just fizzle.
Except of course that the green section is a tiny subset you aren’t going to hit by chance.