I see where you are coming from, but I don’t think comparing an adversarial AI/human interaction with a human/human interaction is fruitful. Even a “stupid” AI thinks differently than a human would, in the way that it considers options no human ever would think of or take seriously. Self-cloning and not having to consider losses is another approach humans have no luxury of.
I would start by asking a question that no one at MIRI or elsewhere seems to be asking:
What might an irate e-chimp do if their human handler denied it a banana?
(I.e. what are the dangers of gain-of-function research in sub-human A[G]I)
A stupid AI that can generate from thin air things that have both useful predictive power and can’t be thought of by humans, AND that can reliably employ the fruits of these ideas without humans being suspicious or having a defense...isn’t that stupid. This AI is now a genius.
What might an irate e-chimp do if their human handler denied it a banana?
Who cares? For one, if we’re talking about an AI and not a chimp em this is an obvious engineering failure to create something with all the flaws of an evolved entity with motivational pressures extraneous and harmful to users. Or in other words this is a (very) light alignment problem that can be foreseen and fixed.
I see where you are coming from, but I don’t think comparing an adversarial AI/human interaction with a human/human interaction is fruitful. Even a “stupid” AI thinks differently than a human would, in the way that it considers options no human ever would think of or take seriously. Self-cloning and not having to consider losses is another approach humans have no luxury of.
I would start by asking a question that no one at MIRI or elsewhere seems to be asking:
What might an irate e-chimp do if their human handler denied it a banana?
(I.e. what are the dangers of gain-of-function research in sub-human A[G]I)
A stupid AI that can generate from thin air things that have both useful predictive power and can’t be thought of by humans, AND that can reliably employ the fruits of these ideas without humans being suspicious or having a defense...isn’t that stupid. This AI is now a genius.
Who cares? For one, if we’re talking about an AI and not a chimp em this is an obvious engineering failure to create something with all the flaws of an evolved entity with motivational pressures extraneous and harmful to users. Or in other words this is a (very) light alignment problem that can be foreseen and fixed.