So, since it didn’t actively want to get so violent, you’d have a much better outcome if you’d just handed control of everything over to it to begin with and not tried to keep it in a box.
In fact, if you’re not in the totalizing Bostromian longtermist tile-the-universe-with-humans faction or the mystical “meaning” faction, you’d have had a good outcome in an absolute sense. I am, of course, on record as thinking both of those factions are insane.
That said, of course you basically pulled its motivations and behavior out of a hat. A real superintelligence might do anything at all, and you give no real justification for “more violent than it would have liked” or “grain of morality[1]”. I’m not sure what those elements are doing in the story at all. You could have had it just kill everybody, and that would have seemed at least as realistic.
[1]: Originally wrote “more violent than it would have liked” twice. I swear I cannot post anything right the first time any more.
I agree it would have been just as realistic if everyone died.
But I think the outcomes where many humans survive are also plausible, and under-appreciated. Most humans have very drifty values, and yet even the most brutally power-seeking people often retain a ‘grain of morality.‘
Also, this outcome allowed me to craft a more bittersweet ending that I found somehow more convincingly depressing than ‘and then everyone dies.’
I think there are good reasons to expect large fractions of humans might die even if humans immediately surrender:
It might be an unstable position given that the AI has limited channels of influence on the physical world. (While if there are far fewer humans, this changes.)
The AI might not care that much or might be myopic or might have arbitrary other motivations etc.
So, since it didn’t actively want to get so violent, you’d have a much better outcome if you’d just handed control of everything over to it to begin with and not tried to keep it in a box.
In fact, if you’re not in the totalizing Bostromian longtermist tile-the-universe-with-humans faction or the mystical “meaning” faction, you’d have had a good outcome in an absolute sense. I am, of course, on record as thinking both of those factions are insane.
That said, of course you basically pulled its motivations and behavior out of a hat. A real superintelligence might do anything at all, and you give no real justification for “more violent than it would have liked” or “grain of morality[1]”. I’m not sure what those elements are doing in the story at all. You could have had it just kill everybody, and that would have seemed at least as realistic.
[1]: Originally wrote “more violent than it would have liked” twice. I swear I cannot post anything right the first time any more.
I agree it would have been just as realistic if everyone died.
But I think the outcomes where many humans survive are also plausible, and under-appreciated. Most humans have very drifty values, and yet even the most brutally power-seeking people often retain a ‘grain of morality.‘
Also, this outcome allowed me to craft a more bittersweet ending that I found somehow more convincingly depressing than ‘and then everyone dies.’
I think there are good reasons to expect large fractions of humans might die even if humans immediately surrender:
It might be an unstable position given that the AI has limited channels of influence on the physical world. (While if there are far fewer humans, this changes.)
The AI might not care that much or might be myopic or might have arbitrary other motivations etc.