Here’s an entirely separate weak argument, improving on your straw man:
AGI will be powerful. Powerful agentic things do whatever they want. People will try to make AGI do what they want. They might succeed or fail. Nobody has tried doing this before, so we have to guess what the odds of their succeeding are. We should notice that they won’t get many second chances because agents want to keep doing what they want to do. And notice that humans have screwed up big projects in surprisingly dumb (in retrospect) ways.
If some people do succeed at making AGI do what they want, they might or might not want things the rest of humanity wants. So we have to estimate the odds that the types of people who will wind up in charge of AGI (not the ones that start out in charge) are “good people”. Do they want things that would be better described as doom or flourishing for humanity? This matters, because they now have AGI which we agree is powerful. It may be powerful enough that a subset of power-hungry people now control the future—quite possibly all of it.
If you look at people who’ve held power historically, the appear to often be pretty damned selfish, and all too often downright sadistic toward their perceived enemies and sometimes toward their own subjects.
I don’t think it’s an improvement to say the same thing with more words. It gives the aura of sophistication without actually improving on the reasoning.
Here’s an entirely separate weak argument, improving on your straw man:
AGI will be powerful. Powerful agentic things do whatever they want. People will try to make AGI do what they want. They might succeed or fail. Nobody has tried doing this before, so we have to guess what the odds of their succeeding are. We should notice that they won’t get many second chances because agents want to keep doing what they want to do. And notice that humans have screwed up big projects in surprisingly dumb (in retrospect) ways.
If some people do succeed at making AGI do what they want, they might or might not want things the rest of humanity wants. So we have to estimate the odds that the types of people who will wind up in charge of AGI (not the ones that start out in charge) are “good people”. Do they want things that would be better described as doom or flourishing for humanity? This matters, because they now have AGI which we agree is powerful. It may be powerful enough that a subset of power-hungry people now control the future—quite possibly all of it.
If you look at people who’ve held power historically, the appear to often be pretty damned selfish, and all too often downright sadistic toward their perceived enemies and sometimes toward their own subjects.
I don’t think it’s an improvement to say the same thing with more words. It gives the aura of sophistication without actually improving on the reasoning.