But isn’t it problematic to start the analysis at “superhuman AGI exists”? Then we need to make assumptions about how that AGI came into being. What are those assumptions, and how robust are they?
I strongly agree with this objection. You might be interested in Comprehensive AI Services, a different story of how AI develops that doesn’t involve a single superintelligent machine, as well as “Prosaic Alignment” and “The case for aligning narrowly superhuman systems”. Right now, I’m working on language model alignment because it seems like a subfield with immediate problems and solutions that could be relevant if we see extreme growth in AI over the next 5-10 years.
But isn’t it problematic to start the analysis at “superhuman AGI exists”? Then we need to make assumptions about how that AGI came into being. What are those assumptions, and how robust are they?
I strongly agree with this objection. You might be interested in Comprehensive AI Services, a different story of how AI develops that doesn’t involve a single superintelligent machine, as well as “Prosaic Alignment” and “The case for aligning narrowly superhuman systems”. Right now, I’m working on language model alignment because it seems like a subfield with immediate problems and solutions that could be relevant if we see extreme growth in AI over the next 5-10 years.