I think one of the key points here is that most possible minds/intelligences are alien, outside the human distribution. See https://www.lesswrong.com/posts/tnWRXkcDi5Tw9rzXw/the-design-space-of-minds-in-general for art of EY’s (15 yr old) discussion on LW of this. Humans were produced by a specific historical evolutionary process constrained by the amount of selection pressure applied to our genes, and the need for humans to all be similar enough to each other to form a single species in each generation, among other things. AI is not that, it will be designed and trained under very different processes even if we don’t know what all of those processes will end up being. This doesn’t mean an AI made by humans will be anything like a random selection from the set of all possible minds, but in any case the alignment problem is largely that we don’t know how to reliably steer what kind of alien mind we get in desired directions.
I think one of the key points here is that most possible minds/intelligences are alien, outside the human distribution. See https://www.lesswrong.com/posts/tnWRXkcDi5Tw9rzXw/the-design-space-of-minds-in-general for art of EY’s (15 yr old) discussion on LW of this. Humans were produced by a specific historical evolutionary process constrained by the amount of selection pressure applied to our genes, and the need for humans to all be similar enough to each other to form a single species in each generation, among other things. AI is not that, it will be designed and trained under very different processes even if we don’t know what all of those processes will end up being. This doesn’t mean an AI made by humans will be anything like a random selection from the set of all possible minds, but in any case the alignment problem is largely that we don’t know how to reliably steer what kind of alien mind we get in desired directions.