I notice one of the big reasons that people view misalignment of AIs as not a threat is they view the human-AI gap like the gap between humans and corporations, where existential risk is low or none.
The hardness of the alignment problem is basically even just one order of magnitude difference in intelligence is essentially the difference between humans and the smarter animals like horses, not the difference between human beings and corporations. And very rarely does the more intelligent entity not hurt or kill the less-intelligent entity, and power differentials this large go very badly for the less intelligent side. Thus the default outcome of misaligned AI is catastrophe or extinction.
power differentials this large go very badly for the less intelligent side
With a bit of sympathy/compassion and cosmic wealth, this doesn’t seem so inevitable. The question is probability of settling on that bit of sympathy, and whether it sparks before or after some less well-considered disaster.
I notice one of the big reasons that people view misalignment of AIs as not a threat is they view the human-AI gap like the gap between humans and corporations, where existential risk is low or none.
The hardness of the alignment problem is basically even just one order of magnitude difference in intelligence is essentially the difference between humans and the smarter animals like horses, not the difference between human beings and corporations. And very rarely does the more intelligent entity not hurt or kill the less-intelligent entity, and power differentials this large go very badly for the less intelligent side. Thus the default outcome of misaligned AI is catastrophe or extinction.
With a bit of sympathy/compassion and cosmic wealth, this doesn’t seem so inevitable. The question is probability of settling on that bit of sympathy, and whether it sparks before or after some less well-considered disaster.