I agree with you that “magical alignment” is implausible. But “relative alignment” presents its risks too, which I have discussed at large in AGI deployment as an act of aggression. The essential problem, I think, is that if you postulate the kind of self-enhancing AGI that basically takes control of the future (if that’s not possible at all for reasons of diminishing returns, then the category of the problem completely shifts), that’s something whose danger doesn’t just lie in it being out of control. It’s inherently dangerous, because it hinges all of humanity’s future on a single pivot. I suppose that doesn’t have to result in extinction, but there are still some really bad almost guaranteed outcomes from it.
I think essentially for a lot of people this is a “whoever wins, we lose” situation. There’s a handful of people, the ones in position to actually control the nascent AI and give it their values, who might have a shot at winning it, and they are the ones pushing harder for this to happen. But I’m not among them, as the vast majority of humanity, so I’m not really inclined to support their enterprise at this moment. AI that improves everyone’s lives requires a level of democratic oversight in its alignment and deployment that right now is just not there.
I agree with you that “magical alignment” is implausible. But “relative alignment” presents its risks too, which I have discussed at large in AGI deployment as an act of aggression. The essential problem, I think, is that if you postulate the kind of self-enhancing AGI that basically takes control of the future (if that’s not possible at all for reasons of diminishing returns, then the category of the problem completely shifts), that’s something whose danger doesn’t just lie in it being out of control. It’s inherently dangerous, because it hinges all of humanity’s future on a single pivot. I suppose that doesn’t have to result in extinction, but there are still some really bad almost guaranteed outcomes from it.
I think essentially for a lot of people this is a “whoever wins, we lose” situation. There’s a handful of people, the ones in position to actually control the nascent AI and give it their values, who might have a shot at winning it, and they are the ones pushing harder for this to happen. But I’m not among them, as the vast majority of humanity, so I’m not really inclined to support their enterprise at this moment. AI that improves everyone’s lives requires a level of democratic oversight in its alignment and deployment that right now is just not there.