Thus in order to arrive at a conclusion of doom, it is not enough to argue that we cannot align AI perfectly.
I am open to being corrected, but I do not recall ever seeing a requirement of “perfect” alignment in the cases made for doom. Eliezer Yudkowsky in “AGI Ruin: A List of Lethalities” only asks for ‘this will not kill literally everyone’.
My impression is that there has been a variety of suggestions about the necessary level of alignment. It is only recently that don’t kill most of humanity has been suggested as a goal and I am not sure that the suggestion was meant to be taken seriously. (Because if you can do that, you can probably do much better; the point of that comment as I understand it was that we aren’t even close to being able to achieve even that goal.)
I am open to being corrected, but I do not recall ever seeing a requirement of “perfect” alignment in the cases made for doom. Eliezer Yudkowsky in “AGI Ruin: A List of Lethalities” only asks for ‘this will not kill literally everyone’.
My impression is that there has been a variety of suggestions about the necessary level of alignment. It is only recently that don’t kill most of humanity has been suggested as a goal and I am not sure that the suggestion was meant to be taken seriously. (Because if you can do that, you can probably do much better; the point of that comment as I understand it was that we aren’t even close to being able to achieve even that goal.)