My point is that even if we solve the technical problem of ‘how do we get goals into an AI’, the ‘what values to put in the AI’ problem is also very hard.
So hard that even in the ‘Groundhog Day’ universe, it’s hard.
And yet people just handwave it.
Almost all the survival probability is in ‘we don’t build the AI’, or ‘we’re just wrong about something fundamental’.
Sure, I read that a few years after he wrote it, and it’s still probably the best idea, but even if it’s feasible it needs superintelligent help! So we have to solve the alignment problem to do it.
My point is that even if we solve the technical problem of ‘how do we get goals into an AI’, the ‘what values to put in the AI’ problem is also very hard.
So hard that even in the ‘Groundhog Day’ universe, it’s hard.
And yet people just handwave it.
Almost all the survival probability is in ‘we don’t build the AI’, or ‘we’re just wrong about something fundamental’.
Yes, it is hard. But Eliezer isn’t just handwaving it. Here for example is a 37-page document he wrote on the subject 19 years ago:
https://intelligence.org/files/CEV.pdf
Sure, I read that a few years after he wrote it, and it’s still probably the best idea, but even if it’s feasible it needs superintelligent help! So we have to solve the alignment problem to do it.