Asking about what reset conditions would avoid this is a bucket error—the rhetorical point is that no such reset is possible; he’s drawing a contrast between normal science and AGI science. I don’t understand why this post and your reply comment are attempting to get into the details of the counterfactual that you and EY both agree is a counterfactual. The whole point is that it can’t happen!
My point is that even if we solve the technical problem of ‘how do we get goals into an AI’, the ‘what values to put in the AI’ problem is also very hard.
So hard that even in the ‘Groundhog Day’ universe, it’s hard.
And yet people just handwave it.
Almost all the survival probability is in ‘we don’t build the AI’, or ‘we’re just wrong about something fundamental’.
Sure, I read that a few years after he wrote it, and it’s still probably the best idea, but even if it’s feasible it needs superintelligent help! So we have to solve the alignment problem to do it.
Asking about what reset conditions would avoid this is a bucket error—the rhetorical point is that no such reset is possible; he’s drawing a contrast between normal science and AGI science. I don’t understand why this post and your reply comment are attempting to get into the details of the counterfactual that you and EY both agree is a counterfactual. The whole point is that it can’t happen!
My point is that even if we solve the technical problem of ‘how do we get goals into an AI’, the ‘what values to put in the AI’ problem is also very hard.
So hard that even in the ‘Groundhog Day’ universe, it’s hard.
And yet people just handwave it.
Almost all the survival probability is in ‘we don’t build the AI’, or ‘we’re just wrong about something fundamental’.
Yes, it is hard. But Eliezer isn’t just handwaving it. Here for example is a 37-page document he wrote on the subject 19 years ago:
https://intelligence.org/files/CEV.pdf
Sure, I read that a few years after he wrote it, and it’s still probably the best idea, but even if it’s feasible it needs superintelligent help! So we have to solve the alignment problem to do it.