If there were any possibility of surviving the first AGI built, then it would be nice to have AGI projects promising to do something that wouldn’t look like trying to seize control of the Future for themselves, when, much later (subjectively?), they became able to do something like CEV. I don’t see much evidence that they’re able to think on the level of abstraction that CEV was stated on, though, nor that they’re able to understand the ‘seizing control of the Future’ failure mode that CEV is meant to prevent, and they would not understand why CEV was a solution to the problem while ‘Apple pie and democracy for everyone forever!’ was not a solution to that problem. If at most one AGI project can understand the problem to which CEV is a solution, then it’s not a solution to races between AGI projects. I suppose it could still be a solution to letting one AGI project scale even when incorporating highly intelligent people with some object-level moral disagreements.
Yes, it was an intentional part of the goal.
If there were any possibility of surviving the first AGI built, then it would be nice to have AGI projects promising to do something that wouldn’t look like trying to seize control of the Future for themselves, when, much later (subjectively?), they became able to do something like CEV. I don’t see much evidence that they’re able to think on the level of abstraction that CEV was stated on, though, nor that they’re able to understand the ‘seizing control of the Future’ failure mode that CEV is meant to prevent, and they would not understand why CEV was a solution to the problem while ‘Apple pie and democracy for everyone forever!’ was not a solution to that problem. If at most one AGI project can understand the problem to which CEV is a solution, then it’s not a solution to races between AGI projects. I suppose it could still be a solution to letting one AGI project scale even when incorporating highly intelligent people with some object-level moral disagreements.