I think a more likely thing we’d want to stick around to do in that world is ‘try to accelerate humanity to AGI ASAP’. “Sufficiently advanced AGI converges to human-friendly values” is weaker than “AGI will just have human-friendly values by default”.
I think a more likely thing we’d want to stick around to do in that world is ‘try to accelerate humanity to AGI ASAP’. “Sufficiently advanced AGI converges to human-friendly values” is weaker than “AGI will just have human-friendly values by default”.