I don’t think the assumption is highly dubious. You don’t need to believe in the possibility of mind uploading or biological immortality to assume radically transformative changes in the human condition due to advanced AI. The “Neuroscience and Mind” section of Dario Amodei’s (who has a formal background in biophysics) essay attempts to clearly speculate what’ll happen in these areas with the helped of advanced AI (even setting aside that mind uploading is probably “possible in principle”).
Even if some goals are unattainable, AGI could still likely (as Dario speculates) drive radical advancements in areas like health, longevity, and cognitive enhancement. The point isn’t to guarantee specific outcomes, but to recognise that AGI will likely push the boundaries of what we currently believe is possible and transform the world unrecognisably. And we should be mentally reflecting and preparing for that.
I agree that there are significant uncertainties on the specific consequences of AI accelerating bio/medicine R&D, but I think even without buying into Amodei’s specific speculations on life extension, you would still get wildly transformative breakthroughs and unforeseen consequences. I do agree it seems to make sense to be wary of just extrapolating past increases in life expectancy.
Time will tell!