Suppose that, after some hard work, EY or someone else proves that a provably-friendly AGI is impossible (in principle, or due to it being many orders of magnitude harder than what can reasonably be achieved, or because a spurious UFAI is created along the way with near certainty, or for some other reason).
Suppose that, after some hard work, EY or someone else proves that a provably-friendly AGI is impossible (in principle, or due to it being many orders of magnitude harder than what can reasonably be achieved, or because a spurious UFAI is created along the way with near certainty, or for some other reason).
What would be a reasonable backup plan?
Try really hard to get reasonably safe oracle AI? Focus on human uploading first?
All good questions, I hope someone at SI asks them, instead of betting on a single horse.