At the end of this, I’m going to try to argue that something like CEV is still justified. Before I started thinking it through I was hoping that taking an eliminativist view of preferences to its conclusion would help tie up the loopholes in CEV, and so far it hasn’t done that for me, but it hasn’t made it any harder either.
CEV has worse problems that worries about convergence. The big one is that it’s such a difficult thing to implement that any AI capable of doing so has already crossed the threshold of extremely dangerous transhuman capability, and there’s no real solution to how to regulate its behavior while it’s in the process of working on the extrapolation. It could very well turn the planet into computronium before it gets a satisfactory implementation, by which point it doesn’t much matter what result it arrives at.
Even if you’re the type who thinks a Star Trek transporter is a transportation device rather than a murder+clone system, there’s no reason to think the AI would have detailed enough records to re-create everyone. Collecting that level of information would be even harder than getting enough to extrapolate CEV.
So I suppose it might matter to the humanity it re-creates, assuming it bothers. But we’d all still be dead, which is a decidedly suboptimal result.
At the end of this, I’m going to try to argue that something like CEV is still justified. Before I started thinking it through I was hoping that taking an eliminativist view of preferences to its conclusion would help tie up the loopholes in CEV, and so far it hasn’t done that for me, but it hasn’t made it any harder either.
CEV has worse problems that worries about convergence. The big one is that it’s such a difficult thing to implement that any AI capable of doing so has already crossed the threshold of extremely dangerous transhuman capability, and there’s no real solution to how to regulate its behavior while it’s in the process of working on the extrapolation. It could very well turn the planet into computronium before it gets a satisfactory implementation, by which point it doesn’t much matter what result it arrives at.
Presumably it matters if it then turns the planet back?
Even if you’re the type who thinks a Star Trek transporter is a transportation device rather than a murder+clone system, there’s no reason to think the AI would have detailed enough records to re-create everyone. Collecting that level of information would be even harder than getting enough to extrapolate CEV.
So I suppose it might matter to the humanity it re-creates, assuming it bothers. But we’d all still be dead, which is a decidedly suboptimal result.
Well, a neverending utopia fit to the exact specifications of humanity’s CEV is still pretty darn good, all things considered.