“Our starting assumptions are probably right, ergo we can say with some confidence that the future will involve trillions of identical uploaded minds maximizing their utility functions, unless of course the Paperclip-Maximizer ‘clips’ it all in the bud”
I accepted the importance and correctness of their inference, but I ran it in the opposite direction:
“It seems obvious that we can’t say such things with any confidence, ergo the starting assumptions ought to be carefully revisited—even the ones about mind and computation that most scientifically-literate people say they agree with.”
I don’t see how Scott’s proposed revision of the starting assumptions actually changes the conclusions. Even if he is right that uploads and AIs with a “digital abstraction layer” can’t be conscious, that’s not going to stop a future involving trillions of uploads, or stop paperclip maximizers.
If these uploads are p-zombies (Scott-zombies?) because they are reversible computations, then their welfare doesn’t matter. I don’t think he says that it prevents paperclip maximizers.
He suggests that all those uploads might not be conscious if they are run on a quantum computer reversibly (or have some other “clean digital abstraction layer”). He states that this is a huge speculation, but it is still an alternative not usually considered by the orthodox reductionists.
I don’t see how Scott’s proposed revision of the starting assumptions actually changes the conclusions. Even if he is right that uploads and AIs with a “digital abstraction layer” can’t be conscious, that’s not going to stop a future involving trillions of uploads, or stop paperclip maximizers.
If these uploads are p-zombies (Scott-zombies?) because they are reversible computations, then their welfare doesn’t matter. I don’t think he says that it prevents paperclip maximizers.
So Scott meant to argue against “the future should involve trillions of uploads” rather than “the future will involve trillions of uploads”?
He suggests that all those uploads might not be conscious if they are run on a quantum computer reversibly (or have some other “clean digital abstraction layer”). He states that this is a huge speculation, but it is still an alternative not usually considered by the orthodox reductionists.