It seems to me that the future goes one of two ways: virtually complete disaster (uFAI, other technology disaster, complete loss of humane values) or a fairly strong win for human values with massive easing of material constraints—superabundance.
In the latter group of scenarios, the cost of reanimating someone by scanning and building a new fleshy body for them would be close to nothing, and the market for sentimentally special objects (such as people from before the positive singularity) would be huge.
The scenarios that scare me are those where a friendly AI is built that reasons timelessly that it would have wanted to precommit to placing massive threats/​rewards on anyone who knew about the singularity risk problem, and that unfortunately you (or I) in particular performed below some set standard of risk mitigation, and the FAI is going to put us in a horrific torture simulation forever. The argument against this is that rewards motivate better than punishments.
It seems to me that the future goes one of two ways: virtually complete disaster (uFAI, other technology disaster, complete loss of humane values) or a fairly strong win for human values with massive easing of material constraints—superabundance.
In the latter group of scenarios, the cost of reanimating someone by scanning and building a new fleshy body for them would be close to nothing, and the market for sentimentally special objects (such as people from before the positive singularity) would be huge.
The scenarios that scare me are those where a friendly AI is built that reasons timelessly that it would have wanted to precommit to placing massive threats/​rewards on anyone who knew about the singularity risk problem, and that unfortunately you (or I) in particular performed below some set standard of risk mitigation, and the FAI is going to put us in a horrific torture simulation forever. The argument against this is that rewards motivate better than punishments.
Disagree that you should be scared rather than curious. Such scenarios are not well worked out.