I hope so. Most UFAI scenarios so far suggested, IIRC, end with everyone either dead or as mindless blobs of endless joy (which may or may not be the same thing, but I’d pick wireheading over death). But remember that the UFAI’s designers, stupid though they may be, will be unlikely to forget “thou shalt not kill featherless bipeds with straight nails”. So there’s a disturbing and non-negligible chance of waking up in Christian heaven.
Edit: So after all this, does cryonics still sound like a good idea? If yes, why? I really, really WANT there to be reasons to sign up. I want to see that world without traffic jams or copyright lawyers. But I’m just not convinced, and that’s depressing.
And given Lawrence’s 42 years of life after reverting the change why ever did he not work on getting another 0.199% right? In fact, what was Caroline thinking reverting the change before they had solid plan for post Prime-Intellect survival?
Fictional characters and their mortality fetishes. Pah.
The correct interpretation of the ending (based on the excerpt from the sequel posted and an interview localroger did with an Australian radio/podcast host) is that Caroline did not really revert the change; Prime Intellect remained in control of the universe.
“The Change” was keeping humans in a simulation of the universe (and turning the actual universe into computronium) instead of in the universe itself. So when it “reversed the Change” it was still as powerful as it was before the Change. What had happened was that Prime Intellect had been convinced that the post-Change world it created was not the best way of achieving its goals, so it set up a different universe. (I imagine that, as of chapter 8, Prime Intellect’s current plan for humanity is something like Buddhist-style reincarnation—after all, its highest priority is to prevent human deaths.)
I agree. Prime Intellect is absolutely friendly in that most important sense of caring about the continued existence and well-being of humans.
It was a good story, but I’m not sure that humans would have actually behaved as in that universe. Or we only saw a small subset of that universe. For example, we saw no one make themselves exponentially smarter. No one cloned themselves. No people merged consciousnesses. No one tried to convince Prime Intellect to reactivate the aliens inside of a zoo that allowed them to exist and for humanity to interact with them, without the danger of the aliens gaining control of Technology.
If I could choose between waiting around for Eliezer to make Friendly AI (or fail), I would choose the universe of Prime Intellect in a heartbeat. I don’t see why Fun Theory doesn’t apply there.
I hope so. Most UFAI scenarios so far suggested, IIRC, end with everyone either dead or as mindless blobs of endless joy (which may or may not be the same thing, but I’d pick wireheading over death). But remember that the UFAI’s designers, stupid though they may be, will be unlikely to forget “thou shalt not kill featherless bipeds with straight nails”. So there’s a disturbing and non-negligible chance of waking up in Christian heaven.
Edit: So after all this, does cryonics still sound like a good idea? If yes, why? I really, really WANT there to be reasons to sign up. I want to see that world without traffic jams or copyright lawyers. But I’m just not convinced, and that’s depressing.
Or in “The Metamorphosis of Prime Intellect”.
Prime Intellect was like this close to being Friendly.
Yep, you’ve got to get your AI like 99.8% right for it to go wrong that way.
And given Lawrence’s 42 years of life after reverting the change why ever did he not work on getting another 0.199% right? In fact, what was Caroline thinking reverting the change before they had solid plan for post Prime-Intellect survival?
Fictional characters and their mortality fetishes. Pah.
The correct interpretation of the ending (based on the excerpt from the sequel posted and an interview localroger did with an Australian radio/podcast host) is that Caroline did not really revert the change; Prime Intellect remained in control of the universe.
http://www.kuro5hin.org/prime-intellect/mopidnf.html
“The Change” was keeping humans in a simulation of the universe (and turning the actual universe into computronium) instead of in the universe itself. So when it “reversed the Change” it was still as powerful as it was before the Change. What had happened was that Prime Intellect had been convinced that the post-Change world it created was not the best way of achieving its goals, so it set up a different universe. (I imagine that, as of chapter 8, Prime Intellect’s current plan for humanity is something like Buddhist-style reincarnation—after all, its highest priority is to prevent human deaths.)
Actually, I’m more tempted to say that he was friendly, just not generally intelligent enough. Some of the humans seemed really silly, though…
I’ve no idea what extrapolated volition would mean in a population with that many freaks :-)
I agree. Prime Intellect is absolutely friendly in that most important sense of caring about the continued existence and well-being of humans.
It was a good story, but I’m not sure that humans would have actually behaved as in that universe. Or we only saw a small subset of that universe. For example, we saw no one make themselves exponentially smarter. No one cloned themselves. No people merged consciousnesses. No one tried to convince Prime Intellect to reactivate the aliens inside of a zoo that allowed them to exist and for humanity to interact with them, without the danger of the aliens gaining control of Technology.
If I could choose between waiting around for Eliezer to make Friendly AI (or fail), I would choose the universe of Prime Intellect in a heartbeat. I don’t see why Fun Theory doesn’t apply there.