For the psychopaths, I figured this was covered with the “block” feature. If it extends from the game into the emulation, Celestia just has to tell the destructive psychopath that the ponies they are “killing” are real (she can lie) and keep them in a shard away from those who do not want to be killed. She doesn’t even need to lie per se: she can create emulation-ponies that harbor a deep desire to be killed while behaving like they want to live. And of course, introduce psychopath-ponies (or gryphons, or diamond dogs) that will act as their “friends.”
Thus, satisfying their values through friendship and ponies.
As for the Olympian… they probably do want to be the best, but that won’t always happen. But there are lots of things to be the best at, and it wouldn’t be unreasonable for there to be different leaderboards for “in shard” and “in universe.” And Celestia explicity notes that she is not maximizing happiness, but values: and the Olympian probably values the effort and work more than just the happiness of believing themselves to be number 1. (For those who go the other way, they get locked in shards where they are number 1 and simply not introduced to the wider world)
But that’s me interpretting another person’s story about a super-intelligent AI that presumably is smarter than either me or the author or both of us combined.
I guess my main objection is how the “value being in the true world” conflicting with other values is portrayed. Celestia sides against valuing a true world, and this theme is covered in Light Spark’s story, but would becomes explicit if there were direct conflicts. In the end, what Celestia does is just very sophisticated networked wireheading.
In my mind, this means Hanna failed, but it doesn’t seem quite portrayed as a failure. There is a hint of the horror as she consumes galaxies and tiles the universe with wireheaded ponies, but just a hint. It’s more subtle than I prefer. Specifically, I’m worried that someone reading it without LessWrong background would miss all that.
(Also, I just assumed the block function is a dummy button and doesn’t actually do anything when you press it.)
For the psychopaths, I figured this was covered with the “block” feature. If it extends from the game into the emulation, Celestia just has to tell the destructive psychopath that the ponies they are “killing” are real (she can lie) and keep them in a shard away from those who do not want to be killed. She doesn’t even need to lie per se: she can create emulation-ponies that harbor a deep desire to be killed while behaving like they want to live. And of course, introduce psychopath-ponies (or gryphons, or diamond dogs) that will act as their “friends.”
Thus, satisfying their values through friendship and ponies.
As for the Olympian… they probably do want to be the best, but that won’t always happen. But there are lots of things to be the best at, and it wouldn’t be unreasonable for there to be different leaderboards for “in shard” and “in universe.” And Celestia explicity notes that she is not maximizing happiness, but values: and the Olympian probably values the effort and work more than just the happiness of believing themselves to be number 1. (For those who go the other way, they get locked in shards where they are number 1 and simply not introduced to the wider world)
But that’s me interpretting another person’s story about a super-intelligent AI that presumably is smarter than either me or the author or both of us combined.
I guess my main objection is how the “value being in the true world” conflicting with other values is portrayed. Celestia sides against valuing a true world, and this theme is covered in Light Spark’s story, but would becomes explicit if there were direct conflicts. In the end, what Celestia does is just very sophisticated networked wireheading.
In my mind, this means Hanna failed, but it doesn’t seem quite portrayed as a failure. There is a hint of the horror as she consumes galaxies and tiles the universe with wireheaded ponies, but just a hint. It’s more subtle than I prefer. Specifically, I’m worried that someone reading it without LessWrong background would miss all that.
(Also, I just assumed the block function is a dummy button and doesn’t actually do anything when you press it.)
I have a Less Wrong background, and I don’t get what the problem is with abandoning the True World for a universe that actually cares about us.