Now we get to the question how detailed the paperclips have to be for the paperclipper to care. I expect the paperclipper to only care when the paperclips are simulated individually and we can’t simulate 3^^^^^^3 paperclips individually.
I see no reason to think any work of fiction can lead to such a distortion of reality.
I see no reason to think works of fiction that lead to such a distortion of reality are impossible.
Now we get to the question how detailed the paperclips have to be for the paperclipper to care. I expect the paperclipper to only care when the paperclips are simulated individually and we can’t simulate 3^^^^^^3 paperclips individually.
We built the paperclipper. Hell, it doesn’t even have to be a literal paperclipper. For that matter, it doesn’t have to be literally 3^^^^^^3 paperclips; all that matters is that we can manipulate it’s utility function without too much strain on our resources. If it values cheesecake, we can reward it with a world of infinite cheesecake. If it values “complexity” we can put it in a fractal. The point is that we can motivate it to co-operate without worrying that it might be a better idea to let it out.
Now we get to the question how detailed the paperclips have to be for the paperclipper to care. I expect the paperclipper to only care when the paperclips are simulated individually and we can’t simulate 3^^^^^^3 paperclips individually.
I see no reason to think works of fiction that lead to such a distortion of reality are impossible.
I assume you concede my point re:guards?
We built the paperclipper. Hell, it doesn’t even have to be a literal paperclipper. For that matter, it doesn’t have to be literally 3^^^^^^3 paperclips; all that matters is that we can manipulate it’s utility function without too much strain on our resources. If it values cheesecake, we can reward it with a world of infinite cheesecake. If it values “complexity” we can put it in a fractal. The point is that we can motivate it to co-operate without worrying that it might be a better idea to let it out.