Edit: Actually, never mind, I missed your point. Will reply again later. (Done.)
What does this robot “actually want”, given that the world is not really a 2D grid of cells that have intrinsic color?
Our world is not really a 2D grid, its world could be. It won’t be a consequentialist about our world then, for that requires the dependence of its decisions on the dependence of our world on its decisions. It looks like the robot you described wants to minimize the 2D grid greenness, and would do that sensibly in the context of 2D grid world, or any world that can influence the 2D grid world. For the robot, our world doesn’t exist in the same sense as 2D grid world doesn’t exist for us, even though we could build an instance of the robot in our world. If we do build such an instance, the robot, if extremely rational and not just acting on heuristics adapted for its natural habitat, could devote its our-worldly existence to finding ways of acausally controlling its 2D world. For example, if there are some 2D-worlders out there simulating our world, it could signal to them something that is expected to reduce greenness.
(This all depends on the details of robot’s decision-making tools, of course. It could really be talking about our world, but then its values collapse, and it could turn out to not be a consequentialist after all, or optimizing something very different, to the extent the conflict in the definitions is strong.)
Edit: Actually, never mind, I missed your point. Will reply again later. (Done.)
Our world is not really a 2D grid, its world could be. It won’t be a consequentialist about our world then, for that requires the dependence of its decisions on the dependence of our world on its decisions. It looks like the robot you described wants to minimize the 2D grid greenness, and would do that sensibly in the context of 2D grid world, or any world that can influence the 2D grid world. For the robot, our world doesn’t exist in the same sense as 2D grid world doesn’t exist for us, even though we could build an instance of the robot in our world. If we do build such an instance, the robot, if extremely rational and not just acting on heuristics adapted for its natural habitat, could devote its our-worldly existence to finding ways of acausally controlling its 2D world. For example, if there are some 2D-worlders out there simulating our world, it could signal to them something that is expected to reduce greenness.
(This all depends on the details of robot’s decision-making tools, of course. It could really be talking about our world, but then its values collapse, and it could turn out to not be a consequentialist after all, or optimizing something very different, to the extent the conflict in the definitions is strong.)