Obviously misery would be avoided because it’s bad, not the other way around.
As mentioned, this isn’t obvious to me, so I’d be interested in your reasoning. Why should evolution build systems that want to avoid intrinsically bad mental states?
We are trying to figure out what is bad by seeing what we avoid. And the problem remains whether we might be accidentally avoiding misery, while trying to avoid its opposite.
Yes, my point here was twofold. One, the formalism used in the paper does not seem to be deeply meaningful, so it would be best to look for some other angle of attack. Two, given the claim about intrinsic badness, the programmer is embedding domain knowledge (about conscious states), not unlearnable assumptions. A computer system would fail to learn this because qualia is a hard problem, not because it’s unlearnable. This makes it asymmetric and circumventable in a way that the no free lunch theorem is not.
As mentioned, this isn’t obvious to me, so I’d be interested in your reasoning. Why should evolution build systems that want to avoid intrinsically bad mental states?
Yes, my point here was twofold. One, the formalism used in the paper does not seem to be deeply meaningful, so it would be best to look for some other angle of attack. Two, given the claim about intrinsic badness, the programmer is embedding domain knowledge (about conscious states), not unlearnable assumptions. A computer system would fail to learn this because qualia is a hard problem, not because it’s unlearnable. This makes it asymmetric and circumventable in a way that the no free lunch theorem is not.