I agree that you have to renorm everyone’s wants for this to work. I also agree that if you can construct broken minds for the purpose of manipulating the FAI, we need provisions to guard against that. My preferred alternative at the moment follows:
Before people become able to construct broken minds, the FAI cares about everything that’s genetically human.
After we find the first genetically human mind deliberately broken for the purpose of manipulating the FAI, we guess when the FAI started to be influenced by that, and retroactive to just before that time we introduce a new policy: new individuals start out with a weight of 0, and can receive weight transferred from their parents, so the total weight is conserved. I don’t want an economy of weight-transfer to arise, so it would be a one-way irreversible transfer.
This might lead to a few people running around with a weight of 0 because their parents never made the transfer. This would be suboptimal, but it would not have horrible conclusions because the AI would care for the parents who probably care for the new child, so the AI would in effect care some for the new child.
Death of the parents doesn’t break this. Caring about the preference of dead people is not a special case.
I encourage people to reply to this post with bugs in this alternative or with other plausible alternatives.
I agree that you have to renorm everyone’s wants for this to work. I also agree that if you can construct broken minds for the purpose of manipulating the FAI, we need provisions to guard against that. My preferred alternative at the moment follows:
Before people become able to construct broken minds, the FAI cares about everything that’s genetically human.
After we find the first genetically human mind deliberately broken for the purpose of manipulating the FAI, we guess when the FAI started to be influenced by that, and retroactive to just before that time we introduce a new policy: new individuals start out with a weight of 0, and can receive weight transferred from their parents, so the total weight is conserved. I don’t want an economy of weight-transfer to arise, so it would be a one-way irreversible transfer.
This might lead to a few people running around with a weight of 0 because their parents never made the transfer. This would be suboptimal, but it would not have horrible conclusions because the AI would care for the parents who probably care for the new child, so the AI would in effect care some for the new child.
Death of the parents doesn’t break this. Caring about the preference of dead people is not a special case.
I encourage people to reply to this post with bugs in this alternative or with other plausible alternatives.