Do you mean that the agent itself must be the person it cares about? What if the agent is carried in a backpack (of the person in question), or works over the Internet?
What if the selfish agent that cares about itself writes an AI that cares about the agent, giving this AI more optimization power, since they share the same goal?
What is your meaning then? What would you call “caring about the welfare of a particular individual (that happens to be myself)”?
Ok, I do mean:
but I don’t mean:
(i.e., without the part in parenthesis) Does that clear it up?
Ah, there was a slight confusion on my part. So if I’m reading this correctly you define formally selfish to mean… selfish. :-)
Do you mean that the agent itself must be the person it cares about? What if the agent is carried in a backpack (of the person in question), or works over the Internet?
What if the selfish agent that cares about itself writes an AI that cares about the agent, giving this AI more optimization power, since they share the same goal?