Will continue responding, but first, after reading the existing comments, I think I do need to explicitly make humans preferred. I propose that in the sim we have some agents whose inner state is “copyable” and they get reincarnated, and some agents who are not “copyable”. Subtract points from all agents whenever a non-copyable agent is harmed/dies.
The idea is that humans are not copyable, and that’s the main reason AIs should treat us well, while AIs are, and that’s the main reason we don’t have to treat them well.
But also, I think we as humans might actually have to learn to treat AIs well in some sense/to some degree...
Will continue responding, but first, after reading the existing comments, I think I do need to explicitly make humans preferred. I propose that in the sim we have some agents whose inner state is “copyable” and they get reincarnated, and some agents who are not “copyable”. Subtract points from all agents whenever a non-copyable agent is harmed/dies. The idea is that humans are not copyable, and that’s the main reason AIs should treat us well, while AIs are, and that’s the main reason we don’t have to treat them well. But also, I think we as humans might actually have to learn to treat AIs well in some sense/to some degree...