it’s the agent’s job to convince other agents based on its behavior
So agents are rewarded for doing stuff that convinces others that they’re a “happy AI”, not necessarily actually being a “happy AI”? Doesn’t that start an arms race of agents coming up with more and more sophisticated ways to deceive each other?
Like, suppose you start with a population of “happy AIs” that cooperate with each other, then if one of them realizes there’s a new way to deceive the others, there’s nothing to stop them until other agents adapt to this new kind of deception and learn to detect it? That feels like training an inherently unsafe and deceptive AI that also are extremely suspicious of others, not something “happy” and “friendly”
Ah, I see. But how would they actually escape the deception arms race? The agents still need some system of detecting cooperation, and if it can be easily abused, it generally will be (Goodhart’s Law and all that). I just can’t see any other outcome other than agents evolving exceedingly more complicated ways to detect if someone is cooperating or not. This is certainly an interesting thing to simulate, but I’m not sure how that is useful for aligning the agents. Aren’t we supposed to make them not even want to deceive others, instead of trying to find a deception strategy and failing? (Also, I think even an average human isn’t that well aligned as we want our AIs to be. You wouldn’t want to give a random guy from the street nuclear codes, would you?)
How do humans do it?
Ultimately, genuine altruism is computationally hard to fake; so it ends up being evolutionarily advantageous to have some measure of the real thing.
This is particularly true in environments with high cooperation rewards and low resource competition; eg where carrying capacity is maintained primarily by wild animals, general hard conditions, and disease, rather than overuse of resources. So we put our thumbs on the scale there to make these AIs better than your average human. And we rely on the AIs themselves to keep each other in check.
Ah, true. I just think this wouldn’t be enough and that there could be distributional shift if the agents are put into an environment with low cooperation rewards and high resource competition. I’ll reply in more detail under your new post, it looks a lot better
So agents are rewarded for doing stuff that convinces others that they’re a “happy AI”, not necessarily actually being a “happy AI”? Doesn’t that start an arms race of agents coming up with more and more sophisticated ways to deceive each other?
Like, suppose you start with a population of “happy AIs” that cooperate with each other, then if one of them realizes there’s a new way to deceive the others, there’s nothing to stop them until other agents adapt to this new kind of deception and learn to detect it? That feels like training an inherently unsafe and deceptive AI that also are extremely suspicious of others, not something “happy” and “friendly”
Yes, just like for humans. But also, if they can escape that game and genuinely cooperate, they’re rewarded, like humans but more so.
Ah, I see. But how would they actually escape the deception arms race? The agents still need some system of detecting cooperation, and if it can be easily abused, it generally will be (Goodhart’s Law and all that). I just can’t see any other outcome other than agents evolving exceedingly more complicated ways to detect if someone is cooperating or not. This is certainly an interesting thing to simulate, but I’m not sure how that is useful for aligning the agents. Aren’t we supposed to make them not even want to deceive others, instead of trying to find a deception strategy and failing? (Also, I think even an average human isn’t that well aligned as we want our AIs to be. You wouldn’t want to give a random guy from the street nuclear codes, would you?)
How do humans do it? Ultimately, genuine altruism is computationally hard to fake; so it ends up being evolutionarily advantageous to have some measure of the real thing. This is particularly true in environments with high cooperation rewards and low resource competition; eg where carrying capacity is maintained primarily by wild animals, general hard conditions, and disease, rather than overuse of resources. So we put our thumbs on the scale there to make these AIs better than your average human. And we rely on the AIs themselves to keep each other in check.
Ah, true. I just think this wouldn’t be enough and that there could be distributional shift if the agents are put into an environment with low cooperation rewards and high resource competition. I’ll reply in more detail under your new post, it looks a lot better