Thank you so much for the comments! I’m pretty new to the platform (and to EA research in general), so feedback is useful for getting a broader perspective on our work.
To add to TurnTrout’s comments about power-scarcity and the CCC, I’d say that the broader vision of the multi-agent formulation is to establish a general notion of power-scarcity as a function of “similarity” between players’ reward functions (I mention this in the post’s final notes). In this paradigm, the constant-sum case is one limiting case of “general power-scarcity”, which I see as the “big idea”. As a simple example, general power-scarcity would provide a direct motivation for fearing robustly instrumental goals, since we’d have reason to believe an AI with goals orthogonal(ish) from human goals would be incentivized to compete with humanity for Power.
We’re planning to continue investigating multi-agent Power and power-scarcity, so hopefully we’ll have a more fleshed-out notion of general power-scarcity in the months to come.
Also, re: “as players’ strategies improve, their collective Power tends to decrease”, I think your intuition is correct? Upon reflection, the effect can be explained reasonably well by “improving your actions has no effect on your Power, but a negative effect on opponents’ Power”.
I go into more detail in my answer to Alex, but what I want to say here is that I don’t feel like you use the power-scarcity idea enough in the post itself. As you said, it’s one of three final notes, and without any emphasis on it.
So while I agree that the power-scarcity is an important research question, it would be helpful IMO if this post put more emphasis on that connection.
Thank you so much for the comments! I’m pretty new to the platform (and to EA research in general), so feedback is useful for getting a broader perspective on our work.
To add to TurnTrout’s comments about power-scarcity and the CCC, I’d say that the broader vision of the multi-agent formulation is to establish a general notion of power-scarcity as a function of “similarity” between players’ reward functions (I mention this in the post’s final notes). In this paradigm, the constant-sum case is one limiting case of “general power-scarcity”, which I see as the “big idea”. As a simple example, general power-scarcity would provide a direct motivation for fearing robustly instrumental goals, since we’d have reason to believe an AI with goals orthogonal(ish) from human goals would be incentivized to compete with humanity for Power.
We’re planning to continue investigating multi-agent Power and power-scarcity, so hopefully we’ll have a more fleshed-out notion of general power-scarcity in the months to come.
Also, re: “as players’ strategies improve, their collective Power tends to decrease”, I think your intuition is correct? Upon reflection, the effect can be explained reasonably well by “improving your actions has no effect on your Power, but a negative effect on opponents’ Power”.
Glad to be helpful!
I go into more detail in my answer to Alex, but what I want to say here is that I don’t feel like you use the power-scarcity idea enough in the post itself. As you said, it’s one of three final notes, and without any emphasis on it.
So while I agree that the power-scarcity is an important research question, it would be helpful IMO if this post put more emphasis on that connection.