I agree in the main, and I think it is worth emphasizing that power-seeking is a skillset, which is orthogonal to values; we should put it in the Dark Arts pile, and anyone involved in running an org should learn it at least enough to defend against it.
Power seeking mostly succeeds by the other agents not realizing what is going on, so it either takes them by surprise or they don’t even notice it happened until the power is exerted.
Yet power seeking is a symmetric behavior, and power is scarce. The defense is to compete for power against the other agent, and try to eliminate them if possible.
I agree. For power that comes from (money/reputation/military/hype) developing AI systems, this is where I was wondering where the symmetry is for those who wish to stop AI being developed. The ‘doomer’ faction over time won’t be benefitting from AI, and thus their relative power would be weaker and weaker with time. Assuming at least initially, AI systems have utility value to humans, if the AI systems treacherous turn it will be later, after initially providing large benefits.
With this corporate battle right now, Altman is going to announce whatever advances they have made, raise billions of dollars, and the next squabble will have the support of people that Altman has personally enriched. I heard a rumor it works out to be 3.6 million per employee with the next funding round, with a 2 year cliff, so 1.8 million/year on the line. This would be why almost all OpenAI employees stated they would leave over it. So it’s a direct example of the general symmetry issue mentioned above. (leaving would also pay: I suspect Microsoft would have offered more liquid RSUs and probably matched the OAI base, maybe large signing bonuses. Not 1.8m but good TC. Just me speculating, I don’t know the offer details and it is likely Microsoft hadn’t decided)
The only ‘obvious’ way I could see doomers gaining power with time would be if early AI systems cause mass murder events or damage similar to nuclear meltdowns. This would give them political support, and it would scale with AI system capability, as more powerful systems cause more and more deaths and more and more damage.
Off the top of my head: evading notice, running away (to a different jurisdiction/market, or in some even more abstract sense), appearing to be a poisoned meal (likely by actually being poisoned, such that eating you will be more harmful to their reputation or demand more of their focused attention than you’re worth to them), or seeking the protection of dead-player institutions (who’d impose a steep-but-flat cost on you, and then won’t actively try to take you over).
I agree in the main, and I think it is worth emphasizing that power-seeking is a skillset, which is orthogonal to values; we should put it in the Dark Arts pile, and anyone involved in running an org should learn it at least enough to defend against it.
What would be your defense? Agents successful at seeking power have more power.
Power seeking mostly succeeds by the other agents not realizing what is going on, so it either takes them by surprise or they don’t even notice it happened until the power is exerted.
Yet power seeking is a symmetric behavior, and power is scarce. The defense is to compete for power against the other agent, and try to eliminate them if possible.
I agree. For power that comes from (money/reputation/military/hype) developing AI systems, this is where I was wondering where the symmetry is for those who wish to stop AI being developed. The ‘doomer’ faction over time won’t be benefitting from AI, and thus their relative power would be weaker and weaker with time. Assuming at least initially, AI systems have utility value to humans, if the AI systems treacherous turn it will be later, after initially providing large benefits.
With this corporate battle right now, Altman is going to announce whatever advances they have made, raise billions of dollars, and the next squabble will have the support of people that Altman has personally enriched. I heard a rumor it works out to be 3.6 million per employee with the next funding round, with a 2 year cliff, so 1.8 million/year on the line. This would be why almost all OpenAI employees stated they would leave over it. So it’s a direct example of the general symmetry issue mentioned above. (leaving would also pay: I suspect Microsoft would have offered more liquid RSUs and probably matched the OAI base, maybe large signing bonuses. Not 1.8m but good TC. Just me speculating, I don’t know the offer details and it is likely Microsoft hadn’t decided)
The only ‘obvious’ way I could see doomers gaining power with time would be if early AI systems cause mass murder events or damage similar to nuclear meltdowns. This would give them political support, and it would scale with AI system capability, as more powerful systems cause more and more deaths and more and more damage.
Off the top of my head: evading notice, running away (to a different jurisdiction/market, or in some even more abstract sense), appearing to be a poisoned meal (likely by actually being poisoned, such that eating you will be more harmful to their reputation or demand more of their focused attention than you’re worth to them), or seeking the protection of dead-player institutions (who’d impose a steep-but-flat cost on you, and then won’t actively try to take you over).