Well gang, it looks like we have come to the part where we are struggling directly over the destiny of humanity. In addition to persuasion and incentives, we’ll have to account for the explicit fights over control of the orgs.
Silver lining: it means we have critical mass for enough of humanity and enough wealth in play to die with our boots on, at least!
The last few days should show it’s not enough to have power cemented in technicalities, board seats, or legal contracts. Power comes from gaining support of billionaires, journalists, and human capital. It’s kind of crazy that Sam Altman essentially rewrote the rules, whether he was justified or not.
The problem is the types of people who are good at gaining power tend to have values that are incompatible with EA. The real silver lining to me, is that while it’s clear Sam Altman is power-seeking, he’s also probably a better option to have there than the rest of the people good at seeking power, who might not even entertain x-risk.
Power comes from gaining support of billionaires, journalists, and human capital. It’s kind of crazy that Sam Altman essentially rewrote the rules, whether he was justified or not.
Until stakes get so high that we go straight to the realm of violence (threatened, by the State or other actors, or enacted), yes, it does. Enough soft power becomes hard. Honestly can’t figure how anyone thought this setup would work, especially with Altman being a deft manipulator as he seems to be. I’ve made the Senate vs Caesar comparison before but I’ll reiterate it, because ultimately rules only matter until personal loyalty persuades enough people to ignore them.
I agree in the main, and I think it is worth emphasizing that power-seeking is a skillset, which is orthogonal to values; we should put it in the Dark Arts pile, and anyone involved in running an org should learn it at least enough to defend against it.
Power seeking mostly succeeds by the other agents not realizing what is going on, so it either takes them by surprise or they don’t even notice it happened until the power is exerted.
Yet power seeking is a symmetric behavior, and power is scarce. The defense is to compete for power against the other agent, and try to eliminate them if possible.
I agree. For power that comes from (money/reputation/military/hype) developing AI systems, this is where I was wondering where the symmetry is for those who wish to stop AI being developed. The ‘doomer’ faction over time won’t be benefitting from AI, and thus their relative power would be weaker and weaker with time. Assuming at least initially, AI systems have utility value to humans, if the AI systems treacherous turn it will be later, after initially providing large benefits.
With this corporate battle right now, Altman is going to announce whatever advances they have made, raise billions of dollars, and the next squabble will have the support of people that Altman has personally enriched. I heard a rumor it works out to be 3.6 million per employee with the next funding round, with a 2 year cliff, so 1.8 million/year on the line. This would be why almost all OpenAI employees stated they would leave over it. So it’s a direct example of the general symmetry issue mentioned above. (leaving would also pay: I suspect Microsoft would have offered more liquid RSUs and probably matched the OAI base, maybe large signing bonuses. Not 1.8m but good TC. Just me speculating, I don’t know the offer details and it is likely Microsoft hadn’t decided)
The only ‘obvious’ way I could see doomers gaining power with time would be if early AI systems cause mass murder events or damage similar to nuclear meltdowns. This would give them political support, and it would scale with AI system capability, as more powerful systems cause more and more deaths and more and more damage.
Off the top of my head: evading notice, running away (to a different jurisdiction/market, or in some even more abstract sense), appearing to be a poisoned meal (likely by actually being poisoned, such that eating you will be more harmful to their reputation or demand more of their focused attention than you’re worth to them), or seeking the protection of dead-player institutions (who’d impose a steep-but-flat cost on you, and then won’t actively try to take you over).
I think it is not even goals but means. When you have big hammer every problem looks like a nail, if you good at talking you start to think you can talk your way out of any problem.
Well gang, it looks like we have come to the part where we are struggling directly over the destiny of humanity. In addition to persuasion and incentives, we’ll have to account for the explicit fights over control of the orgs.
Silver lining: it means we have critical mass for enough of humanity and enough wealth in play to die with our boots on, at least!
The last few days should show it’s not enough to have power cemented in technicalities, board seats, or legal contracts. Power comes from gaining support of billionaires, journalists, and human capital. It’s kind of crazy that Sam Altman essentially rewrote the rules, whether he was justified or not.
The problem is the types of people who are good at gaining power tend to have values that are incompatible with EA. The real silver lining to me, is that while it’s clear Sam Altman is power-seeking, he’s also probably a better option to have there than the rest of the people good at seeking power, who might not even entertain x-risk.
Until stakes get so high that we go straight to the realm of violence (threatened, by the State or other actors, or enacted), yes, it does. Enough soft power becomes hard. Honestly can’t figure how anyone thought this setup would work, especially with Altman being a deft manipulator as he seems to be. I’ve made the Senate vs Caesar comparison before but I’ll reiterate it, because ultimately rules only matter until personal loyalty persuades enough people to ignore them.
I agree in the main, and I think it is worth emphasizing that power-seeking is a skillset, which is orthogonal to values; we should put it in the Dark Arts pile, and anyone involved in running an org should learn it at least enough to defend against it.
What would be your defense? Agents successful at seeking power have more power.
Power seeking mostly succeeds by the other agents not realizing what is going on, so it either takes them by surprise or they don’t even notice it happened until the power is exerted.
Yet power seeking is a symmetric behavior, and power is scarce. The defense is to compete for power against the other agent, and try to eliminate them if possible.
I agree. For power that comes from (money/reputation/military/hype) developing AI systems, this is where I was wondering where the symmetry is for those who wish to stop AI being developed. The ‘doomer’ faction over time won’t be benefitting from AI, and thus their relative power would be weaker and weaker with time. Assuming at least initially, AI systems have utility value to humans, if the AI systems treacherous turn it will be later, after initially providing large benefits.
With this corporate battle right now, Altman is going to announce whatever advances they have made, raise billions of dollars, and the next squabble will have the support of people that Altman has personally enriched. I heard a rumor it works out to be 3.6 million per employee with the next funding round, with a 2 year cliff, so 1.8 million/year on the line. This would be why almost all OpenAI employees stated they would leave over it. So it’s a direct example of the general symmetry issue mentioned above. (leaving would also pay: I suspect Microsoft would have offered more liquid RSUs and probably matched the OAI base, maybe large signing bonuses. Not 1.8m but good TC. Just me speculating, I don’t know the offer details and it is likely Microsoft hadn’t decided)
The only ‘obvious’ way I could see doomers gaining power with time would be if early AI systems cause mass murder events or damage similar to nuclear meltdowns. This would give them political support, and it would scale with AI system capability, as more powerful systems cause more and more deaths and more and more damage.
Off the top of my head: evading notice, running away (to a different jurisdiction/market, or in some even more abstract sense), appearing to be a poisoned meal (likely by actually being poisoned, such that eating you will be more harmful to their reputation or demand more of their focused attention than you’re worth to them), or seeking the protection of dead-player institutions (who’d impose a steep-but-flat cost on you, and then won’t actively try to take you over).
I think it is not even goals but means. When you have big hammer every problem looks like a nail, if you good at talking you start to think you can talk your way out of any problem.