Yeah. I think a key point that is often overlooked is that even if powerful AI is technically controllable, i.e. we solve inner alignment, that doesn’t mean society will handle it safely. I think by default it looks like every company and military is forced to start using a ton of AI agents (or they will be outcompeted by someone else who does). Competition between a bunch of superhuman AIs that are trying to maximize profits or military tech seems really bad for us. We might not lose control all at once, but rather just be gradually outcompeted by machines, where “gradually” might actually be pretty quick. Basically, we die by Moloch.
Yes, I see Moloch as my, and humanity’s, primary enemy here. I think there are quite a few different plausible future paths in which Moloch rears its ugly head. The challenge, and duty, of coordination to defeat Moloch goes beyond what we think of as governance. We need coordination between AI researchers, AI alignment researchers, forecasters, politicians, investors, CEOs. We need people realizing their lives are at stake and making sacrifices and compromises to reduce the risks.
The challenge, and duty, of coordination to defeat Moloch goes beyond what we think of as governance. We need coordination between AI researchers, AI alignment researchers, forecasters, politicians, investors, CEOs.
The problem is that an entity with that kind of real world coordination capacity would practically need to be so strong that it would likely be more controversial, and face more backlash, then the rogue AGI(s) itself.
At which point some fraction of humans would likely defect and cooperate with the AGI(s) in order to take it down.
Oh, I wasn’t imagining a singleton AI solving the coordination problem. I was more imagining that a series of terrifying near misses and minor catastrophes convinced people to work together for their own best interest. The coordination being done by the people involved, not applied to them by an external force.
Even a purely human organization with kind of potential power would be controversial enough that probably at least a single digit percentage of adults would not accept it. Which is to say hundreds of millions of humans would likely consider it an enemy too.
And that’s assuming it can even be done considering the level of global cooperation demonstrated in 2023.
Yes, I think you are right about both the difficulty / chance of failure and about the fact that there would inevitably be a lot of people opposed. Those aren’t enough to guarantee such coordination would fail, perhaps especially if it was enacted through a redundant mishmash of organizations?
I’m pretty sure there’s going to be some significant conflict along the way, no matter which path the future stumbles down.
I doubt you, or any human being, would even want to live in a world where such coordination ‘succeeded’, since it would almost certainly be in the ruins of society wrecked by countless WMDs, flung by the warring parties until all were exhausted except the ‘winners’, who would probably not have long to live.
In that sense the possible futures where control of powerful AI ‘succeeded’ could be even worse then where it failed.
I really hoping it doesn’t go that way, but I do see us as approaching a time in which the military and economic implications of AI will become so pressing that large-scale international conflict is likely unless agreements are reached. There are specific ways I anticipate tool AI advances affecting the power balance between superpower countries, even before autonomous AGI is a threat. I wake in the night in a cold sweat worrying about these things. I am terrified. I think there’s a real chance we all die soon, or that there is massive suffering and chaos, perhaps with or without war. The balance of power has shifted massively in favor of offense, and a new tenuous balance of Mutually Assured Destruction has not yet been established. This is a very dangerous time.
Yeah. I think a key point that is often overlooked is that even if powerful AI is technically controllable, i.e. we solve inner alignment, that doesn’t mean society will handle it safely. I think by default it looks like every company and military is forced to start using a ton of AI agents (or they will be outcompeted by someone else who does). Competition between a bunch of superhuman AIs that are trying to maximize profits or military tech seems really bad for us. We might not lose control all at once, but rather just be gradually outcompeted by machines, where “gradually” might actually be pretty quick. Basically, we die by Moloch.
Yes, I see Moloch as my, and humanity’s, primary enemy here. I think there are quite a few different plausible future paths in which Moloch rears its ugly head. The challenge, and duty, of coordination to defeat Moloch goes beyond what we think of as governance. We need coordination between AI researchers, AI alignment researchers, forecasters, politicians, investors, CEOs. We need people realizing their lives are at stake and making sacrifices and compromises to reduce the risks.
The problem is that an entity with that kind of real world coordination capacity would practically need to be so strong that it would likely be more controversial, and face more backlash, then the rogue AGI(s) itself.
At which point some fraction of humans would likely defect and cooperate with the AGI(s) in order to take it down.
Oh, I wasn’t imagining a singleton AI solving the coordination problem. I was more imagining that a series of terrifying near misses and minor catastrophes convinced people to work together for their own best interest. The coordination being done by the people involved, not applied to them by an external force.
Even a purely human organization with kind of potential power would be controversial enough that probably at least a single digit percentage of adults would not accept it. Which is to say hundreds of millions of humans would likely consider it an enemy too.
And that’s assuming it can even be done considering the level of global cooperation demonstrated in 2023.
Yes, I think you are right about both the difficulty / chance of failure and about the fact that there would inevitably be a lot of people opposed. Those aren’t enough to guarantee such coordination would fail, perhaps especially if it was enacted through a redundant mishmash of organizations?
I’m pretty sure there’s going to be some significant conflict along the way, no matter which path the future stumbles down.
I doubt you, or any human being, would even want to live in a world where such coordination ‘succeeded’, since it would almost certainly be in the ruins of society wrecked by countless WMDs, flung by the warring parties until all were exhausted except the ‘winners’, who would probably not have long to live.
In that sense the possible futures where control of powerful AI ‘succeeded’ could be even worse then where it failed.
I really hoping it doesn’t go that way, but I do see us as approaching a time in which the military and economic implications of AI will become so pressing that large-scale international conflict is likely unless agreements are reached. There are specific ways I anticipate tool AI advances affecting the power balance between superpower countries, even before autonomous AGI is a threat. I wake in the night in a cold sweat worrying about these things. I am terrified. I think there’s a real chance we all die soon, or that there is massive suffering and chaos, perhaps with or without war. The balance of power has shifted massively in favor of offense, and a new tenuous balance of Mutually Assured Destruction has not yet been established. This is a very dangerous time.