Are you expecting that the team behind the “melt all GPU farms” pivotal act to be backed by a major government or coalition of governments?
If not, I expect that the team and its AGI will be arrested/confiscated by the nearest authority as soon as the pivotal act occurs, and forced by them to apply the AGI to other goals. Do you see things happening differently, or expect things to come out well despite this?
“Melt all GPUs” is indeed an unrealistic pivotal act—which is why I talk about it, since like any pivotal act it is outside the Overton Window, and then if any children get indignant about the prospect of doing something other than letting the world end miserably, I get to explain the child-reassuring reasons why you would never do the particular thing of “melt all GPUs” in real life. In this case, the reassuring reason is that deploying open-air nanomachines to operate over Earth is a huge alignment problem, that is, relatively huger than the least difficult pivotal act I can currently see.
That said, if unreasonably-hypothetically you can give your AI enough of a utility function and have it deploy enough intelligence to create nanomachines that safely move through the open-ended environment of Earth’s surface, avoiding bacteria and not damaging any humans or vital infrastructure, in order to surveil all of Earth and find the GPU farms and then melt them all, it’s probably not very much harder to tell those nanomachines to melt other things, or demonstrate the credibly threatening ability to do so.
That said, I indeed don’t see how we sociologically get into this position in a realistic way, in anything like the current world, even assuming away the alignment problem. Unless Demis Hassabis suddenly executes an emergency pact with the Singaporean government, or something else I have trouble visualizing? I don’t see any of the current owners or local governments of the big AI labs knowingly going along with any pivotal act executed deliberately (though I expect them to think it’s just fine to keep cranking up the dial on an AI until it destroys the world, so long as it looks like it’s not being done on purpose).
It is indeed the case that, conditional on the alignment problem being solvable, there’s a further sociological problem—which looks a lot less impossible, but which I do not actually know how to solve—wherein you then have to do something pivotal, and there’s no grownups in government in charge who would understand why that was something necessary to do. But it’s definitely a lot easier to imagine Demis forming a siloed team or executing an emergency pact with Singapore, than it is to see how you would safely align the AI that does it. And yes, the difficulty of any pivotal act to stabilize the Earth includes the difficulty of what you had to do, before or after you had sufficiently powerful AGI, in order to execute that act and then prevent things from falling over immediately afterwards.
the least difficult pivotal act I can currently see.
Do you have a plan to communicate the content of this to people whom it would be beneficial to communicate to? E.g., write about it in some deniable way, or should such people just ask you about it privately? Or more generally, how do you think that discussions / intellectual progress on this topic should go?
Do you think the least difficult pivotal act you currently see has sociopolitical problems that are similar to “melt all GPUs”?
That said, I indeed don’t see how we sociologically get into this position in a realistic way, in anything like the current world, even assuming away the alignment problem.
Thanks for the clarification. I suggest mentioning this more often (like in the Arbital page), as I previously didn’t think that your version of “pivotal act” had a significant sociopolitical component. If this kind of pivotal act is indeed how the world gets saved (conditional on the world being saved), one of my concerns is that “a miracle occurs” and the alignment problem gets solved, but the sociopolitical problem doesn’t because nobody was working on it (even if it’s easier in some sense).
But it’s definitely a lot easier to imagine Demis forming a siloed team or executing an emergency pact with Singapore
(Not a high priority to discuss this here and now, but) I’m skeptical that backing by a small government like Singapore is sufficient, since any number of major governments would be very tempted to grab the AGI(+team) from the small government, and the small government will be under tremendous legal and diplomatic stress from having nonconsensually destroyed a lot of very valuable other people’s property. Having a partially aligned/alignable AGI in the hands of a small, geopolitically weak government seems like a pretty precarious state.
Can you please clarify:
Are you expecting that the team behind the “melt all GPU farms” pivotal act to be backed by a major government or coalition of governments?
If not, I expect that the team and its AGI will be arrested/confiscated by the nearest authority as soon as the pivotal act occurs, and forced by them to apply the AGI to other goals. Do you see things happening differently, or expect things to come out well despite this?
“Melt all GPUs” is indeed an unrealistic pivotal act—which is why I talk about it, since like any pivotal act it is outside the Overton Window, and then if any children get indignant about the prospect of doing something other than letting the world end miserably, I get to explain the child-reassuring reasons why you would never do the particular thing of “melt all GPUs” in real life. In this case, the reassuring reason is that deploying open-air nanomachines to operate over Earth is a huge alignment problem, that is, relatively huger than the least difficult pivotal act I can currently see.
That said, if unreasonably-hypothetically you can give your AI enough of a utility function and have it deploy enough intelligence to create nanomachines that safely move through the open-ended environment of Earth’s surface, avoiding bacteria and not damaging any humans or vital infrastructure, in order to surveil all of Earth and find the GPU farms and then melt them all, it’s probably not very much harder to tell those nanomachines to melt other things, or demonstrate the credibly threatening ability to do so.
That said, I indeed don’t see how we sociologically get into this position in a realistic way, in anything like the current world, even assuming away the alignment problem. Unless Demis Hassabis suddenly executes an emergency pact with the Singaporean government, or something else I have trouble visualizing? I don’t see any of the current owners or local governments of the big AI labs knowingly going along with any pivotal act executed deliberately (though I expect them to think it’s just fine to keep cranking up the dial on an AI until it destroys the world, so long as it looks like it’s not being done on purpose).
It is indeed the case that, conditional on the alignment problem being solvable, there’s a further sociological problem—which looks a lot less impossible, but which I do not actually know how to solve—wherein you then have to do something pivotal, and there’s no grownups in government in charge who would understand why that was something necessary to do. But it’s definitely a lot easier to imagine Demis forming a siloed team or executing an emergency pact with Singapore, than it is to see how you would safely align the AI that does it. And yes, the difficulty of any pivotal act to stabilize the Earth includes the difficulty of what you had to do, before or after you had sufficiently powerful AGI, in order to execute that act and then prevent things from falling over immediately afterwards.
Do you have a plan to communicate the content of this to people whom it would be beneficial to communicate to? E.g., write about it in some deniable way, or should such people just ask you about it privately? Or more generally, how do you think that discussions / intellectual progress on this topic should go?
Do you think the least difficult pivotal act you currently see has sociopolitical problems that are similar to “melt all GPUs”?
Thanks for the clarification. I suggest mentioning this more often (like in the Arbital page), as I previously didn’t think that your version of “pivotal act” had a significant sociopolitical component. If this kind of pivotal act is indeed how the world gets saved (conditional on the world being saved), one of my concerns is that “a miracle occurs” and the alignment problem gets solved, but the sociopolitical problem doesn’t because nobody was working on it (even if it’s easier in some sense).
(Not a high priority to discuss this here and now, but) I’m skeptical that backing by a small government like Singapore is sufficient, since any number of major governments would be very tempted to grab the AGI(+team) from the small government, and the small government will be under tremendous legal and diplomatic stress from having nonconsensually destroyed a lot of very valuable other people’s property. Having a partially aligned/alignable AGI in the hands of a small, geopolitically weak government seems like a pretty precarious state.
Singapore probably looks a lot less attractive to threaten if it’s allied with another world power that can find and melt arbitrary objects.