I don’t think this revolutionises my argument. First, there’s a lot of talking about example possible pivotal acts and they’re mostly just not that believable on their own. The typical “melt all GPUs” is obviously incredibly hostile and disruptive, but yes, of course, it’s only an example. The problem is that without an actual outline for what a perfect pivotal act is, you can’t even hope to do it with “just” a narrow superintelligence, because in that case, you need to work out the details yourself, and the details are likely horribly complicated.
But the core, fundamental problem with the “pivotal act” notion is that it tries to turn a political problem into a technological one. “Do not build AGIs” is fundamentally a political problem: it’s about restricting human freedom. Now you can either do that voluntarily, by consensus, with some enforcement mechanism for the majority to impose its will on the minority, or you can do that by force, with a minority using overwhelming power to make the majority go along even against their will. That’s it. A pivotal act is just a nice name for the latter thing. The essence of the notion is “we can’t get everyone on board quickly enough; therefore, we should just build some kind of superweapon that allows us to stop everyone else from building unsafe AGI as we define it, whether they like it or not”. It’s not a lethal weapon, and you can argue the utilitarian trade-off from your viewpoint is quite good, but it is undeniably a weapon. And therefore it’s just not something that can be politically acceptable because people don’t like to have weapons pointed at them, not even when the person making the weapon assures them it’s for their own good. If “pivotal act” became the main paradigm the race dynamics would only intensify because then everyone knows they’ll only have one shot and they won’t trust the others to either get it right or actually limit themselves to just the pivotal act once they’re the only ones with AI power in the world. And if instead the world came together to agree on a pivotal act… well that’s just regulation, first, as described in this post. And then moving on to develop a kind of special nanobot police to enforce that regulation (which would still be a highly controversial action, and if deployed worldwide, essentially an act of war against any country not subscribing to the AI safety treaty or whatever).
I was just claiming that your description of pivotal acts / of people that support pivotal acts was incorrect in a way that people that think pivotal acts are worth considering would consider very significant and in a way that significantly reduces the power of your argument as applying to what people mean by pivotal acts — I don’t see anything in your comment as a response to that claim. I would like it to be a separate discussion whether pivotal acts are a good idea with this in mind.
Now, in this separate discussion: I agree that executing a pivotal act with just a narrow, safe, superintelligence is a difficult problem. That said, all paths to a state of safety from AGI that I can think of seem to contain difficult steps, so I think a more fine-grained analysis of the difficulty of various steps would be needed. I broadly agree with your description of the political character of pivotal acts, but I disagree with what you claim about associated race dynamics — it seems plausible to me that if pivotal acts became the main paradigm, then we’d have a world in which a majority of relevant people are willing to cooperate / do not want to race that much against others in the majority, and it’d mostly be a race between this group and e/acc types. I would also add, though, that the kinds of governance solutions/mechanisms I can think of that are sufficient to (for instance) make it impossible to perform distributed training runs on consumer devices also seem quite authoritarian.
it seems plausible to me that if pivotal acts became the main paradigm, then we’d have a world in which a majority of relevant people are willing to cooperate / do not want to race that much against others in the majority, and it’d mostly be a race between this group and e/acc types
I disagree, I think in many ways the current race already seems motivated by something of the sort—“if I don’t get to it first, they will, and they’re sure to fuck it up”. Though with no apparent planning for pivotal acts in sight (but who knows).
I would also add, though, that the kinds of governance solutions/mechanisms I can think of that are sufficient to (for instance) make it impossible to perform distributed training runs on consumer devices also seem quite authoritarian.
Oh, agreed. It’s a choice between shitty options all around.
I don’t think this revolutionises my argument. First, there’s a lot of talking about example possible pivotal acts and they’re mostly just not that believable on their own. The typical “melt all GPUs” is obviously incredibly hostile and disruptive, but yes, of course, it’s only an example. The problem is that without an actual outline for what a perfect pivotal act is, you can’t even hope to do it with “just” a narrow superintelligence, because in that case, you need to work out the details yourself, and the details are likely horribly complicated.
But the core, fundamental problem with the “pivotal act” notion is that it tries to turn a political problem into a technological one. “Do not build AGIs” is fundamentally a political problem: it’s about restricting human freedom. Now you can either do that voluntarily, by consensus, with some enforcement mechanism for the majority to impose its will on the minority, or you can do that by force, with a minority using overwhelming power to make the majority go along even against their will. That’s it. A pivotal act is just a nice name for the latter thing. The essence of the notion is “we can’t get everyone on board quickly enough; therefore, we should just build some kind of superweapon that allows us to stop everyone else from building unsafe AGI as we define it, whether they like it or not”. It’s not a lethal weapon, and you can argue the utilitarian trade-off from your viewpoint is quite good, but it is undeniably a weapon. And therefore it’s just not something that can be politically acceptable because people don’t like to have weapons pointed at them, not even when the person making the weapon assures them it’s for their own good. If “pivotal act” became the main paradigm the race dynamics would only intensify because then everyone knows they’ll only have one shot and they won’t trust the others to either get it right or actually limit themselves to just the pivotal act once they’re the only ones with AI power in the world. And if instead the world came together to agree on a pivotal act… well that’s just regulation, first, as described in this post. And then moving on to develop a kind of special nanobot police to enforce that regulation (which would still be a highly controversial action, and if deployed worldwide, essentially an act of war against any country not subscribing to the AI safety treaty or whatever).
I was just claiming that your description of pivotal acts / of people that support pivotal acts was incorrect in a way that people that think pivotal acts are worth considering would consider very significant and in a way that significantly reduces the power of your argument as applying to what people mean by pivotal acts — I don’t see anything in your comment as a response to that claim. I would like it to be a separate discussion whether pivotal acts are a good idea with this in mind.
Now, in this separate discussion: I agree that executing a pivotal act with just a narrow, safe, superintelligence is a difficult problem. That said, all paths to a state of safety from AGI that I can think of seem to contain difficult steps, so I think a more fine-grained analysis of the difficulty of various steps would be needed. I broadly agree with your description of the political character of pivotal acts, but I disagree with what you claim about associated race dynamics — it seems plausible to me that if pivotal acts became the main paradigm, then we’d have a world in which a majority of relevant people are willing to cooperate / do not want to race that much against others in the majority, and it’d mostly be a race between this group and e/acc types. I would also add, though, that the kinds of governance solutions/mechanisms I can think of that are sufficient to (for instance) make it impossible to perform distributed training runs on consumer devices also seem quite authoritarian.
I disagree, I think in many ways the current race already seems motivated by something of the sort—“if I don’t get to it first, they will, and they’re sure to fuck it up”. Though with no apparent planning for pivotal acts in sight (but who knows).
Oh, agreed. It’s a choice between shitty options all around.