In fact, before you get to AGI, your company will probably develop other surprising capabilities, and you can demonstrate those capabilities to neutral-but-influential outsiders who previously did not believe those capabilities were possible or concerning. In other words, outsiders can start to help you implement helpful regulatory ideas...
It is not for lack of regulatory ideas that the world has not banned gain-of-function research.
It is not for lack of demonstration of scary gain-of-function capabilities that the world has not banned gain-of-function research.
What exactly is the model by which some AI organization demonstrating AI capabilities will lead to world governments jointly preventing scary AI from being built, in a world which does not actually ban gain-of-function research?
(And to be clear: I’m not saying that gain-of-function research is a great analogy. Gain-of-function research is a much easier problem, because the problem is much more legible and obvious. People know what plagues look like and why they’re scary. In AI, it’s the hard-to-notice problems which are the central issue. Also, there’s no giant economic incentive for gain-of-function research.)
Gain of Function Ban as Practice-Run/Learning for relevant AI Bans
I have heard vague-musings-of-plans in the direction of “get the world to successfully ban Gain of Function research, as a practice-case for getting the world to successfully ban dangerous AI.”
I have vague memories of the actual top bio people around not being too focused on this, because they thought there were easier ways to make progress on biosecurity. (I may be conflating a few different statements – they might have just critiquing a particular strategy I mentioned for banning Gain of Function research)
A few considerations for banning Gain of Function research for AI-related reasons:
because you will gain skills / capacities that transfer into banning relevant AI systems. (i.e. “you’ll learn what works”)
you won’t learn what works, but you’ll hit a bunch of brick walls that teaches you what doesn’t work.
A gain of function ban is “lower stakes” (biorisk is way less likely to kill everyone than AI), and (hopefully?) won’t have many side effects that specifically make it harder to ban AI-stuff later. (By contrast, if you try ineffectually to regulate AI in some way, you will cause the AI industry to raise it’s hackles, and maybe cause one political party to get a reputation as “the anti-AI party”, causing the other party to become “anti-anti-AI” in response, or maybe you will get everyone’s epistemics about what’s worth banning all clouded.
Distinction between “Regulating AI is possible/impossible” vs “pivotal act framing is harmful/unharmful”.
I currently believe John’s point of “man, sure seems real hard to actually usefully regulate things even when it’s comparatively easy, I don’t feel that hopeful about regulation processes working.”
But, that doesn’t necessarily contradict the point that it’s really hard to build an organization capable of unilaterally implementing a pivotal act, and that the process of doing is likely to create enemies, erode coordination-fabric, make people fearful, etc.
It seems obvious to me that the arguments in this post are true-to-some-degree. There’s some actual math / hashing-out that I haven’t seen to my satisfaction of how all the arguments actually balance against each toher.
Something feels off about the way people relate to “everyone else’s ideas seeming more impossible than mine own, even if we all agree it’s pretty impossible.”
+1 to the distinction between “Regulating AI is possible/impossible” vs “pivotal act framing is harmful/unharmful”.
I’m sympathetic to a view that says something like “yeah, regulating AI is Hard, but it’s also necessary because a unilateral pivotal act would be Bad”. (TBC, I’m not saying I agree with that view, but it’s at least coherent and not obviously incompatible with how the world actually works.) To properly make that case, one has to argue some combination of:
A unilateral pivotal act would be so bad that it’s worth accepting a much higher chance of human extinction in order to avoid it, OR
Aiming for a unilateral pivotal act would not reduce the chance of human extinction much more than aiming for a multilateral pivotal act
I generally expect people opposed to the pivotal act framing to have the latter in mind rather than the former. The obvious argument that aiming for a unilateral pivotal act does reduce the chance of human extinction much more than aiming for a multilateral pivotal act is that it’s much more likely that someone could actually perform a unilateral pivotal act; it is a far easier problem, even after accounting for the problems the OP mentions in Part 1. That, I think, is the main view one would need to argue against in order to make the case for multilateral over unilateral pivotal act as a goal. The OP doesn’t really make that case at all; it argues that aiming for unilateral introduces various challenges, but it doesn’t even attempt to argue that those challenges would be harder than (or even comparably hard to) getting all the major world governments to jointly implement an actually-effective pivotal act.
John, it seems like you’re continuing to make the mistake-according-to-me of analyzing the consequences of a pivotal act without regard for the consequences of the intentions leading up to the act. The act can’t come out of a vacuum, and you can’t built a project compatible with the kind of invasive pivotal acts I’m complaining about without causing a lot of problems leading up to the act, including triggering a lot of fear and panic for other labs and institutions. To summarize from the post title: pivotal act intentions directly have negative consequences fox x-safety, and people thinking about the acts alone seem to be ignoring the consequences of the intentions leading up to the act, which is a fallacy.
I see the argument you’re making there. I still think my point stands: the strategically relevant question is not whether unilateral pivotal act intentions will cause problems, the question is whether aiming for a unilateral pivotal act would or would not reduce the chance of human extinction much more than aiming for a multilateral pivotal act. The OP does not actually attempt to compare the two, it just lists some problems with aiming for a unilateral pivotal act.
I do think that aiming for a unilateral act increases the chance of successfully executing the pivotal act by multiple orders of magnitude, even accounting for the part where other players react to the intention, and that completely swamps the other considerations.
Just as a related idea, in my mind, I often do a kind of thinking that HPMOR!Harry would call “Hufflepuff Bones”, where I look for ways a problem is solvable in physical reality at all, before considering ethical and coordination and even much in the way of practical concerns.
it’s much more likely that someone could actually perform a unilateral pivotal act; it is a far easier problem, even after accounting for the problems the OP mentions in Part 1.
What I’ve never understood about the pivotal act plan is exactly what the successful AGI team is supposed to do after melting the GPUs or whatever. Every government on Earth will now consider them their enemy; they will immediately be destroyed unless they can defend themselves militarily, then countries will simply rebuild the GPU factories and continue on as before(except now in a more combative, disrupted, AI-race-encouraging geopolitical situation). So any pivotal act seems to require, at a minimum, an AI capable of militarily defeating all countries’ militaries. Then in order to not have society collapse, you probably need to become the government yourself, or take over or persuade existing governments to go along with your agenda. But an AGI that would be capable of doing all this safely seems...not much easier to create than a full-on FAI? It’s not like you could get by with an AI that was freakishly skilled at designing nanomachines but nothing else, you’d need something much more general. But isn’t the whole idea of the pivotal act plan that you don’t need to solve alignment in full generality to execute a pivotal act? For these reasons, executing a unilateral pivotal act(that actually results in an x-risk reduction) does not seem obviously easier than convincing governments to me.
Oh, melting the GPUs would not actually be a pivotal act. There would need to be some way to prevent new GPUs from being built in order for it to be a pivotal act.
Military capability is not strictly necessary; a pivotal act need not necessarily piss off world governments. AGI-driven propaganda, for instance, might avoid that.
Alternatively, an AGI could produce nanomachines which destroy GPUs, are extremely hard to eradicate, but otherwise don’t do much of anything.
(Note that these aren’t intended to be very good/realistic suggestions, they’re just meant to point to different dimensions of the possibility space.)
Oh, melting the GPUs would not actually be a pivotal act
Well yeah, that’s my point. It seems to me that any pivotal act worthy of the name would essentially require the AI team to become an AGI-powered world government, which seems pretty darn difficult to pull off safely. The superpowered-AI-propaganda plan falls under this category. The long-lasting nanomachines idea is cute, but I bet people would just figure out ways to evade the nanomachines’ definition of ‘GPU’.
Note that these aren’t intended to be very good/realistic suggestions, they’re just meant to point to different dimensions of the possibility space
Fair enough...but if the pivotal act plan is workable, there should be some member of that space which actually is good/seems like it has a shot of working out in reality(and which wouldn’t require a full FAI). I’ve never heard any and am having a hard time thinking of one. Now it could be that MIRI or others think they have a workable plan which they don’t want to share the details of due to infohazard concerns. But as an outside observer, I have to assign a certain amount of probability to that being self-delusion.
Well yeah, that’s my point. It seems to me that any pivotal act worthy of the name would essentially require the AI team to become an AGI-powered world government, which seems pretty darn difficult to pull off safely. The superpowered-AI-propaganda plan falls under this category.
Yeah. I think this sort of thing is why Eliezer thinks we’re doomed – getting the humanity to coordinate collectively seems doomed (i.e. see Gain of Function Research), and there are no weak pivotal acts that aren’t basically impossible to execute safely.
The nanomachine gpu-melting pivotal act is meant to be a gesture at the difficulty / power level, not an actual working example. The other gestured-example I’ve heard is “upload aligned people who think hard for 1000 subjective years and hopefully figure something out.” I’ve heard someone from MIRI argue that one is also unworkable but wasn’t sure on the exact reasons.
The other gestured-example I’ve heard is “upload aligned people who think hard for 1000 subjective years and hopefully figure something out.” I’ve heard someone from MIRI argue that one is also unworkable but wasn’t sure on the exact reasons.
Standard counterargument to that one is “by the time we can do that we’ll already have beyond-human AI capabilities (since running humans is a lower bound on what AI can do), and therefore foom”.
You could have another limited AI design a nanofactory to make ultra-fast computers to run the emulations. I think a more difficult problem is getting a limited AI to do neuroscience well. Actually I think this whole scenario is kind of silly, but given the implausible premise of a single AI lab having a massive tech lead over all others, neuroscience may be the bigger barrier.
Yeah. I think this sort of thing is why Eliezer thinks we’re doomed
Hmm, interesting...but wasn’t he more optimistic a few years ago, when his plan was still “pull off a pivotal act with a limited AI”? I thought the thing that made him update towards doom was the apparent difficulty of safely making even a limited AI, plus shorter timelines.
other gestured-example I’ve heard is “upload aligned people who think hard for 1000 subjective years and hopefully figure something out.”
Ah, that actually seems like it might work. I guess the problem is that an AI that can competently do neuroscience well enough to do this would have to be pretty general. Maybe a more realistic plan along the same lines might be to try using ML to replicate the functional activity of various parts of the human brain and create ‘pseudo-uploads’. Or just try to create an AI with similar architecture and roughly-similar reward function to us, hoping that human values are more generic than they might appear.
It seems relatively plausible that you could use a Limited AGI to build a nanotech system capable of uploading a diverse assortment of (non-brain, or maybe only very small brains) living tissue without damaging them, and that this system would learn how to upload tissue in a general way. Then you could use the system (not the AGI) to upload humans (tested on increasingly complex animals). It would be a relatively inefficient emulation, but it doesn’t seem obviously doomed to me.
Probably too late once hardware is available to do this though.
Followup point on the Gain-of-Function-Ban as practice-run for AI:
My sense is that the biorisk people who were thinking about Gain-of-Function-Ban were not primarily modeling it as a practice run for regulating AGI. This may result in them not really prioritizing it.
I think biorisk is significantly lower than AGI risk, so if it’s tractable and useful to regulate Gain of Function research as a practice run for regulating AGI, it’s plausible this is actually much more important than business-as-usual biorisk.
BUT I think smart people I know seem to disagree about how any of this works, so the “if tractable and useful” conditional is pretty non-obvious to me.
If bio-and-AI-people haven’t had a serious conversation about this where they mapped out the considerations in more detail, I do think that should happen.
What if someone proposed banning GoF research via the following way: release a harmless virus which is engineered to make people more amenable to banning GoF research. That’s what a pivotal act proposal looks like me. You were supposed to destroy the dark side, not join them!
Slightly stronger analogy: release a harmless virus which grants universal immunity to other viruses. That’s what a pivotal act proposal looks like.
This idea has various technical problems, but let’s pretend that those are solved for the sake of this discussion; the interesting question is whether one ought to deploy such a virus if it were technically feasible and worked as advertised. I claim the answer is “yes, obviously”. Waiting around for government bureaucracies to approve it as a treatment and deploy it through the traditional medical system would take a decade, optimistically. Probably multiple decades before it reaches a majority of the population (if ever). How many people die of viruses every ten years?
You were supposed to destroy the dark side, not join them!
I sometimes advise people that it is useful to self-identify as a villain. The central reason is that Good Is Dumb:
A common recurring thread in fiction is the idea that the hero, for various reasons, will engage in stupid or illogical actions largely because it is the “heroic”, sometimes idealistic thing to do.
Problem is, peoples’ sense of what’s “good” tends to draw rather heavily on fictional portrayals of good and bad, so people trying to be “good” end up mixing in a large dose of outright stupidity. No fictional hero ever sacrifices one bystander to save ten.
Objecting to pivotal acts on the basis of “you were supposed to destroy the dark side, not join them” sounds to me like a very central example of Good Is Dumb. Like, sure, one can make up reasonable reasons to oppose the pivotal act, but I think that the actual original objection for most people is probably “this sounds like something a villain would do, not something a hero would do”.
I sometimes advise people that it is useful to self-identify as a villain...
Perhaps “antihero” is better here? The “heroic” tend to be stupid and rely on the laws of narrative saving them. Villains tend to have exciting/intricate/dastardly… but overcomplicated and fatally flawed plans.
My first thought on “No fictional hero ever sacrifices one bystander to save ten”, was of Zakalwe (use of weapons) - but of course he’s squarely in antihero territory.
Does an organization’s ability to execute a “pivotal act” overlap with Samo Burja’s idea of organizations as “live players”? How many are there, and are there any orgs that you would place in one category and not the other?
My current belief is that there is no organization has the know-how to execute a pivotal act, so I would place all live orgs in one category and not the other.
That’s fair. Maybe I was more trying to get at the chances that current live orgs will develop this know-how, or if it would require new orgs designed with that purpose.
There are/could be crucial differences between GoF and some AGI examples.
Eg, a convincing demonstration of the ability to overthrow the government. States are also agents, also have convergent instrumental goals. GoF research seems much more threatening to individual humans, but not that much threatening to states or governments.
I think GoF research can also be quite threatening to states. COVID-19 has stressed the politics and economies of both the US and China (among others). Imagine the effects of a disease significantly deadlier.
It is not for lack of regulatory ideas that the world has not banned gain-of-function research.
It is not for lack of demonstration of scary gain-of-function capabilities that the world has not banned gain-of-function research.
What exactly is the model by which some AI organization demonstrating AI capabilities will lead to world governments jointly preventing scary AI from being built, in a world which does not actually ban gain-of-function research?
(And to be clear: I’m not saying that gain-of-function research is a great analogy. Gain-of-function research is a much easier problem, because the problem is much more legible and obvious. People know what plagues look like and why they’re scary. In AI, it’s the hard-to-notice problems which are the central issue. Also, there’s no giant economic incentive for gain-of-function research.)
Various thoughts that this inspires:
Gain of Function Ban as Practice-Run/Learning for relevant AI Bans
I have heard vague-musings-of-plans in the direction of “get the world to successfully ban Gain of Function research, as a practice-case for getting the world to successfully ban dangerous AI.”
I have vague memories of the actual top bio people around not being too focused on this, because they thought there were easier ways to make progress on biosecurity. (I may be conflating a few different statements – they might have just critiquing a particular strategy I mentioned for banning Gain of Function research)
A few considerations for banning Gain of Function research for AI-related reasons:
because you will gain skills / capacities that transfer into banning relevant AI systems. (i.e. “you’ll learn what works”)
you won’t learn what works, but you’ll hit a bunch of brick walls that teaches you what doesn’t work.
A gain of function ban is “lower stakes” (biorisk is way less likely to kill everyone than AI), and (hopefully?) won’t have many side effects that specifically make it harder to ban AI-stuff later. (By contrast, if you try ineffectually to regulate AI in some way, you will cause the AI industry to raise it’s hackles, and maybe cause one political party to get a reputation as “the anti-AI party”, causing the other party to become “anti-anti-AI” in response, or maybe you will get everyone’s epistemics about what’s worth banning all clouded.
Distinction between “Regulating AI is possible/impossible” vs “pivotal act framing is harmful/unharmful”.
I currently believe John’s point of “man, sure seems real hard to actually usefully regulate things even when it’s comparatively easy, I don’t feel that hopeful about regulation processes working.”
But, that doesn’t necessarily contradict the point that it’s really hard to build an organization capable of unilaterally implementing a pivotal act, and that the process of doing is likely to create enemies, erode coordination-fabric, make people fearful, etc.
It seems obvious to me that the arguments in this post are true-to-some-degree. There’s some actual math / hashing-out that I haven’t seen to my satisfaction of how all the arguments actually balance against each toher.
Something feels off about the way people relate to “everyone else’s ideas seeming more impossible than mine own, even if we all agree it’s pretty impossible.”
+1 to the distinction between “Regulating AI is possible/impossible” vs “pivotal act framing is harmful/unharmful”.
I’m sympathetic to a view that says something like “yeah, regulating AI is Hard, but it’s also necessary because a unilateral pivotal act would be Bad”. (TBC, I’m not saying I agree with that view, but it’s at least coherent and not obviously incompatible with how the world actually works.) To properly make that case, one has to argue some combination of:
A unilateral pivotal act would be so bad that it’s worth accepting a much higher chance of human extinction in order to avoid it, OR
Aiming for a unilateral pivotal act would not reduce the chance of human extinction much more than aiming for a multilateral pivotal act
I generally expect people opposed to the pivotal act framing to have the latter in mind rather than the former. The obvious argument that aiming for a unilateral pivotal act does reduce the chance of human extinction much more than aiming for a multilateral pivotal act is that it’s much more likely that someone could actually perform a unilateral pivotal act; it is a far easier problem, even after accounting for the problems the OP mentions in Part 1. That, I think, is the main view one would need to argue against in order to make the case for multilateral over unilateral pivotal act as a goal. The OP doesn’t really make that case at all; it argues that aiming for unilateral introduces various challenges, but it doesn’t even attempt to argue that those challenges would be harder than (or even comparably hard to) getting all the major world governments to jointly implement an actually-effective pivotal act.
John, it seems like you’re continuing to make the mistake-according-to-me of analyzing the consequences of a pivotal act without regard for the consequences of the intentions leading up to the act. The act can’t come out of a vacuum, and you can’t built a project compatible with the kind of invasive pivotal acts I’m complaining about without causing a lot of problems leading up to the act, including triggering a lot of fear and panic for other labs and institutions. To summarize from the post title: pivotal act intentions directly have negative consequences fox x-safety, and people thinking about the acts alone seem to be ignoring the consequences of the intentions leading up to the act, which is a fallacy.
I see the argument you’re making there. I still think my point stands: the strategically relevant question is not whether unilateral pivotal act intentions will cause problems, the question is whether aiming for a unilateral pivotal act would or would not reduce the chance of human extinction much more than aiming for a multilateral pivotal act. The OP does not actually attempt to compare the two, it just lists some problems with aiming for a unilateral pivotal act.
I do think that aiming for a unilateral act increases the chance of successfully executing the pivotal act by multiple orders of magnitude, even accounting for the part where other players react to the intention, and that completely swamps the other considerations.
Just as a related idea, in my mind, I often do a kind of thinking that HPMOR!Harry would call “Hufflepuff Bones”, where I look for ways a problem is solvable in physical reality at all, before considering ethical and coordination and even much in the way of practical concerns.
What I’ve never understood about the pivotal act plan is exactly what the successful AGI team is supposed to do after melting the GPUs or whatever. Every government on Earth will now consider them their enemy; they will immediately be destroyed unless they can defend themselves militarily, then countries will simply rebuild the GPU factories and continue on as before(except now in a more combative, disrupted, AI-race-encouraging geopolitical situation). So any pivotal act seems to require, at a minimum, an AI capable of militarily defeating all countries’ militaries. Then in order to not have society collapse, you probably need to become the government yourself, or take over or persuade existing governments to go along with your agenda. But an AGI that would be capable of doing all this safely seems...not much easier to create than a full-on FAI? It’s not like you could get by with an AI that was freakishly skilled at designing nanomachines but nothing else, you’d need something much more general. But isn’t the whole idea of the pivotal act plan that you don’t need to solve alignment in full generality to execute a pivotal act? For these reasons, executing a unilateral pivotal act(that actually results in an x-risk reduction) does not seem obviously easier than convincing governments to me.
Oh, melting the GPUs would not actually be a pivotal act. There would need to be some way to prevent new GPUs from being built in order for it to be a pivotal act.
Military capability is not strictly necessary; a pivotal act need not necessarily piss off world governments. AGI-driven propaganda, for instance, might avoid that.
Alternatively, an AGI could produce nanomachines which destroy GPUs, are extremely hard to eradicate, but otherwise don’t do much of anything.
(Note that these aren’t intended to be very good/realistic suggestions, they’re just meant to point to different dimensions of the possibility space.)
Well yeah, that’s my point. It seems to me that any pivotal act worthy of the name would essentially require the AI team to become an AGI-powered world government, which seems pretty darn difficult to pull off safely. The superpowered-AI-propaganda plan falls under this category. The long-lasting nanomachines idea is cute, but I bet people would just figure out ways to evade the nanomachines’ definition of ‘GPU’.
Fair enough...but if the pivotal act plan is workable, there should be some member of that space which actually is good/seems like it has a shot of working out in reality(and which wouldn’t require a full FAI). I’ve never heard any and am having a hard time thinking of one. Now it could be that MIRI or others think they have a workable plan which they don’t want to share the details of due to infohazard concerns. But as an outside observer, I have to assign a certain amount of probability to that being self-delusion.
Yeah. I think this sort of thing is why Eliezer thinks we’re doomed – getting the humanity to coordinate collectively seems doomed (i.e. see Gain of Function Research), and there are no weak pivotal acts that aren’t basically impossible to execute safely.
The nanomachine gpu-melting pivotal act is meant to be a gesture at the difficulty / power level, not an actual working example. The other gestured-example I’ve heard is “upload aligned people who think hard for 1000 subjective years and hopefully figure something out.” I’ve heard someone from MIRI argue that one is also unworkable but wasn’t sure on the exact reasons.
Standard counterargument to that one is “by the time we can do that we’ll already have beyond-human AI capabilities (since running humans is a lower bound on what AI can do), and therefore foom”.
You could have another limited AI design a nanofactory to make ultra-fast computers to run the emulations. I think a more difficult problem is getting a limited AI to do neuroscience well. Actually I think this whole scenario is kind of silly, but given the implausible premise of a single AI lab having a massive tech lead over all others, neuroscience may be the bigger barrier.
Hmm, interesting...but wasn’t he more optimistic a few years ago, when his plan was still “pull off a pivotal act with a limited AI”? I thought the thing that made him update towards doom was the apparent difficulty of safely making even a limited AI, plus shorter timelines.
Ah, that actually seems like it might work. I guess the problem is that an AI that can competently do neuroscience well enough to do this would have to be pretty general. Maybe a more realistic plan along the same lines might be to try using ML to replicate the functional activity of various parts of the human brain and create ‘pseudo-uploads’. Or just try to create an AI with similar architecture and roughly-similar reward function to us, hoping that human values are more generic than they might appear.
It seems relatively plausible that you could use a Limited AGI to build a nanotech system capable of uploading a diverse assortment of (non-brain, or maybe only very small brains) living tissue without damaging them, and that this system would learn how to upload tissue in a general way. Then you could use the system (not the AGI) to upload humans (tested on increasingly complex animals). It would be a relatively inefficient emulation, but it doesn’t seem obviously doomed to me.
Probably too late once hardware is available to do this though.
Followup point on the Gain-of-Function-Ban as practice-run for AI:
My sense is that the biorisk people who were thinking about Gain-of-Function-Ban were not primarily modeling it as a practice run for regulating AGI. This may result in them not really prioritizing it.
I think biorisk is significantly lower than AGI risk, so if it’s tractable and useful to regulate Gain of Function research as a practice run for regulating AGI, it’s plausible this is actually much more important than business-as-usual biorisk.
BUT I think smart people I know seem to disagree about how any of this works, so the “if tractable and useful” conditional is pretty non-obvious to me.
If bio-and-AI-people haven’t had a serious conversation about this where they mapped out the considerations in more detail, I do think that should happen.
What if someone proposed banning GoF research via the following way: release a harmless virus which is engineered to make people more amenable to banning GoF research. That’s what a pivotal act proposal looks like me. You were supposed to destroy the dark side, not join them!
Slightly stronger analogy: release a harmless virus which grants universal immunity to other viruses. That’s what a pivotal act proposal looks like.
This idea has various technical problems, but let’s pretend that those are solved for the sake of this discussion; the interesting question is whether one ought to deploy such a virus if it were technically feasible and worked as advertised. I claim the answer is “yes, obviously”. Waiting around for government bureaucracies to approve it as a treatment and deploy it through the traditional medical system would take a decade, optimistically. Probably multiple decades before it reaches a majority of the population (if ever). How many people die of viruses every ten years?
I sometimes advise people that it is useful to self-identify as a villain. The central reason is that Good Is Dumb:
Problem is, peoples’ sense of what’s “good” tends to draw rather heavily on fictional portrayals of good and bad, so people trying to be “good” end up mixing in a large dose of outright stupidity. No fictional hero ever sacrifices one bystander to save ten.
Objecting to pivotal acts on the basis of “you were supposed to destroy the dark side, not join them” sounds to me like a very central example of Good Is Dumb. Like, sure, one can make up reasonable reasons to oppose the pivotal act, but I think that the actual original objection for most people is probably “this sounds like something a villain would do, not something a hero would do”.
Perhaps “antihero” is better here? The “heroic” tend to be stupid and rely on the laws of narrative saving them. Villains tend to have exciting/intricate/dastardly… but overcomplicated and fatally flawed plans.
My first thought on “No fictional hero ever sacrifices one bystander to save ten”, was of Zakalwe (use of weapons) - but of course he’s squarely in antihero territory.
Does an organization’s ability to execute a “pivotal act” overlap with Samo Burja’s idea of organizations as “live players”? How many are there, and are there any orgs that you would place in one category and not the other?
My current belief is that there is no organization has the know-how to execute a pivotal act, so I would place all live orgs in one category and not the other.
That’s fair. Maybe I was more trying to get at the chances that current live orgs will develop this know-how, or if it would require new orgs designed with that purpose.
(edited)
There are/could be crucial differences between GoF and some AGI examples.
Eg, a convincing demonstration of the ability to overthrow the government. States are also agents, also have convergent instrumental goals. GoF research seems much more threatening to individual humans, but not that much threatening to states or governments.
I think GoF research can also be quite threatening to states. COVID-19 has stressed the politics and economies of both the US and China (among others). Imagine the effects of a disease significantly deadlier.