So, I think the difference is that ASI is ultimately far more difficult to prove on either side. However, the general framework maps pretty similarly.
Allow me to compare ASI with another X-risk scenario we’re familiar with, cold war MAD. The general argument goes:
The Cold War argument is:
(1) USSR improves and builds thousands of non-hypersonic nuclear tipped missiles. (did actually happen)
(2) USSR decides to risk nuclear annihilation by killing all it’s rivals on the planet
(3) due to miscalculations, perceived nuclear attack, and/or security threats, USSR gains:
(a) credible (or whatever passes for credible in that paranoid era) evidence they’re getting nuked
(4) at a certain point, Russia determines that today is the day to launch the nukes, and everyone dies
What’s the difference between this and hypersonics, or ASI? Ultimately, even if Washington and Moscow sat down and tried to give an accurate assessment of P(Armageddon) I doubt they’d have succeeded in producing an accurate estimate. The narrative is difficult to prove or disprove, all we know was that we came close (see Cuban missile crisis) but it never actually happened.
The issue for hypersonics isn’t the framework, it’s that the narrative itself fails to stand up to scrutiny (see my explanation). We know for a fact that those probabilities are extraordinarily unlikely. NATO doesn’t leave coordinates to nuclear launch sites lying around! Governments take nuclear threats very seriously! Unlike in the cold war I’d consider this narrative easily disprovable.
I have flagrantly disregarded relevant evidence suggesting that point 3 doesn’t happen.
With ASI we’re more or less completely in the dark. You can’t really verify if a point is ‘obviously not going to happen’, to the best of my understanding. Sure, you can say ‘probably’ or ‘probably not’, but you’d have to be the judge of that. There is less empirical evidence (that you presented, anyways) in regards to ASI being legitimate or not legitimate.
Is there an argument suggesting that ASI X risk is highly unlikely? I think it probably does exist, but then there may be rebuttals to that. Without full context it’s difficult to judge.
That said, this only applies to the ASI argument as you presented it. I’m sure my assessment will vary based off who and how the argument is presented, and what evidence is cited. But to the best of my understanding your ASI argument as presented is improvable on both sides. I could call it ultra-BS, but I think speculation is just as accurate a descriptor. To make it more than that you’ll need to cite evidence and address counterarguments, that’s what distinguished a good theory from BS and speculation.
So I think what you are saying is an ultra-BS argument is one that you know is obviously wrong. ASI doom or acceleration arguments are speculation where we don’t know which argument is wrong, since we don’t have access to an ASI. While for example we do know it’s difficult to locate a quiet submarine, it’s difficult to stop even subsonic bombers, it’s difficult to react in time to a large missile attack, and we have direct historical examples of all this in non-nuclear battles with the same weapons. For example the cruise missiles in Ukraine that keep impacting both side’s positions are just a warhead swap from being nuclear.
If you don’t know, you cannot justify a policy of preemptive nuclear war over AI. That’s kinda my point. I’m not even trying to say, object level, whether or not ASI actually will be a threat humans need to be willing to go to nuclear war over. I am saying the evidence right now does not support that conclusion. (it doesn’t support the conclusion that ASI is safe either, but it doesn’t justify the most extreme policy action)
So I think what you are saying is an ultra-BS argument is one that you know is obviously wrong.
Yep, pretty much. Part of the technique is knowing the ins and outs of our own argument. As I use ultra-BS prominently in debate, I need to be able to rebut the argument when I’m inevitably forced to argue the other side. I thus draw the distinction between ultra-BS along these lines. If it’s not obviously wrong (to me, anyways) it’s speculation. I can thus say that extended Chinese real economic stagnation for the next 10 years is educated speculation, while imminent Chinese economic collapse is ultra-BS.
If you don’t know, you cannot justify a policy of preemptive nuclear war over AI. That’s kinda my point. I’m not even trying to say, object level, whether or not ASI actually will be a threat humans need to be willing to go to nuclear war over. I am saying the evidence right now does not support that conclusion. (it doesn’t support the conclusion that ASI is safe either, but it doesn’t justify the most extreme policy action)
So, this is where I withdraw into acknowledging my limits. I don’t believe I have read sufficient ASI literature to fully understand this point, so I’m not too comfortable offering any object level predictions or narrative assessments. I can agree that many ASI arguments follow the same narrative format as ultra-BS, and there are likely many bad ASI arguments which can be revealed as wrong through careful (or even cursory) research. However, I’m not sufficiently educated on the subject to actually evaluate the narrative, thus the unsatisfactory response of ‘I’m not sure, sorry’.
However, if your understanding of ASI is correct, and there indeed is insufficient provable evidence, then yes, I can agree ASI policies cannot be argued for with provable evidence. Note again, however, that this would essentially be me taking your word for everything, which I’m not comfortable doing.
Currently, my priors on ASI ruin are limited, and I’ll likely need to do more specific research on the topic.
However, if your understanding of ASI is correct, and there indeed is insufficient provable evidence, then yes, I can agree ASI policies cannot be argued for with provable evidence. Note again, however, that this would essentially be me taking your word for everything, which I’m not comfortable doing.
So in this particular scenario, those concerned about ASI doom aren’t asking for a small or reasonable policy action proportional to today’s uncertainty. They are asking for AI pauses and preemptive nuclear war.
AI pauses will cost an enormous amount of money, some of which is tax revenue.
Preemptive nuclear war is potential suicide. It’s asking for a country to risk the deaths of approximately 50% of it’s population in the near term, and to lose all it’s supply chains, turning it into broken third world country separated by radioactive craters on all the transit and food supply hubs, which would likely kill a large fraction of it’s remaining citizens.
To justify (1) you would need to have some level of evidence that the threat exists. To justify (2) I would expect you would need beyond a shadow of a doubt evidence that the threat exists.
So for (1) convincing evidence might be a weak ASI that is hostile needs to exist in the lab before the threat can be claimed to be real. For (2) researchers would need to have produced in an isolated lab strong ASI, demonstrated that they were hostile, and tried thousands of times to make a safe ASI with a 100% failure rate.
I think we could argue about the exact level of evidence needed, or briefly establish plausible ways that (1) and (2) could fail to show a threat, but in general I would say the onus is on AI doom advocates to prove the threat is real, not on advocates for “business as usual” technology development to prove it is not. I think this last part is the dark arts scam, that and other hidden assumptions that get treated as certainty. (a lot of the hidden assumptions are in the technical details of how an ASI is assumed to work by someone with less detailed technical knowledge, vs the way actual ML systems work today)
Another part of the scam is the whole calling this “rational”. If your evidence on any topic is uncertain, and you can’t prove your point, certainty is unjustified, and it’s not a valid “agree to disagree” opinion. See: https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem .
So with this said, it seems like all I would need to do is show with a cite that ASI don’t exist yet, and show with a cite a reason, any reason at all, that plausibly could mean ASI are unable to be a threat. I don’t have to prove the reason is anything but plausible.
It does bother me that my proposal for proving ASI might not be a threat is suspiciously similar to how tobacco companies delayed any action to ban cigarettes essentially forever, but first they started with shoddy science to show that maybe the cigarettes weren’t the reason people were dying. Or how fossil fuel advocates have pulled the same scam, amplifying any doubts over climate change and thus delaying meaningful action for decades. (meaningful action is to research alternatives, which did succeed, but also to price carbon, which https://www.barrons.com/articles/europe-carbon-tax-emissions-climate-policy-1653e360 doesn’t even start until 2026, 50 years after the discovery of climate change)
These historical examples lead to a conclusion as well, I will see if you realize what this means for AI.
Thanks for the update! I think this is probably something important to take into consideration when evaluating ASI arguments.
That said, I think we’re starting to stray from the original topic of the Dark Arts, as we’re focusing more on ASI specifically rather than the Dark Arts element of it. In the interest of maintaining discussion focus on this post, would you agree to continuing AGI discussion in private messages?
And I was trying to focus on the dark arts part of the arguments. Note I don’t make any arguments about ASI in the above, just state that fairly weak evidence should be needed to justify not doing anything drastic about it at this time, because the drastic actions have high measurable costs. It’s not provable at present to state that “ASI could find a way to take over the planet with limited resources, because we don’t have an ASI or know the intelligence ROI on a given amount of flops”, but it is provable to state that “an AI pause of 6 months would cost tens of billions, possibly hundreds of billions of dollars and would reduce the relative power of the pausing countries internationally”. It’s also provable to state the damage of a nuclear exchange.
Look how it’s voted down to −10 on agreement : others feel very strongly about this issue.
Interesting question!
So, I think the difference is that ASI is ultimately far more difficult to prove on either side. However, the general framework maps pretty similarly.
Allow me to compare ASI with another X-risk scenario we’re familiar with, cold war MAD. The general argument goes:
What’s the difference between this and hypersonics, or ASI? Ultimately, even if Washington and Moscow sat down and tried to give an accurate assessment of P(Armageddon) I doubt they’d have succeeded in producing an accurate estimate. The narrative is difficult to prove or disprove, all we know was that we came close (see Cuban missile crisis) but it never actually happened.
The issue for hypersonics isn’t the framework, it’s that the narrative itself fails to stand up to scrutiny (see my explanation). We know for a fact that those probabilities are extraordinarily unlikely. NATO doesn’t leave coordinates to nuclear launch sites lying around! Governments take nuclear threats very seriously! Unlike in the cold war I’d consider this narrative easily disprovable.
I have flagrantly disregarded relevant evidence suggesting that point 3 doesn’t happen.
With ASI we’re more or less completely in the dark. You can’t really verify if a point is ‘obviously not going to happen’, to the best of my understanding. Sure, you can say ‘probably’ or ‘probably not’, but you’d have to be the judge of that. There is less empirical evidence (that you presented, anyways) in regards to ASI being legitimate or not legitimate.
Is there an argument suggesting that ASI X risk is highly unlikely? I think it probably does exist, but then there may be rebuttals to that. Without full context it’s difficult to judge.
That said, this only applies to the ASI argument as you presented it. I’m sure my assessment will vary based off who and how the argument is presented, and what evidence is cited. But to the best of my understanding your ASI argument as presented is improvable on both sides. I could call it ultra-BS, but I think speculation is just as accurate a descriptor. To make it more than that you’ll need to cite evidence and address counterarguments, that’s what distinguished a good theory from BS and speculation.
So I think what you are saying is an ultra-BS argument is one that you know is obviously wrong. ASI doom or acceleration arguments are speculation where we don’t know which argument is wrong, since we don’t have access to an ASI. While for example we do know it’s difficult to locate a quiet submarine, it’s difficult to stop even subsonic bombers, it’s difficult to react in time to a large missile attack, and we have direct historical examples of all this in non-nuclear battles with the same weapons. For example the cruise missiles in Ukraine that keep impacting both side’s positions are just a warhead swap from being nuclear.
With that said, isn’t “high confidence” in your speculation by definition wrong? If you don’t know, how can you know that it’s almost certain an ASI will defeat and kill humanity? AGI Ruin: A List of Lethalities and https://twitter.com/ESYudkowsky/status/1658616828741160960 . Of course thermodynamics has taken it’s sweet time building ASI : https://twitter.com/BasedBeffJezos/status/1670640570388336640
If you don’t know, you cannot justify a policy of preemptive nuclear war over AI. That’s kinda my point. I’m not even trying to say, object level, whether or not ASI actually will be a threat humans need to be willing to go to nuclear war over. I am saying the evidence right now does not support that conclusion. (it doesn’t support the conclusion that ASI is safe either, but it doesn’t justify the most extreme policy action)
Yep, pretty much. Part of the technique is knowing the ins and outs of our own argument. As I use ultra-BS prominently in debate, I need to be able to rebut the argument when I’m inevitably forced to argue the other side. I thus draw the distinction between ultra-BS along these lines. If it’s not obviously wrong (to me, anyways) it’s speculation. I can thus say that extended Chinese real economic stagnation for the next 10 years is educated speculation, while imminent Chinese economic collapse is ultra-BS.
So, this is where I withdraw into acknowledging my limits. I don’t believe I have read sufficient ASI literature to fully understand this point, so I’m not too comfortable offering any object level predictions or narrative assessments. I can agree that many ASI arguments follow the same narrative format as ultra-BS, and there are likely many bad ASI arguments which can be revealed as wrong through careful (or even cursory) research. However, I’m not sufficiently educated on the subject to actually evaluate the narrative, thus the unsatisfactory response of ‘I’m not sure, sorry’.
However, if your understanding of ASI is correct, and there indeed is insufficient provable evidence, then yes, I can agree ASI policies cannot be argued for with provable evidence. Note again, however, that this would essentially be me taking your word for everything, which I’m not comfortable doing.
Currently, my priors on ASI ruin are limited, and I’ll likely need to do more specific research on the topic.
So in this particular scenario, those concerned about ASI doom aren’t asking for a small or reasonable policy action proportional to today’s uncertainty. They are asking for AI pauses and preemptive nuclear war.
Pause: https://futureoflife.org/open-letter/pause-giant-ai-experiments/
Nuclear war: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/
AI pauses will cost an enormous amount of money, some of which is tax revenue.
Preemptive nuclear war is potential suicide. It’s asking for a country to risk the deaths of approximately 50% of it’s population in the near term, and to lose all it’s supply chains, turning it into broken third world country separated by radioactive craters on all the transit and food supply hubs, which would likely kill a large fraction of it’s remaining citizens.
To justify (1) you would need to have some level of evidence that the threat exists. To justify (2) I would expect you would need beyond a shadow of a doubt evidence that the threat exists.
So for (1) convincing evidence might be a weak ASI that is hostile needs to exist in the lab before the threat can be claimed to be real. For (2) researchers would need to have produced in an isolated lab strong ASI, demonstrated that they were hostile, and tried thousands of times to make a safe ASI with a 100% failure rate.
I think we could argue about the exact level of evidence needed, or briefly establish plausible ways that (1) and (2) could fail to show a threat, but in general I would say the onus is on AI doom advocates to prove the threat is real, not on advocates for “business as usual” technology development to prove it is not. I think this last part is the dark arts scam, that and other hidden assumptions that get treated as certainty. (a lot of the hidden assumptions are in the technical details of how an ASI is assumed to work by someone with less detailed technical knowledge, vs the way actual ML systems work today)
Another part of the scam is the whole calling this “rational”. If your evidence on any topic is uncertain, and you can’t prove your point, certainty is unjustified, and it’s not a valid “agree to disagree” opinion. See: https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem .
So with this said, it seems like all I would need to do is show with a cite that ASI don’t exist yet, and show with a cite a reason, any reason at all, that plausibly could mean ASI are unable to be a threat. I don’t have to prove the reason is anything but plausible.
It does bother me that my proposal for proving ASI might not be a threat is suspiciously similar to how tobacco companies delayed any action to ban cigarettes essentially forever, but first they started with shoddy science to show that maybe the cigarettes weren’t the reason people were dying. Or how fossil fuel advocates have pulled the same scam, amplifying any doubts over climate change and thus delaying meaningful action for decades. (meaningful action is to research alternatives, which did succeed, but also to price carbon, which https://www.barrons.com/articles/europe-carbon-tax-emissions-climate-policy-1653e360 doesn’t even start until 2026, 50 years after the discovery of climate change)
These historical examples lead to a conclusion as well, I will see if you realize what this means for AI.
Thanks for the update! I think this is probably something important to take into consideration when evaluating ASI arguments.
That said, I think we’re starting to stray from the original topic of the Dark Arts, as we’re focusing more on ASI specifically rather than the Dark Arts element of it. In the interest of maintaining discussion focus on this post, would you agree to continuing AGI discussion in private messages?
Sure. Feel free to PM.
And I was trying to focus on the dark arts part of the arguments. Note I don’t make any arguments about ASI in the above, just state that fairly weak evidence should be needed to justify not doing anything drastic about it at this time, because the drastic actions have high measurable costs. It’s not provable at present to state that “ASI could find a way to take over the planet with limited resources, because we don’t have an ASI or know the intelligence ROI on a given amount of flops”, but it is provable to state that “an AI pause of 6 months would cost tens of billions, possibly hundreds of billions of dollars and would reduce the relative power of the pausing countries internationally”. It’s also provable to state the damage of a nuclear exchange.
Look how it’s voted down to −10 on agreement : others feel very strongly about this issue.