Have you recently changed your estimate about the nearest x-risk? I ended up to believe that now nuclear war > runaway biotech > UFAI, where > means nearer / more probable than. Possibly, a global nuclear war would not be existential to the point of obliterating humanity, but setting it back a couple of millennia seems to be negative enough to be classified as existential.
Given rising biotech/chem/drone capability I don’t think that nuclear war is the biggest war related x-risk.
The situation in North Korea might be very bad for South Korea but I don’t see it as threatening the global stability.
Trump seems to have been very open for listening to the Chinese president. The fact that he changed his opinion about North Korea within 10 minutes of talking to the Chinese president suggests to me that the Chinese president is quite capable of communicating well with Trump.
Trump doesn’t care about any sacred values, so I imagine he’s willing to make deals when China wants to have some islands.
I don’t think the recent events in Syria suggest that war with Russia is likely.
The situation in North Korea might be very bad for South Korea but I don’t see it as threatening the global stability.
You mean that you think nuclear escalation unlikely or that, even in case of a nuclear conflict, that it would stay local? On the whole situation I’m using the outside view, since I’ve no specific knowledge about this side of the globe. But I would gladly read what you have to say.
North Korea can destroy South Korea with nukes or other weapons. It doesn’t have the capability to do more global damage.
China itself has no interest in WWIII. North Korea isn’t important to them. China has more trade with South Korea than with North Korea.
China doesn’t want Western troops on the territory of North Korea but I don’t think a Trump administration would want to occupy North Korea anyway. If there are Chinese troops in North Korea I don’t think there would be any objection from the US.
China might want to have something in exchange but the kind of things that China wants like sovereignty over islands, are negotiable for Trump.
As much as the media likes to call North Korea crazy, they’re not suicidal. But given their self-inflicted weak position, they’re more willing to take extreme risks to survive. A miscalculation could escalate to war. Trump isn’t exactly stable either. We cannot afford to let them develop second strike capability, or we risk nuclear blackmail. If crushing sanctions don’t work, or don’t work fast enough, our only option is a preemptive strike. But their defenses are so entrenched that Trump might be tempted to make it a nuclear preemptive strike. At that point, can we trust Russia and China to stay out of it?
We can’t just contain North Korea instead. They’ve proliferated every weapon system they’ve ever developed for cash. What’s to stop them from selling a few to Islamic terrorists, who as non-state actors with no territory are immune to the MAD doctrine? This is an apocalyptic culture willing to fly airplanes into buildings. Don’t think they wouldn’t use atomic suicide bombers if they had them. Civilization will have absolutely no defense against this until Musk’s self-sustaining Mars colony. That such a regime has nukes at all is already intolerable.
And the worst part is, they don’t even need missiles for second strike capability. What’s to stop them from loading one into a cargo container, sailing the cargo ship into New York under a foreign flag of convenience and setting it off before they even unload it? Or load one into a crate labeled “farm equipment” on the next cargo plane bound for D.C.? They could plant one in every major city, and then set them off by remote control in the event of a U.S. invasion.
Q. How hard is it to smuggle a nuke into the country? A. Easy, you hide it in the next bale of marijuana.
Why do we think they haven’t already smuggled some in? Well, it would be an extreme risk. They might get caught. And then they’d really be in trouble. Any other nuclear power wouldn’t risk it. But North Korea has lasted this long by being willing to take extreme risks. Why else? It might not be a credible deterrent until after they set one off on foreign soil to prove they can. But that also risks a war. But then so did shelling a South Korean fishing village. A bioweapon would be even easier to smuggle in. But it’s also not a deterrent until they prove it works.
As much as the media likes to call North Korea crazy, they’re not suicidal. But given their self-inflicted weak position, they’re more willing to take extreme risks to survive.
To make a good analysis of what North Korea is likely to do it’s helpful to think about it being made of a leadership of humans instead of being an abstract country that makes decisions.
North Korea is very much driven by different North Korean actors having to signal to each other who stronlgy patriotic they are.
At that point, can we trust Russia and China to stay out of it?
Why would we want China to stay out of North Korea in the case of a war? If they put their troops into North Korea to take control of it, that would be a nice outcome.
Neither Russia nor China would be interested in having WWIII.
If crushing sanctions don’t work, or don’t work fast enough, our only option is a preemptive strike.
What exactly do you try to argue here? That you don’t know how an effective North Korea policy that uses other tools than military strikes or sanctions looks like?
Q. How hard is it to smuggle a nuke into the country?
A. Easy, you hide it in the next bale of marijuana.
Ports do scan for radioactivity. Uranian is also very heavy which provides further ways to detect that a transport of a nuclear weapon.
I’m also not sure about whether North Korea has the capability of making a decision to deploy a nuclear weapon without US and Chinese intelligence agencies getting to know about it.
It’s also not the kind of action that’s good for signaling purposes. If agency A in North Korea smuggled the nuke and they want to get adminiration from agency B, they have to share information about it.
There’s also the risk that information about the location of the nuke leakes and the person responsible for the placement of the nuke get’s into problems as a result.
Apart from that the fact that North Korea has nuclear weapons is no recent event that warrants any change.
To make a good analysis of what North Korea is likely to do it’s helpful to think about it being made of a leadership of humans instead of being an abstract country that makes decisions.
Indeed, when I say “North Korea” I mean the Kim Family Regime. That’s the self-inflicted weak position I mentioned. They have to terrorize and indoctrinate the population to stay in power. Any meaningful reforms are poison to the regime, since they prove its illegitimacy. They’ve painted themselves in a corner. They have to be evil. That’s why we can’t just have a peace treaty and end the war.
Why would we want China to stay out of North Korea in the case of a war? If they put their troops into North Korea to take control of it, that would be a nice outcome. Neither Russia nor China would be interested in having WWIII.
The ideal outcome is that South Korea takes over, but yes, if China takes over that’s still better than the status quo. I meant “stay out of the nuclear confrontation.”. If the U.S. unilaterally uses nukes first, what repercussions does that have for the rest of the world? Would that weaken or strengthen the NPT? The MAD doctrine? Would China use the opportunity to take Taiwan? Would China retaliate (even accidentally) against a U.S. ally (like South Korea) for using nukes so close to its territory? Would that escalate?
What exactly do you try to argue here? That you don’t know how an effective North Korea policy that uses other tools than military strikes or sanctions looks like?
And it looks like no-one else does either. We don’t have any good options. Containment and “strategic patience” isn’t a good option either because the problem is steadily getting worse. North Korea continues to build more weapons. How bad does it have to get? What’s the tipping point? That is, at what point will we wish we’d ended the Korean War even at the cost of half of Seoul? The intervention should come before that. But another problem is, we can’t get good intelligence. It’s an isolated totalitarian state with extensive underground facilities. We can’t rely on spies on the ground. We just occasionally learn things from low-ranking defectors. We have spy satellites, but can’t see underground from orbit. If our intelligence is that unreliable, then we must intervene at a point long enough before the tipping point to account for our margin of error. What does the end of this story look like?
Ports do scan for radioactivity.
That’s a very important point I had not considered, and a possible defense against smuggled nukes. I’m not confident in the technical details though. The alpha and beta radiation is too easily shielded, but at what distance can we distinguish the gamma from background? If it’s only a few meters, that’s not really helpful. If it’s several kilometers, then we could perhaps interdict or sink a cargo ship before it threatens the coast.
Indeed, when I say “North Korea” I mean the Kim Family Regime.
The important thing about the family is that they kill each other. Jang Song-thaek who was rumored to be the defacto leader of North Korea in 2009 died in 2013. Finding yourself at the wrong side of a struggle inside the regime means death.
Kim Kyong-hui is a member of the Kim Family clan but I doubt her first priority is to worry about the US.
And it looks like no-one else does either. We don’t have any good options.
There’s the option to trade and thus push for information flow between North Korea and the outside world.
But another problem is, we can’t get good intelligence.
Without access to classified intelligence this is really hard to tell. There are drones flying around and there’s SIGINT intelligence. Information that’s communicated electronically inside of North Korea is subject to interception.
If the U.S. unilaterally uses nukes first, what repercussions does that have for the rest of the world?
That’s why the U.S. is very unlikely to nuke first. Even under Trump that’s unlikely to happen.
Yes, I have. Nuclear war lost its top spot to antimicrobial resistance.
Given recent events on the Korean peninsula it may seem strange to downgrade the risk of nuclear war. Explanation:
While the probability of conflict is at a local high, the potential severity of the conflict is lower than I’d thought. This is because I’ve downgraded my estimate of how many nukes DPRK is likely to successfully deploy. (Any shooting war would still be a terrible event, especially for Seoul, which is only about 60 km from the border—firmly within conventional artillery range.)
An actual conflict with DPRK may deter other aspiring nuclear states, while a perpetual lack of conflict may have the opposite effect. As the number of nuclear states rises, both the probability and severity of a nuclear war rise, so the expected damage rises as the square. The chance of accident or terrorist use of nukes rises too.
Rising tensions with DPRK, even without a war, can result in a larger global push for stronger anti-proliferation measures.
Perhaps paradoxically, because (a) DPRK’s capabilities are improving over time and (b) a conflict now ends the potential for a future conflict, a higher chance of a sooner (and smaller) conflict means a lower chance of a later (and larger) conflict.
You say:
I ended up to believe that now nuclear war > runaway biotech > UFAI
What was your ranking before, and on what information did you update?
Why does antimicrobial resistance rank so high in your estimation? It seems like a catastrophic risk at worst, not an existential one. New antibiotics are developed rather infrequently because they’re currently not that profitable. Incentives would change if the resistance problem got worse. I don’t think we’ve anywhere near exhausted antibiotic candidates found in nature, and even if we had, there are alternatives like phage therapy and monoclonal antibodies that we could potentially use instead.
It’s true that the probability of an existential-level AMR event is very low. But the probability of any existential-level threat event is very low; it’s the extreme severity, not the high probability, that makes such risks worth considering.
Concretely? I’m not sure. One way is for a pathogen to jump from animals (or a lab) to humans, and then manage to infect and kill billions of people.
Humanity existed for the great majority of its history without antibiotics.
True. But it’s much easier for a disease to spread long distances and among populations than in the past.
Note: I just realized there might be some terminological confusion, so I checked Bostrom’s terminology. My “billions of deaths” scenario would not be “existential,” in Bostrom’s sense, because it isn’t terminal: Many people would survive, and civilization would eventually recover. But if a pandemic reduced today’s civilization to the state in which humanity existed for the majority of its history, that would be much worse than most nuclear scenarios, right?
if a pandemic reduced today’s civilization to the state in which humanity existed for the majority of its history
Why would it? A pandemic wouldn’t destroy knowledge or technology.
Consider Black Death—it reduced the population of Europe by something like a third, I think. Was it a big deal? Sure it was. Did it send Europe back to the time when it was populated by some hunter-gatherer bands? Nope, not even close.
We have a lot of systems that depend on one another; perhaps a severe enough pandemic would cause a sort of cascade of collapse. I’d think it would have to be really bad, though, certainly worse than killing 1⁄3 of the population.
I am sure there would be some collapse, the question is how long will it take to rebuild. I would imagine that the survivors would just abandon large swathes of land and concentrate themselves. Having low population density overall is not a problem—look at e.g. Australia or Canada.
But we are now really in movie-plots land. Are you prepared for the zombie apocalypse?
I’m not sure how to rank these if the ordering relation is “nearer / more probable than”. Nuclear war seems like the most imminent threat, and UFAI the most inevitable.
We all know the arguments regarding UFAI. The only things that could stop the development of general AI at this point are themselves existential threats. Hence the inevitability. I think we already agree that FAI is a more difficult problem than superintelligence. But we might underestimate how much more difficult. The naiive approach is to solve ethics in advance. Right. That’s not going to happen in time. Our best known alternative is to somehow bootstrap machine learning into solving ethics for us without it killing us in the mean time. This still seems really damn difficult.
We’ve already had several close calls with nukes during the cold war. The USA has been able to reduce her stockpile since the collapse of the Soviet Union, but nukes have since proliferated to other countries. (And Russia, of course, sill has leftover Soviet nukes.) If the NPT system fails due to the influence of rogue states like Iran and North Korea, there could be a domino effect as the majority of nations that can afford it race to develop arms to counter their neighbors. This has arguably already happened in the case of Pakistan countering India, which didn’t join the NPT. Now notice that Iran borders Pakistan. How long can we hold the line there?
I should also point out that there are risks worse than even existential, which Bostrom called “hellish”, meaning that is a human extinction event would be a better outcome than a hellish one. A perverse kind of near miss with AI is the most likely to produce such an outcome. The AI would have to be friendly enough not to kill us all for spare atoms, and yet not friendly enough to produce an outcome we would consider desirable.
There are many other known existential risks, and probably some that are unknown. I’ve pointed out that AMR seems like a low risk, but I also think bioweapons are the next most imminent threat after nukes. Nukes are expensive. We can kind of see them coming and apply sanctions. We’ve developed game theory strategies to make use of the existing weapons unlikely. But bioweapons will be comparatively cheap and stealthy. Even so, I expect any such catastrophe to likely be self limiting. The more deadly an infection, the less it spreads. Zombies are not realistic. There would have to be a long incubation period or an animal reservoir, which would give us time to detect and treat it. One would have to engineer a pathogen very carefully to overcome these many limitations, to get to existential threat level, but most actors motivated to produce bioweapons would consider the self-limiting nature a benefit, to avoid blowback. These limitations are also what makes me think that AMR events are less risk than bioweapons.
What was your ranking before, and on what information did you update?
Well, before it was: runaway bioweapon > UFAI > nuclear extinction, but the recent news about the international situation made me update. As I said elsewhere, I’m adopting the outside view on all these subjects, so I will gladly stand corrected.
Have you recently changed your estimate about the nearest x-risk?
I ended up to believe that now nuclear war > runaway biotech > UFAI, where > means nearer / more probable than.
Possibly, a global nuclear war would not be existential to the point of obliterating humanity, but setting it back a couple of millennia seems to be negative enough to be classified as existential.
Given rising biotech/chem/drone capability I don’t think that nuclear war is the biggest war related x-risk.
The situation in North Korea might be very bad for South Korea but I don’t see it as threatening the global stability.
Trump seems to have been very open for listening to the Chinese president. The fact that he changed his opinion about North Korea within 10 minutes of talking to the Chinese president suggests to me that the Chinese president is quite capable of communicating well with Trump. Trump doesn’t care about any sacred values, so I imagine he’s willing to make deals when China wants to have some islands.
I don’t think the recent events in Syria suggest that war with Russia is likely.
You mean that you think nuclear escalation unlikely or that, even in case of a nuclear conflict, that it would stay local? On the whole situation I’m using the outside view, since I’ve no specific knowledge about this side of the globe. But I would gladly read what you have to say.
North Korea can destroy South Korea with nukes or other weapons. It doesn’t have the capability to do more global damage.
China itself has no interest in WWIII. North Korea isn’t important to them. China has more trade with South Korea than with North Korea. China doesn’t want Western troops on the territory of North Korea but I don’t think a Trump administration would want to occupy North Korea anyway. If there are Chinese troops in North Korea I don’t think there would be any objection from the US.
China might want to have something in exchange but the kind of things that China wants like sovereignty over islands, are negotiable for Trump.
As much as the media likes to call North Korea crazy, they’re not suicidal. But given their self-inflicted weak position, they’re more willing to take extreme risks to survive. A miscalculation could escalate to war. Trump isn’t exactly stable either. We cannot afford to let them develop second strike capability, or we risk nuclear blackmail. If crushing sanctions don’t work, or don’t work fast enough, our only option is a preemptive strike. But their defenses are so entrenched that Trump might be tempted to make it a nuclear preemptive strike. At that point, can we trust Russia and China to stay out of it?
We can’t just contain North Korea instead. They’ve proliferated every weapon system they’ve ever developed for cash. What’s to stop them from selling a few to Islamic terrorists, who as non-state actors with no territory are immune to the MAD doctrine? This is an apocalyptic culture willing to fly airplanes into buildings. Don’t think they wouldn’t use atomic suicide bombers if they had them. Civilization will have absolutely no defense against this until Musk’s self-sustaining Mars colony. That such a regime has nukes at all is already intolerable.
And the worst part is, they don’t even need missiles for second strike capability. What’s to stop them from loading one into a cargo container, sailing the cargo ship into New York under a foreign flag of convenience and setting it off before they even unload it? Or load one into a crate labeled “farm equipment” on the next cargo plane bound for D.C.? They could plant one in every major city, and then set them off by remote control in the event of a U.S. invasion.
Q. How hard is it to smuggle a nuke into the country?
A. Easy, you hide it in the next bale of marijuana.
Why do we think they haven’t already smuggled some in? Well, it would be an extreme risk. They might get caught. And then they’d really be in trouble. Any other nuclear power wouldn’t risk it. But North Korea has lasted this long by being willing to take extreme risks. Why else? It might not be a credible deterrent until after they set one off on foreign soil to prove they can. But that also risks a war. But then so did shelling a South Korean fishing village. A bioweapon would be even easier to smuggle in. But it’s also not a deterrent until they prove it works.
How does this not threaten global stability?
To make a good analysis of what North Korea is likely to do it’s helpful to think about it being made of a leadership of humans instead of being an abstract country that makes decisions.
North Korea is very much driven by different North Korean actors having to signal to each other who stronlgy patriotic they are.
Why would we want China to stay out of North Korea in the case of a war? If they put their troops into North Korea to take control of it, that would be a nice outcome. Neither Russia nor China would be interested in having WWIII.
What exactly do you try to argue here? That you don’t know how an effective North Korea policy that uses other tools than military strikes or sanctions looks like?
Ports do scan for radioactivity. Uranian is also very heavy which provides further ways to detect that a transport of a nuclear weapon.
I’m also not sure about whether North Korea has the capability of making a decision to deploy a nuclear weapon without US and Chinese intelligence agencies getting to know about it.
It’s also not the kind of action that’s good for signaling purposes. If agency A in North Korea smuggled the nuke and they want to get adminiration from agency B, they have to share information about it. There’s also the risk that information about the location of the nuke leakes and the person responsible for the placement of the nuke get’s into problems as a result.
Apart from that the fact that North Korea has nuclear weapons is no recent event that warrants any change.
Indeed, when I say “North Korea” I mean the Kim Family Regime. That’s the self-inflicted weak position I mentioned. They have to terrorize and indoctrinate the population to stay in power. Any meaningful reforms are poison to the regime, since they prove its illegitimacy. They’ve painted themselves in a corner. They have to be evil. That’s why we can’t just have a peace treaty and end the war.
The ideal outcome is that South Korea takes over, but yes, if China takes over that’s still better than the status quo. I meant “stay out of the nuclear confrontation.”. If the U.S. unilaterally uses nukes first, what repercussions does that have for the rest of the world? Would that weaken or strengthen the NPT? The MAD doctrine? Would China use the opportunity to take Taiwan? Would China retaliate (even accidentally) against a U.S. ally (like South Korea) for using nukes so close to its territory? Would that escalate?
And it looks like no-one else does either. We don’t have any good options. Containment and “strategic patience” isn’t a good option either because the problem is steadily getting worse. North Korea continues to build more weapons. How bad does it have to get? What’s the tipping point? That is, at what point will we wish we’d ended the Korean War even at the cost of half of Seoul? The intervention should come before that. But another problem is, we can’t get good intelligence. It’s an isolated totalitarian state with extensive underground facilities. We can’t rely on spies on the ground. We just occasionally learn things from low-ranking defectors. We have spy satellites, but can’t see underground from orbit. If our intelligence is that unreliable, then we must intervene at a point long enough before the tipping point to account for our margin of error. What does the end of this story look like?
That’s a very important point I had not considered, and a possible defense against smuggled nukes. I’m not confident in the technical details though. The alpha and beta radiation is too easily shielded, but at what distance can we distinguish the gamma from background? If it’s only a few meters, that’s not really helpful. If it’s several kilometers, then we could perhaps interdict or sink a cargo ship before it threatens the coast.
But this doesn’t apply to a smuggled bioweapon.
The important thing about the family is that they kill each other. Jang Song-thaek who was rumored to be the defacto leader of North Korea in 2009 died in 2013. Finding yourself at the wrong side of a struggle inside the regime means death.
Kim Kyong-hui is a member of the Kim Family clan but I doubt her first priority is to worry about the US.
There’s the option to trade and thus push for information flow between North Korea and the outside world.
Without access to classified intelligence this is really hard to tell. There are drones flying around and there’s SIGINT intelligence. Information that’s communicated electronically inside of North Korea is subject to interception.
That’s why the U.S. is very unlikely to nuke first. Even under Trump that’s unlikely to happen.
Yes, I have. Nuclear war lost its top spot to antimicrobial resistance.
Given recent events on the Korean peninsula it may seem strange to downgrade the risk of nuclear war. Explanation:
While the probability of conflict is at a local high, the potential severity of the conflict is lower than I’d thought. This is because I’ve downgraded my estimate of how many nukes DPRK is likely to successfully deploy. (Any shooting war would still be a terrible event, especially for Seoul, which is only about 60 km from the border—firmly within conventional artillery range.)
An actual conflict with DPRK may deter other aspiring nuclear states, while a perpetual lack of conflict may have the opposite effect. As the number of nuclear states rises, both the probability and severity of a nuclear war rise, so the expected damage rises as the square. The chance of accident or terrorist use of nukes rises too.
Rising tensions with DPRK, even without a war, can result in a larger global push for stronger anti-proliferation measures.
Perhaps paradoxically, because (a) DPRK’s capabilities are improving over time and (b) a conflict now ends the potential for a future conflict, a higher chance of a sooner (and smaller) conflict means a lower chance of a later (and larger) conflict.
You say:
What was your ranking before, and on what information did you update?
Why does antimicrobial resistance rank so high in your estimation? It seems like a catastrophic risk at worst, not an existential one. New antibiotics are developed rather infrequently because they’re currently not that profitable. Incentives would change if the resistance problem got worse. I don’t think we’ve anywhere near exhausted antibiotic candidates found in nature, and even if we had, there are alternatives like phage therapy and monoclonal antibodies that we could potentially use instead.
It’s true that the probability of an existential-level AMR event is very low. But the probability of any existential-level threat event is very low; it’s the extreme severity, not the high probability, that makes such risks worth considering.
What, in your view, gets the top spot?
What would that look like? Humanity existed for the great majority of its history without antibiotics.
Concretely? I’m not sure. One way is for a pathogen to jump from animals (or a lab) to humans, and then manage to infect and kill billions of people.
True. But it’s much easier for a disease to spread long distances and among populations than in the past.
Note: I just realized there might be some terminological confusion, so I checked Bostrom’s terminology. My “billions of deaths” scenario would not be “existential,” in Bostrom’s sense, because it isn’t terminal: Many people would survive, and civilization would eventually recover. But if a pandemic reduced today’s civilization to the state in which humanity existed for the majority of its history, that would be much worse than most nuclear scenarios, right?
Why would it? A pandemic wouldn’t destroy knowledge or technology.
Consider Black Death—it reduced the population of Europe by something like a third, I think. Was it a big deal? Sure it was. Did it send Europe back to the time when it was populated by some hunter-gatherer bands? Nope, not even close.
We have a lot of systems that depend on one another; perhaps a severe enough pandemic would cause a sort of cascade of collapse. I’d think it would have to be really bad, though, certainly worse than killing 1⁄3 of the population.
I am sure there would be some collapse, the question is how long will it take to rebuild. I would imagine that the survivors would just abandon large swathes of land and concentrate themselves. Having low population density overall is not a problem—look at e.g. Australia or Canada.
But we are now really in movie-plots land. Are you prepared for the zombie apocalypse?
I’m not sure how to rank these if the ordering relation is “nearer / more probable than”. Nuclear war seems like the most imminent threat, and UFAI the most inevitable.
We all know the arguments regarding UFAI. The only things that could stop the development of general AI at this point are themselves existential threats. Hence the inevitability. I think we already agree that FAI is a more difficult problem than superintelligence. But we might underestimate how much more difficult. The naiive approach is to solve ethics in advance. Right. That’s not going to happen in time. Our best known alternative is to somehow bootstrap machine learning into solving ethics for us without it killing us in the mean time. This still seems really damn difficult.
We’ve already had several close calls with nukes during the cold war. The USA has been able to reduce her stockpile since the collapse of the Soviet Union, but nukes have since proliferated to other countries. (And Russia, of course, sill has leftover Soviet nukes.) If the NPT system fails due to the influence of rogue states like Iran and North Korea, there could be a domino effect as the majority of nations that can afford it race to develop arms to counter their neighbors. This has arguably already happened in the case of Pakistan countering India, which didn’t join the NPT. Now notice that Iran borders Pakistan. How long can we hold the line there?
I should also point out that there are risks worse than even existential, which Bostrom called “hellish”, meaning that is a human extinction event would be a better outcome than a hellish one. A perverse kind of near miss with AI is the most likely to produce such an outcome. The AI would have to be friendly enough not to kill us all for spare atoms, and yet not friendly enough to produce an outcome we would consider desirable.
There are many other known existential risks, and probably some that are unknown. I’ve pointed out that AMR seems like a low risk, but I also think bioweapons are the next most imminent threat after nukes. Nukes are expensive. We can kind of see them coming and apply sanctions. We’ve developed game theory strategies to make use of the existing weapons unlikely. But bioweapons will be comparatively cheap and stealthy. Even so, I expect any such catastrophe to likely be self limiting. The more deadly an infection, the less it spreads. Zombies are not realistic. There would have to be a long incubation period or an animal reservoir, which would give us time to detect and treat it. One would have to engineer a pathogen very carefully to overcome these many limitations, to get to existential threat level, but most actors motivated to produce bioweapons would consider the self-limiting nature a benefit, to avoid blowback. These limitations are also what makes me think that AMR events are less risk than bioweapons.
Well, before it was: runaway bioweapon > UFAI > nuclear extinction, but the recent news about the international situation made me update. As I said elsewhere, I’m adopting the outside view on all these subjects, so I will gladly stand corrected.