When I imagine what a fight between humanity and a moderately-superhuman AGI looks like… well, mostly I imagine there is no fight, humanity just gets wiped out overnight. But if humanity turns out to be more resilient than expected, and the AGI doesn’t immediately foom for some reason, then I start to think about what it’s like to fight an opponent smarter than us.
Story 1: OODA Loops
I once played laser tag against someone who was really good at laser tag. They’d shoot from one spot, and by the time you turn to look, they’d already be moving somewhere else. Then they’d loop around, and hit you from the back while you were looking where they were previously. Then you’d try to retreat, but they’d already be headed that way and hit you again. So you’d try to loop around and get the drop on them, but they’d see where you’re going, and they’d once again pop out from someplace you didn’t expect.
This sort of thing is straight out of standard US military doctrine: it’s all about “getting inside the opponent’s OODA loop”. You want to observe what the opponent is doing and react to it faster than they can orient to your own actions. When someone is “inside your OODA loop” like this, it feels confusing and disorienting, you’re constantly being hit from surprising directions and you have no idea where the opponent is or what they’re doing.
This sort of fight is very cognition-heavy. Not just in the generic sense of “being smart”, but also in the sense of noticing things, tracking what’s going on in your environment, predicting your opponent, making good decisions quickly, acting efficiently, etc. It’s the sort of thing you’d expect even relatively weak AGI to be very good at.
Story 2: Technology Indistinguishable From Magic
Imagine a medieval lord in a war against someone with slightly more advanced technological knowledge. We’re not talking modern weaponry here, just gunpowder.
To the lord, it doesn’t look like the technologist is doing anything especially dangerous; mostly the technologist looks like an alchemist or a witch doctor. The technologist digs a hole, stretches a cloth over, dumps a pile of shit on top, then runs water through the shit-pile for a while. Eventually they remove the cloth and shit, throw some coal and brimstone in the hole, and mix it all together.
From the lord’s perspective, this definitely looks weird and mysterious, and they may be somewhat worried about weird and mysterious things in general. But it’s not obviously any more dangerous than, say, a shaman running a spiritual ceremony.
It’s not until after the GIANT GODDAMN EXPLOSION that the shit-pile starts to look unusually dangerous.
Again, this is the sort of thing I’d expect AGI to be good at. Advancing technology is, after all, one of the most central use-cases of AGI.
Story 3: AGI
When I imagine what a fight between humanity and an AGI looks like, it’s a combination of the previous two. The threat isn’t obvious and salient and scary, like a swarm of killer drones. The AGI’s actions mostly seem weird and random, and then bad things happen, and by the time we’ve figured out what’s going on with one bad thing, a different bad thing is already happening.
Like, one day the AGI is throwing cupcakes at a puppy in a very precisely temperature-controlled room. A few days later, a civil war breaks out in Brazil. Then 2 million people die of an unusually nasty flu, and also it’s mostly the 2 million people who are best at handling emergencies but that won’t be obvious for a while, because of course first responders are exposed more than most. At some point there’s a Buzzfeed article on how, through a series of surprising accidents, a puppy-cupcake meme triggered the civil war in Brazil, but this is kind of tongue-in-cheek and nobody’s taking it seriously and also not paying attention because THE ANTARCTIC ICE CAP JUST MELTED which SURE IS ALARMING but it’s actually just a distraction and the thing everybody should have paid attention to is the sudden shift in the isotope mix of biological nitrogen in algae blooms but that never made the mainstream news at all and page 1 of every news source is all about the former Antarctic ice cap right up until the corn crop starts to fail and the carrying capacity of humanity’s food supply drops by 70% overnight.
That’s what it’s like fighting an opponent smarter than ourselves. It’s confusing, disorienting. Weird surprising things just keep coming out of nowhere, and we have no idea what’s going on until after the fact.
Why Does This Matter?
What I actually expect, in a hypothetical “fight” between humanity and AGI, is that humanity just loses overnight. But I think having this intuitive picture about what a fight would look like is useful to inform other intuitions—for instance, about deception, or military applications of weak AI, or about the strategic importance of intelligence in general.
What Would A Fight Between Humanity And AGI Look Like?
When I imagine what a fight between humanity and a moderately-superhuman AGI looks like… well, mostly I imagine there is no fight, humanity just gets wiped out overnight. But if humanity turns out to be more resilient than expected, and the AGI doesn’t immediately foom for some reason, then I start to think about what it’s like to fight an opponent smarter than us.
Story 1: OODA Loops
I once played laser tag against someone who was really good at laser tag. They’d shoot from one spot, and by the time you turn to look, they’d already be moving somewhere else. Then they’d loop around, and hit you from the back while you were looking where they were previously. Then you’d try to retreat, but they’d already be headed that way and hit you again. So you’d try to loop around and get the drop on them, but they’d see where you’re going, and they’d once again pop out from someplace you didn’t expect.
This sort of thing is straight out of standard US military doctrine: it’s all about “getting inside the opponent’s OODA loop”. You want to observe what the opponent is doing and react to it faster than they can orient to your own actions. When someone is “inside your OODA loop” like this, it feels confusing and disorienting, you’re constantly being hit from surprising directions and you have no idea where the opponent is or what they’re doing.
This sort of fight is very cognition-heavy. Not just in the generic sense of “being smart”, but also in the sense of noticing things, tracking what’s going on in your environment, predicting your opponent, making good decisions quickly, acting efficiently, etc. It’s the sort of thing you’d expect even relatively weak AGI to be very good at.
Story 2: Technology Indistinguishable From Magic
Imagine a medieval lord in a war against someone with slightly more advanced technological knowledge. We’re not talking modern weaponry here, just gunpowder.
To the lord, it doesn’t look like the technologist is doing anything especially dangerous; mostly the technologist looks like an alchemist or a witch doctor. The technologist digs a hole, stretches a cloth over, dumps a pile of shit on top, then runs water through the shit-pile for a while. Eventually they remove the cloth and shit, throw some coal and brimstone in the hole, and mix it all together.
From the lord’s perspective, this definitely looks weird and mysterious, and they may be somewhat worried about weird and mysterious things in general. But it’s not obviously any more dangerous than, say, a shaman running a spiritual ceremony.
It’s not until after the GIANT GODDAMN EXPLOSION that the shit-pile starts to look unusually dangerous.
Again, this is the sort of thing I’d expect AGI to be good at. Advancing technology is, after all, one of the most central use-cases of AGI.
Story 3: AGI
When I imagine what a fight between humanity and an AGI looks like, it’s a combination of the previous two. The threat isn’t obvious and salient and scary, like a swarm of killer drones. The AGI’s actions mostly seem weird and random, and then bad things happen, and by the time we’ve figured out what’s going on with one bad thing, a different bad thing is already happening.
Like, one day the AGI is throwing cupcakes at a puppy in a very precisely temperature-controlled room. A few days later, a civil war breaks out in Brazil. Then 2 million people die of an unusually nasty flu, and also it’s mostly the 2 million people who are best at handling emergencies but that won’t be obvious for a while, because of course first responders are exposed more than most. At some point there’s a Buzzfeed article on how, through a series of surprising accidents, a puppy-cupcake meme triggered the civil war in Brazil, but this is kind of tongue-in-cheek and nobody’s taking it seriously and also not paying attention because THE ANTARCTIC ICE CAP JUST MELTED which SURE IS ALARMING but it’s actually just a distraction and the thing everybody should have paid attention to is the sudden shift in the isotope mix of biological nitrogen in algae blooms but that never made the mainstream news at all and page 1 of every news source is all about the former Antarctic ice cap right up until the corn crop starts to fail and the carrying capacity of humanity’s food supply drops by 70% overnight.
That’s what it’s like fighting an opponent smarter than ourselves. It’s confusing, disorienting. Weird surprising things just keep coming out of nowhere, and we have no idea what’s going on until after the fact.
Why Does This Matter?
What I actually expect, in a hypothetical “fight” between humanity and AGI, is that humanity just loses overnight. But I think having this intuitive picture about what a fight would look like is useful to inform other intuitions—for instance, about deception, or military applications of weak AI, or about the strategic importance of intelligence in general.