Lessons on AI Takeover from the conquistadors
(Talk given at an event on Sunday 28th of June. Daniel Kokotajlo is responsible for the talk, Jacob Lagerros and David Lambert edited the transcript.
If you’re a curated author and interested in giving a 5-min talk, which will then be transcribed and edited, sign up here.)
Daniel Kokotajlo: I’m basically going to recap my conquistadors posts. I’m interested to hear what people think about it. I know a lot more about the situation now because I’ve been reading some history books. But my overall opinion hasn’t changed.
I wrote this post because in talking to various people and reading some things, there was what seemed to me to be a simplistic model of how military conflicts work, or how a military takeover would work. Here’s the simple model.
If we’re imagining a scenario where an AI appears, it will likely be in some lab somewhere. So it will start off with very few resources, very small amounts of money, for example. So you would have to have some sort of god-like intelligence or a technological advantage in order to have more strength than the rest of the world combined.
I think the conquistadors are counterexamples to this. So for a brief overview, I picked out three conquistadors. I had already known about Cortes and Pizarro, which were probably the most extreme examples. But I predicted that there would be a general trend here, so I did some Googling and after 20 minutes I found Alfonso, who is another example of this trend.
Alfonso was a Portuguese explorer general. Here is a map of his exploration.
Ben Pace: Can I ask what period of history this was?
Daniel Kokotajlo: This is all around the early 1500s. Alfonso was 1506 to 1513, and then Cortes was 1519 and Pizarro was 1530.
If you look at this map, basically, Columbus went across the Atlantic in 1492, and the Portuguese were, in the same time, exploring down the coast of Africa. They were all trying to get to China and India’s lucrative trade routes. Basically, within 30 to 50 years, the Spanish had turned this entire region into a Spanish lake, controlling the main ports and only allowing Spanish ships.
The Portuguese did the same thing. Alfonso was responsible for the blue region. It only took them six years or so, and involved defeating the Ottoman Empire in several naval battles, conquering various port cities around the edge, defeating some Indian empires, and so forth.
Here is the map of Portuguese territory in India which blew my mind.
I didn’t realize that they had conquered so much so quickly. I haven’t looked up this exactly, but it took many more years to conquer all of this territory.
When I usually think of colonialism in India, I think of the British. But according to this map, the Portuguese made quite a lot of conquests themselves before then. Anyway, the advantages were not god-like, in my opinion. I go into more detail in the post, but it was not like they had nanobots or any other super weapons in which there were no defenses against.
They also didn’t have machine guns, or even normal guns. They had extremely ancient arquebuses that took two minutes to reload after you fired them. Most of the time, they just used swords for their fighting.. I could go on.
I do think that they won because of their technology, but it definitely was not a god-like advantage. So I think this disrupts the simple model that I mentioned earlier. And it’s not like they just had more strength than the regions that they conquered either.
Or if we do say that they had more strength than the regions that they conquered, it’s not because they had some sort of god-like advantage. Here’s my model which I think better describes how these, and possibly many other, conquests worked.
Military takeover is a diplomatic process in which you convince people to obey you. You do that by offering a better Schelling point than rival claimants to the throne, and by defeating your rivals in battle. You don’t need to defeat them all at once; you can start small and scale up.
Each victory gets you more allies which you can deploy in the next battle. Rivals can become allies after, or even without, a fight. This is how these conquests worked. It wasn’t like Pizarro showed up at the Incan Empire’s borders and was met by a massive army, which he then defeated. Same thing for Cortes. There were a series of episodes in which the conquistadors incrementally gained more power.
If there’s an AI takeover, maybe the AI will have god-like powers with nanobots and other powerful technologies. But it is also entirely possible that there could be an AI takeover without these things. Thank you.
Questions
Ben Pace: Thank you very much, Daniel. That was fascinating and you crammed quite a lot into it. I have some questions but so does everyone else. Daniel Filan, would you like to ask the first question?
Daniel Filan: Yes. This is a combination of comment and question.
Thinking about god-like powers, I recently read a piece about Cortes taking over the Aztec Empire, and apparently a really big advantage was armor impervious to the kinds of weapons that Aztecs had. If other people can’t hurt you, then it is, in a sense, a god-like power. What I am wondering is to what extent they had the god-like power of ratcheting, where the conquistadors couldn’t go backwards or get hurt while still being able to hurt the Aztecs.
Daniel Kokotajlo: I think that fits, to some extent, with the story of Cortes, which I’ve now read in great detail. I think that the post that you mentioned, exaggerates the extent to which the armor was superior.
For one thing, a lot of the Spanish gave up their metal armor and took up a Aztec cloth armor instead because it was more comfortable and almost as good, which suggests that their metal armor wasn’t superior. Also, plenty of conquistadors were killed in battle. I think that a lot of their success had to do with their tactics rather than just the armor. That said, armor did likely play a large part in their victory. It is wrong, however, to conclude that they were invincible because of it when in fact they came close to being wiped out on several occasions.
Ben Pace: John, would you like to ask the next question?
johnswentworth: Let’s say, hypothetically, that I’m planning to take over the world through technological means, what sort of technology should I be focusing on in order to get the most bang for my buck?
Daniel Kokotajlo: I think that realistically, an AI takeover would mostly not be military.
johnswentworth: I didn’t ask about AI takeover.
Daniel Kokotajlo: Okay, so I have a list of technologies that I think would be the modern equivalents of the armor.
Maybe I’ll draft a blog post sometime about my own speculations on technologies that aren’t that far ahead into the future, but could be pretty powerful. That’s what I really should have said in response to Daniel’s question. That yes, the armor was really consequential in some sense.
It enabled the conquistadors to go really far in battle and in politics. But it wasn’t fundamentally different from what the Aztecs had. It was similar but better.
And similarly, I think that an AI designing better guns, armor, and coordination technologies for soldiers could end up being extremely advantageous militarily, even if it’s not something fundamentally new, like nanobots. Even if it is something simple like aim assist for soldiers’ guns or steering assist for vehicles, these could provide a massive military advantage.
Ben Pace: Go on. Throw us a bone. Give us one example of a technology, especially a slightly more surprising one. We’ve heard of guns.
Daniel Kokotajlo: Sure. Aim assists for infantry rifles would be really beneficial . The US military is currently looking into this. But a simple version could just be a camera hooked up to a gun and trigger so that when you activate it, it fires the bullet when it calculates that the bullet will actually hit the target.
You wouldn’t have to aim carefully, you could just wave the gun in front of the target and it shoots at exactly the right moment.. A more sophisticated version of this, which I have thought of, but not sure if anyone else has f, is having some sort of cold gas thruster on the tip of the gun.
Or maybe a gyroscope allowing the gun to move itself although it sounds more complicated. This doesn’t sound like much, but I think it would probably be a pretty big deal on the battlefield. For one thing, you wouldn’t have to poke your head out of cover to shoot at the enemy.
You could just poke the gun out and it would do the shooting I also think being substantially more accurate would make cover fire a lot more effective. And that could allow you to be much more mobile on the battlefield. Anyhow, this is just one example.
Ben Pace: It is at this point, I realize I’ve really not actually tried to figure out how I would use modern tech to improve military practices. And I am a bit scared now. All right, the next questioner is orthonormal.
orthonormal: Are there any known examples of attempted conquistadors that failed?
Daniel Kokotajlo: That’s a really good question. There’s probably a half dozen to a dozen parties that wandered around the Americas and didn’t really find much and then died of starvation or something.
In fact, that very nearly happens to Pizarro and Cortes. There weren’t any other major empires like the Incas or the Aztecs in the Americas. As for the rest of the world, I don’t know the full answer to that question. There were likely attempts to conquer India that failed before eventually succeeding.
- Against GDP as a metric for timelines and takeoff speeds by (29 Dec 2020 17:42 UTC; 140 points)
- Clarifying “What failure looks like” by (20 Sep 2020 20:40 UTC; 97 points)
- What a 20-year-lead in military tech might look like by (29 Jul 2020 20:10 UTC; 74 points)
- Modeling Failure Modes of High-Level Machine Intelligence by (6 Dec 2021 13:54 UTC; 54 points)
- Against GDP as a metric for timelines and takeoff speeds by (EA Forum; 29 Dec 2020 17:50 UTC; 47 points)
- Forecasting Thread: Existential Risk by (22 Sep 2020 3:44 UTC; 43 points)
- Carl Shulman On Dwarkesh Podcast June 2023 by (11 Feb 2024 21:02 UTC; 18 points)
- 's comment on Cortés, AI Risk, and the Dynamics of Competing Conquerors by (2 Jan 2024 21:53 UTC; 7 points)
The most obvious example to me is a Stuart Russell slaughterbot—although those aren’t really plausible near-future technologies at this point since they already exist. Using some really good AI to control them directly (deep RL becomes good enough for effective robot control?), along with some high energy density battery storage might make them long-range and much more lethal. Though countermeasures (directed energy weapons) do exist and are being researched.
I think there’s another element to it. In Guns, Germs and Steel Jared Diamond argued that a lot of the reason for the conquistador’s successes came down to their abilities to do things that were outside the frame of reference of the native culture, like invite the Emperor to parley and then just kidnap him. In other words, to the Inca the conquistadors exhibited a weak kind of cognitive uncontainability, furthering the analogy with AI takeover.
On Diamond and writing, see previous discussion here. It is highly unlikely that writing was critical:
Pizarro was illiterate
The Aztecs had writing, yet didn’t beat the Spaniards (or avoid having their leader kidnapped)
Cortes’ conquests were only a decade or so before- a short enough period that writing wasn’t necessary to communicate the lessons. Pizarro was physically present in the Americas at the time.
There’s not actually any clear pathway from “have writing” → “Atahualpa refuses to leave his army to meet with Pizarro”. Writing did not make all European monarchs cautious and immune to ambushes or kidnapping; it is not the case that the Inca didn’t understand the idea of deception.
In the linked thread, Daniel Kokotajlo suggests that the relevant difference was that the Spaniards had experience with more cultures than the Inca, and in particular were far more experienced with first contacts. This sounds plausible to me.
I don’t think I can really claim that much credit for this one.
Lol, fixed, thanks.
One way to think about this is that often people say “god-like” in the monotheistic sense of omnipotence. But classical humans had polytheistic gods (especially demigods) with sharply limited profiles of powers, many of which are met or exceeded by modern humans! That is, the standard you have to hit to become ruler looks more like the difference between regular humans and Perseus or Heracles, not the difference between regular humans and the God of the Old Testament.
[Like, it’s not for nothing that the Aztecs told the Conquistadors that they thought the latter group were gods!]
This existed in 2013, at least according to a press release; similar claims of the military ‘testing’ them are still written in 2020. (The latter article notes that these systems have been available for tanks for a long time, but only recently have been miniaturized to the point that they make sense to include on rifles.)
johnswentsworth earlier brings up Alexander the Great’s attempt.
It is unlikely that the Aztecs actually believed that the Conquistadors were gods. (No primary sources state this; the original source for the gods claim was Francisco Lopez de Gomara, writing based on interviews with conquistadors who returned to Spain decades later; his writing contains many other known inaccuracies.)
Claims that are related to, but distinct from, the Aztecs believing that the Conquistadors were gods:
The Aztecs, and other natives, plausibly believed or said that the Conquistadors were sent by God(s). This is likely because the conquistadors repeatedly and explicitly said that they had been sent by God.
There is substantially stronger evidence that the Aztecs said that they had long-awaited the return of their rightful rulers (implying that the Spanish were the rightful rulers.) Cortes, Bernal Diaz, and the Florentine Codex all agree that this occurred; however, it is impossible to say if it was meant literally.
Thanks for these examples. Are you suggesting this is evidence that this technology isn’t going to be useful?
I read somewhere that India has been conquered 19 times, out of 21 attempts. Or something like that. Alexander was pretty far back in history and doesn’t seem particularly relevant, I think the Turkish incursions into India just prior to the Portuguese might be worth looking at too. I do think it would be worth looking at great conquerors like Ghengis Khan, Alexander, Napoleon, etc. to see how much luck plays a role. There’s some work that’s been done on ranking generals by “Wins Above Replacement” which I’d like to look into.
re: Gods: Sure. I’m looking for more precise ways to put the point, too… maybe something about how many economic doubling times of progress is needed, or what the “on paper vs. reality multiplier” is. (On paper, prior to learning about the conquistadors, being presented with an inventory of all their assets vs. the tech level of the empires they attacked, I would have guessed that they could take on a force ten times their size, probably. But in reality they defeated empires four orders of magnitude bigger than them.)
For the “on paper vs reality” thing, I think a multiplier on population size might not be the right approach. If conquering some group helps you conquer others, then the right model might be more like an epidemiological one, or an evolutionary one that models the spread of genes throughout a population.
In other words, once the advantage you have crosses some threshold, your scope of conquest might be more dependent on the total size of population to be conquered than on the size of the advantage itself.
By analogy, why do the richest people in the world have tens of billions of dollars? Are they a million times smarter than the average person? A million times more hard working? A million times luckier? No. If you multiply all their advantages together do you get to a million? I don’t think so.
What’s actually going on is that they’re enough better that they’re able to stay at the top of some enterprise that, in an interconnected world, is able to grow to world scale. And so they’re able to accumulate a small but non-negligible fraction of the world’s wealth.
I predict that if the world was 1000 times bigger, the richest people would be roughly 1000x richer. (I could be convinced that it might scale super- or sub-linearly, but the main point is that if you wanted to predict the size of their wealth, the first thing you’d want to look at would be the size of the whole world, rather than the size of their absolute advantage to the median person.)
I completely agree with your point about models. So yeah, I guess my “on paper vs reality” idea is a non-starter.
My naïve model is that within ~a generation, maintaining control of a newly conquered region is a net cost, or at least near zero
This seems wrong to me, can you explain?
In my model, to retain control of a conquered area you must commit military forces for some time, and you are not much able to convert the conquered populace into additional military force.
This is not a very data-informed model though.
Ah, I see. Well, there have been many cases in history where the conquered populace was quickly converted into additional military force. For example, in all of the three conquistadors cases I’ve studied. Cortes’ army was 98% native allies, even when he was still fighting to take over the aztecs, and immediately after beating the aztecs he launched new expeditions against neighboring empires, with his native armies. Same goes for Pizarro. Even Afonso had local forces assisting him militarily on many of his fights, and of course he used local forces to help police and garrison the cities and ports he conquered. And then of course the plunder gained from conquest can be used to fund more reinforcements and supplies for your own forces.
Yeah, I’m confused about how quickly one can convert plunder into reinforcements. But I certainly update on the conversion of the conquered. Thanks!
I feel like there’s two points causing the confusion:
(1) The assumption that natives are an undifferentiated mass. There were a variety of mutually hostile indigenous peoples, who themselves sough out allies against each other; and, in particular, who sought to balance the strongest local powers. Seven Myths of the Spanish Conquest, page 48:
(2) This also neglects native power structures, which conquistadors mostly left intact in the years immediately after the conquest; note also that part of the conquered region had previously been conquered by the Aztecs, so there the Spanish were simply substituting one empire for another. The Spanish initially didn’t speak the native languages, worked through local elites, and reused existing systems of tribute and corvee labor. The Spanish eventually exercised more direct control, but this took an extremely long time. (Fun fact: the last native rebellion against European control in Mexico ended in 1933).
(I am less familiar with India.)
Thanks for sharing this data.
The lesson I draw from (1) is that in fact I should not think that conquering some areas help you conquer others. Rather, when entering some area, it is possible to draw local support in the first stages of a war. This updates me back towards thinking it’s costly to control a newly conquered area.
The lesson I draw from (2) is that you can continue to make use of some of the state capacity of the native power structures. But it seems like you have fairly low fidelity control (at least in the language barrier case, and probably in all cases, because you lack a lot of connections to informal power structures). This seems like mostly a wash?
Are these the same as the lessons you draw from this data?
(1) Local support doesn’t end after the first stages of the war, or after the war ends. I mentioned having favored local elites within one society/ethnicity continue to do most of the direct work in (2); colonizers also set up some groups as favored identities who did much of the work of local governance. For example, after the Spanish conquest, the Tlaxcala had a favored status and better treatment.
(2) Not sure why you’d expect low fidelity control to imply that it ends up as a wash in terms of extracting resources, can you clarify?
(2) It seems expensive to run a state (maintain power structures, keep institutions intact for future benefit, keep everything running well enough that the stuff that depends on other things running well keeps running). Increasing the cost by a large factor seems like it would reduce the net resources extracted.
It seems even more expensive if the native population will continue intermittently fighting you for 400 years (viz your rebellion fact)
Pongo, I think you are drawing the wrong lessons. Yeah, maintaining a state costs stuff. But there’s no law that says it has to cost more than the benefits. In fact historically the “benefits” have almost always been greater than the costs. This is why empires tend to grow bigger. If ruling territory is usually a net cost, well, you’d never see any empires at all really, because no city would be wealthy enough to maintain more than a few small pieces of territory.
That’s a good point!
Edit: I think my confusion is that there’s never been a whole world empire. Shouldn’t that happen if conquering a neighbouring region tends to make you more able to conquer other regions?
Alexander’s empire didn’t last. It seems like shortly after being founded, the Roman Empire has a lot of trouble and then (from my skim of Wikipedia) Diocletian sort of split it in four?
There’s also never been someone with 100% of the money, even though getting money tends to make it easier to get more money. (For example, you can just invest it in an index fund!)
There definitely are ways in which maintaining an empire gets harder the bigger it gets. For example, communication is more difficult over longer distances and higher language barriers. Also, ingroup cohesion becomes harder to maintain when the outgroup is weaker and more distant. And there are of course special cases—e.g. England for the Roman Empire—where holding on to a piece of territory is more trouble than it’s worth. Nevertheless it’s still true that, in most cases, conquering something gives you more resources, military force, etc.
I wrote some interesting speculation on this topic a few years ago. Roughly speaking, large premodern empires seem to consistently max out around roughly the same population size (~60M, with large-but-less-than-a-factor-of-two error bars). The few which manage to get larger than that through conquest rapidly split apart.
How rapidly?
One generation. The Mongols were the only empire to get a lot bigger than 50-70M (they capped around 110M), and they promptly split in a war of succession.
Oh yeah! What’s up with that?
Yeah, that seems to be true. My intuition is still having trouble with the success of converting the conquered military. Wikipedia tells me it was a big deal, and it remains surprising to me
I don’t really have a great answer to that, except that empirically in this specific case, Spain was indeed able to extract very large amounts of resources from America within a single generation. (The Spanish government directly spent very little on America; the flow of money was overwhelming towards Europe, to the point where it caused notable inflation in Spain and in Europe as a whole.) I don’t disagree that running a state is expensive, but I don’t see why the expense would necessarily be higher than the extracted resources?
OK, so maybe the idea is “Conquered territory has reified net production across however long a period → take all the net production and spend it on ships / horses / mercenaries”?
I expect that the administrative parts of states expand to be about as expensive as the resources they can get under their direct control. (Perhaps this is the dumb part, and ancient states regularly stored >90% of tax revenue as treasure?). Then, when you are making the state more expensive to run, you have less of a surplus. You also can’t really make the state do something different than it was before if you have low fidelity control. The state doing what it was doing before probably wasn’t helping you conquer more territory.
Well said. The implications for AI takeover are interesting to think about.
The post I was referring to
Related reading—https://mattlakeman.org/2020/06/25/polygamy-human-sacrifices-and-steel-why-the-aztecs-were-awesome/