Superintelligence 7: Decisive strategic advantage
This is part of a weekly reading group on Nick Bostrom’s book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI’s reading guide.
Welcome. This week we discuss the seventh section in the reading guide: Decisive strategic advantage. This corresponds to Chapter 5.
This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.
There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).
Reading: Chapter 5 (p78-91)
Summary
Question: will a single artificial intelligence project get to ‘dictate the future’? (p78)
We can ask, will a project attain a ‘decisive strategic advantage’ and will they use this to make a ‘singleton’?
‘Decisive strategic advantage’ = a level of technological and other advantages sufficient for complete world domination (p78)
‘Singleton’ = a single global decision-making agency strong enough to solve all major global coordination problems (p78, 83)
A project will get a decisive strategic advantage if there is a big enough gap between its capability and that of other projects.
A faster takeoff would make this gap bigger. Other factors would too, e.g. diffusion of ideas, regulation or expropriation of winnings, the ease of staying ahead once you are far enough ahead, and AI solutions to loyalty issues (p78-9)
For some historical examples, leading projects have a gap of a few months to a few years with those following them. (p79)
Even if a second project starts taking off before the first is done, the first may emerge decisively advantageous. If we imagine takeoff accelerating, a project that starts out just behind the leading project might still be far inferior when the leading project reaches superintelligence. (p82)
How large would a successful project be? (p83) If the route to superintelligence is not AI, the project probably needs to be big. If it is AI, size is less clear. If lots of insights are accumulated in open resources, and can be put together or finished by a small team, a successful AI project might be quite small (p83).
We should distinguish the size of the group working on the project, and the size of the group that controls the project (p83-4)
If large powers anticipate an intelligence explosion, they may want to monitor those involved and/or take control. (p84)
It might be easy to monitor very large projects, but hard to trace small projects designed to be secret from the outset. (p85)
Authorities may just not notice what’s going on, for instance if politically motivated firms and academics fight against their research being seen as dangerous. (p85)
Various considerations suggest a superintelligence with a decisive strategic advantage would be more likely than a human group to use the advantage to form a singleton (p87-89)
Typically new technologies do not allow small groups to obtain a “decisive strategic advantage”—they usually diffuse throughout the whole world, or perhaps are limited to a single country or coalition during war. This is consistent with intuition: a small group with a technological advantage will still do further research slower than the rest of the world, unless their technological advantage overwhelms their smaller size.
The result is that small groups will be overtaken by big groups. Usually the small group will sell or lease their technology to society at large first, since a technology’s usefulness is proportional to the scale at which it can be deployed. In extreme cases such as war these gains might be offset by the cost of empowering the enemy. But even in this case we expect the dynamics of coalition-formation to increase the scale of technology-sharing until there are at most a handful of competing factions.
So any discussion of why AI will lead to a decisive strategic advantage must necessarily be a discussion of why AI is an unusual technology.
In the case of AI, the main difference Bostrom highlights is the possibility of an abrupt increase in productivity. In order for a small group to obtain such an advantage, their technological lead must correspond to a large productivity improvement. A team with a billion dollar budget would need to secure something like a 10,000-fold increase in productivity in order to outcompete the rest of the world. Such a jump is conceivable, but I consider it unlikely. There are other conceivable mechanisms distinctive to AI; I don’t think any of them have yet been explored in enough depth to be persuasive to a skeptical audience.
Notes
1. Extreme AI capability does not imply strategic advantage. An AI program could be very capable—such that the sum of all instances of that AI worldwide were far superior (in capability, e.g. economic value) to the rest of humanity’s joint efforts—and yet the AI could fail to have a decisive strategic advantage, because it may not be a strategic unit. Instances of the AI may be controlled by different parties across society. In fact this is the usual outcome for technological developments.
2. On gaps between the best AI project and the second best AI project (p79) A large gap might develop either because of an abrupt jump in capability or extremely fast progress (which is much like an abrupt jump), or from one project having consistent faster growth than other projects for a time. Consistently faster progress is a bit like a jump, in that there is presumably some particular highly valuable thing that changed at the start of the fast progress. Robin Hanson frames his Foom debate with Eliezer as about whether there are ‘architectural’ innovations to be made, by which he means innovations which have a large effect (or so I understood from conversation). This seems like much the same question. On this, Robin says:
Yes, sometimes architectural choices have wider impacts. But I was an artificial intelligence researcher for nine years, ending twenty years ago, and I never saw an architecture choice make a huge difference, relative to other reasonable architecture choices. For most big systems, overall architecture matters a lot less than getting lots of detail right. Researchers have long wandered the space of architectures, mostly rediscovering variations on what others found before.
3. What should activists do? Bostrom points out that activists seeking maximum expected impact might wish to focus their planning on high leverage scenarios, where larger players are not paying attention (p86). This is true, but it’s worth noting that changing the probability of large players paying attention is also an option for activists, if they think the ‘high leverage scenarios’ are likely to be much better or worse.
4. Trade. One key question seems to be whether successful projects are likely to sell their products, or hoard them in the hope of soon taking over the world. I doubt this will be a strategic decision they will make—rather it seems that one of these options will be obviously better given the situation, and we are uncertain about which. A lone inventor of writing should probably not have hoarded it for a solitary power grab, even though it could reasonably have seemed like a good candidate for radically speeding up the process of self-improvement.
5. Disagreement. Note that though few people believe that a single AI project will get to dictate the future, this is often because they disagree with things in the previous chapter—e.g. that a single AI project will plausibly become more capable than the world in the space of less than a month.
6. How big is the AI project? Bostrom distinguishes between the size of the effort to make AI and the size of the group ultimately controlling its decisions. Note that the people making decisions for the AI project may also not be the people making decisions for the AI—i.e. the agents that emerge. For instance, the AI making company might sell versions of their AI to a range of organizations, modified for their particular goals. While in some sense their AI has taken over the world, the actual agents are acting on behalf of much of society.
In-depth investigations
If you are particularly interested in these topics, and want to do further research, these are a few plausible directions, some inspired by Luke Muehlhauser’s list, which contains many suggestions related to parts of Superintelligence. These projects could be attempted at various levels of depth.
When has anyone gained a ‘decisive strategic advantage’ at a smaller scale than the world? Can we learn anything interesting about what characteristics a project would need to have such an advantage with respect to the world?
How scalable is innovative project secrecy? Examine past cases: Manhattan project, Bletchly park, Bitcoin, Anonymous, Stuxnet, Skunk Works, Phantom Works, Google X.
How large are the gaps in development time between modern software projects? What dictates this? (e.g. is there diffusion of ideas from engineers talking to each other? From people changing organizations? Do people get far enough ahead that it is hard to follow them?)
If you are interested in anything like this, you might want to mention it in the comments, and see whether other people have useful thoughts.
How to proceed
This has been a collection of notes on the chapter. The most important part of the reading group though is discussion, which is in the comments section. I pose some questions for you there, and I invite you to add your own. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!
Next week, we will talk about Cognitive superpowers (section 8). To prepare, read Chapter 6. The discussion will go live at 6pm Pacific time next Monday 3 November. Sign up to be notified here.
- An overview of 11 proposals for building safe advanced AI by (29 May 2020 20:38 UTC; 213 points)
- Long Reflection Reading List by (EA Forum; 24 Mar 2024 16:27 UTC; 92 points)
- AI strategy nearcasting by (25 Aug 2022 17:26 UTC; 79 points)
- Six AI Risk/Strategy Ideas by (27 Aug 2019 0:40 UTC; 69 points)
- AI strategy nearcasting by (EA Forum; 26 Aug 2022 16:25 UTC; 61 points)
- Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios by (12 May 2022 20:01 UTC; 58 points)
- A “weak” AGI may attempt an unlikely-to-succeed takeover by (28 Jun 2023 20:31 UTC; 56 points)
- The Slippery Slope from DALLE-2 to Deepfake Anarchy by (EA Forum; 5 Nov 2022 14:47 UTC; 55 points)
- A newcomer’s guide to the technical AI safety field by (4 Nov 2022 14:29 UTC; 42 points)
- Superintelligence reading group by (31 Aug 2014 14:59 UTC; 31 points)
- Will AI undergo discontinuous progress? by (21 Feb 2020 22:16 UTC; 27 points)
- The Slippery Slope from DALLE-2 to Deepfake Anarchy by (5 Nov 2022 14:53 UTC; 17 points)
- A newcomer’s guide to the technical AI safety field by (EA Forum; 4 Nov 2022 14:29 UTC; 16 points)
- 's comment on How common is it for one entity to have a 3+ year technological lead on its nearest competitor? by (19 Nov 2019 1:57 UTC; 4 points)
- 's comment on Where are you giving and why? by (EA Forum; 13 Dec 2014 1:54 UTC; 2 points)
- Looking at how Superforecasting might improve some EA projects response to Superintelligence by (EA Forum; 29 Aug 2017 22:22 UTC; 2 points)
Example of ‘decisive strategic advantage’: The Spanish in America, 16th century.
Although this example does not present research by a small group, it does show what happens when a small group holds superior technology: A few hundred men conquered a large empire, without resupply or reinforcement from the Old World. This happened at least twice—Cortés and Pizarro—indicating that it was not a fluke.
Well, kind of. The Spaniards succeeded by killing off/holding hostage their opposition’s leadership cadre, and through the accidental deployment of biological weapons-they were immune the illnesses that spread from them to the Aztecs, Incas and Mayans.
They won the battle, and they might have won later if they engaged in a full-scale invasion, but it is not completely clear form historical outcomes that they actually had a decisive strategic advantage.
Sure, the diseases were accidental. But beyond that, they used various methods, including hostage-taking and building alliances, and a few hundred men defeated millions, in two cases. Doesn’t that show a decisive strategic advantage?
Way more than a few hundred—local rebellions and malcontents joined their cause in droves on the (mistaken) assumption that they would remain in an advantaged state in the aftermath.
I think you are looking at this wrong. Yes, they had help from local rebellions and malcontents. So would an AGI. An AGI taking over the world wouldn’t necessarily look like robots vs. humans; it might look like the outbreak of World War 3 between various human factions, except that the AGI was manipulating things behind the scenes and/or acting as a “strategic advisor” to one of the factions. And when the dust settles, somehow the AGI is in charge...
So yeah, I think it really is fair to say that the Spanish managed to conquer empires of millions with just a few hundred men. Twice.
Non-immunity to illnesses is very important to us. Our computers and network infrastructure is more or less immune against script-kiddies and polymorphal viruses and standard attack schemes.
Our systems are not immune against tailored attacks from intelligence agencies or AIs.
Minor nitpick: Though the Aztecs and Incas were empires that were defeated by the Spanish, the Mayan Empire had long since fallen when the Spanish arrived. There were Mayans, but only in the sense of the ethnic group that remained in small villages in the region that had formerly been a powerful kingdom.
If someone has some money, they can invest it to get more money. Do you know what the difference is between money and intelligence that makes it plausible to expect an abrupt intelligence explosion, but reasonable to expect steady exponential growth for financial investment returns?
One answer is that using your intelligence to to improve your own cognitive architecture is an entirely new field of investment. The economic growth that accrues from modern investing looks steady from inside the modern economy, but it’s explosive from the perspective of a premodern society.
One distinction that may matter here is that something is only counted for as money if it is actually traded between distinct agents. If I got 10 000 times better at making houses from scratch and made a house for me but no one else, there would be no increase in the local economy. If I started selling houses, this would be different.
Superintelligence can be thought of as a large island, where there is a lot of internal trade going on that is invisible to the outside economy. If that was counted as value, then the economy would also be exploding. This internal trade opacity that is proper of intelligence and other intra-agent changes seems to be responsible for the distinctive growth extrapolations.
While not exactly investment, consider the case of an AI competing with a human to devise a progressively better high-frequency trading strategy. An AI would probably:
be able to bear more things in mind at one time than the human
evaluate outcomes faster than the human
be able to iterate on its strategies faster than the human
I expect the AI’s superior capacity to “drink from the fire hose” together with its faster response time to yield a higher exponent for the growth function than that resulting from the human’s iterative improvement.
A more realistic example would be “competing with a human teamed up with a narrow AI”.
You’re right, that is more realistic. Even so, I get the feeling that the human would have less and less to do as time goes on. I quote:
As another data point, a recent chess contest between a chess grandmaster (Daniel Naroditsky) working together with an older AI (Rybka, rated ~3050) and the current best chess AI (Stockfish 5, rated 3290) ended with a 3.5 − 0.5 win for Stockfish.
I don’t think an article which compares a hedge fund’s returns to the Dow (a price-weighted index of about 30 stocks!) can be considered very credible. And there are fewer Quant funds, managing less money, than there were 7 years ago.
One difference is hardware overhang. When new AI technologies are created, many times the amount of hardware they need is available to run them.
However, if an international agreement was reached in advance, and some of the AGI sandbox problems were solved, we might be able to restrict AGI technology to a bounded amount of hardware for a time-if theoretical results which we do not have yet showed that was the appropriate course of action.
We have all kinds of work to do.
There are some qualitative shifts in growth rate with investment. For example, very small amounts of money basically cannot be invested in the financial due to the high fixed costs associated with money laundering regulation, etc.
On the other hand, the more income people have, the more they consume, which reduces growth. Levels of consumption deemed adequate by our ancestors, and indeed consistent with high savings rates, are not viewed as acceptable any more.
Market is more or less stabilized. There are powers and superpowers in some balance. (gain money sometimes could be illusion like bet (and get) more and more in casino).
If you are thinking about money making—you have to count sum of all money in society. If investments means bigger sum of values or just exchange in economic wars or just inflation. (if foxes invest more to hunting and eat more rabbits, there could be more foxes right? :)
In AI sector there is much higher probability of phase-transition (=explosion). I think that’s the diference.
How?
Possibility: There could be probably enough HW and we just wait for spark of new algorithm.
Possibility: If we count agriculture revolution as explosion—we could also count massive change in productivity from AI (which is probably obvious).
This requirement seems too strong.
First a project doesn’t need to monopolize more than half of the world economic throughput to succeed in obtaining a decisive strategic advantage. A small targeted fraction of the economy (say the armies of all countries in the northern hemisphere) may be more than enough. If the pond is chosen correctly, the fish needn’t be that big.
Second: a project that is able to efficiently steal at a faster absolute pace than the world economy grows could over the long run dominate it by capturing most created value, without having to go through the honest toil itself.
Third: if we narrow our defition of achievement of a strategic advantage as being merely the ability to stop the behemoth of evolutionary selection pressures at the many levels where they exist—the original intent of the concept of Singleton in papers like “What is a Singleton?” and “The Future of Human Evolution”—then even less would be necessary. A Singleton could be a specialist in it’s ability to countenance abrupt changes in evolution—and nothing else—and it would still serve the purpose of avoiding maximal efficiency clans, thereby protecting flamboyant displays and possibly happiness and consciousness on the side.
Note that a 10,000-fold increase would leave you with a modest fraction of total output. A 1,000-fold increase would still leave you smaller than world’s militaries.
The theft scenario relies on you being better at theft than the rest of the world. From the outside AI looks more suited to productive activity than war (which also requires big capital investments in personnel and equipment), so I normally think of this as being a counterbalancing factor (that is, it seems significantly more likely that an economically dominant firm would find many of its productive assets confiscated by force, than that a productive firm would begin stealing the world’s resources without serious repercussions). Of course a primary consideration in this discussion is the nature of conflict; in the current world a sufficiently sophisticated AI might fare reasonably well in all-out conflict due to the world being completely unprepared, but I would be quite surprised if that were still the case when the development of human-level AI actually looked plausible. It seems to again come down to the possibility of a rapid and unexpected jump in capabilities.
The most realistic paths along these lines seem to depend on abrupt changes in the importance of different resources. For example, you might think that AI research capacity would change from a small part of the world to the lion’s share (or some even more specific kind of research). If such a change were slow it wouldn’t much matter, but if it were fast it could also lead to a decisive strategic advantage, as a company with 0.01% of the world’s resources could find itself with 90% after a drastic reevaluation.
Of course, whether such an abrupt change should happen is again closely related to whether an abrupt change in capacities will happen (though it relies on the further assumption that such a change is unanticipated). Overall I think that it is not too likely, and that the probability can be driven down substantially by clearer communication and decision-making. It still seems like a scenario worth thinking about.
All told, agree with Bostrom that there are a number of reasons to think that very fast changes in capability would lead to a decisive strategic advantage, especially if it came as a surprise to most of the world. Short of rapid changes, I don’t see much reason to think of AI as exceptional.
Regarding evolution I disagree completely with the implicit model of how the future will go or why it might contain morally valuable things. See here for a discussion of my view. On my view, a “singleton” with this more limited capacity would not be very important. Indeed, I’m not even sure what it would mean.
There are two mixed strategies worth noting that facilitate theft.
One is propagating and stealing identities. Instead of stealing from a company, you merely become the company wherever it is represented in coded from (name, signature, brand, bank account, online representation etc...). Propagating identities would be the just creating other AI’s, subsets of you or changed copies, that by design were made such that they are considered different entities, and therefore you, the AI, cannot be held accountable for their actions. I’d expect Ben Goertzel to have interesting thoughts on this, though no pointers come to mind.
The other is mixing theft, production and destruction of coordination. The Italian mafia for instance, did all three. They control legitimate businesses, they control part of the law-enforcement agency (that is, they act as if they were the police or legitimate power) and they destroy crucial nodes of coordination within the legitimate police force. So not only they steal power, but they also make sure that (valuable) coordination is shattered.
Promoting anti-coordination is an interesting move in warfare, and I see no reason why an AI would refrain from using it to obtain a decisive strategic advantage, if growth in power and theft were not proving to be enough.
I agree with your conditional statements. IF coordination and clear communication THEN low probability of abrupt transformational deleterious change. IF rapid and strategically relevant power obtained THEN it was likely preceded by swift changes in importance of different resources.
Our disagreement seems to hinge on the comparative likelihood of those premises, more than different views on how systems work in this context.
I’ll transfer discussion of evolution—where real disagreement seems to lurk—to your blog to save brainpower from readers here.
Diametral opposing to theft and war szenarios you discuss in your paper “Rational Altruist—Why might the future be good?”:
I fully agree with you and this aspect is lacking in Bostrums book. The FOOM—singleton theory intrinsically assumes egoistic AIs.
Altruism is for me one of the core ingredience towards sustainably incorporating friendly AIs into society. I support your view that the future will be more altruistic than the present: AIs will have more memory to remember behavior of their contacts. The Dunbar’s number of social contacts will be higher. Social contacts recognize altruistic behavior and remember this good deed for the future. The wider the social net the higher is the reward for altruistic behavior.
Recent research confirms this perspective: Curry, O., & Dunbar, R. I. M. (2011). Altruism in networks: the effect of connections. Biology Letters, 7(5), 651-653:
The idea of AIs and humans monitoring AIs in a constitutional society is not new. Stephen Omohundro presented it in October 2007 at the Stanford EE380 Computer Systems Colloquium on “Self-Improving Artificial Intelligence and the Future of Computing”.
I transcribed part of the Q&A of his talk (starting 51:43)
Stephen Omohundro:
No, that’s wrong. The speed of takeoff is largely a technical question; from a strategic planning POV, going through a rapid takeoff likely makes sense regardless of what your goals are (unless your friendliness design is incomplete /and/ you have corrigibility aspects; but that’s a very special case).
As for what you do once you’re done, that does indeed depend on your goals; but forming a singleton doesn’t imply egoism or egocentrism of any kind. Your goals can still be entirely focused on other entities in society; it’s just that if have certain invariants you want to enforce on them (could be anything, really; things like “no murder”, “no extensive torture”, “no destroying society” would be unoffensive and relevant examples) - or indeed, more generally, certain aspects to optimize for—it helps a lot if you can stay in ultimate control to do these things.
As Bostrom explains in his footnotes, there are many kinds of singletons. In general, it simply refers to an entity that has attained and keeps ultimate power in society. How much or how little it uses that power to control any part of the world is independent of that, and some singletons would interfere little with the rest of society.
Your argumentation based on the orthogonality principle is clear to me. But even if the utility function includes human values (fostering humankind, preserving a sustainable habitat on earth for humans, protecting humans against unfriendly AI developments, solving the control problem) strong egoistic traits are needed to remain superior to other upcoming AIs. Ben Goertzel coined the term “global AI Nanny” for a similar concept.
How would we get notion of existence of a little interfering FAI singleton?
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
The AI’s values would likely have to be specifically chosen to get this outcome; something like “let human development continue normally, except for blocking existential catastrophes”. Something like that won’t impact what you’re trying to do, unless that involves destroying society or something equally problematic.
Above hypothetical singleton AI would end up either sabotaging the project, or containing the resulting AI. It wouldn’t have to stop the UFAI before release, necessarily; with enough of a hardware headstart, later safe containment can be guaranteed fine. Either way, the intervention needn’t involve attacking humans; interfering with just the AI’s hardware can accomplish the same result. And certainly the development project shouldn’t get much chance to fight back; terms like “interdiction”, “containment”, “sabotage”, and maybe “police action” (though that one has unfortunate anthropomorphic connotations) are a better fit than “war”.
We could test it in thought experiment.
Chess game human-grandmaster against AI.
it is not rapid (not checkmate in begining).
We could also suppose one move per year to slow it down. It bring to AI next advantage because it’s ability to concentrate so long time.
capabilities
a) intellectual capabilities we could suppose at same level during the game (if it is played in one day, otherwise we have to think Moore’s law)
b) human lose (step by step) positional and material capabilities during the game. And it is expected
Could we still talk about decisive advantage if it is not rapid and not unexpected? I think so. At least if we won’t break the rules.
Before crossover, most of the improvement is coming from the rest of the world. Note that this suggests the rest of the world is also improving similarly fast in that period.
Our canonical example, the Manhattan Project, may have been at that level. If the US had chosen to attack Soviet cities with the few atom bombs that they had in the first year, then quickly replenishing their supply quickly while the Soviet Union remained off-balance (scary thought), the US could have dominated the world. Thankfully, they did not, but it does suggest that a superweapon used without moral constraint could have world-dominating effect.
Alternate history is not falsifiable, of course, but that scenario doesn’t look all that likely to me. Russia successfully recovered from losing a very large chunk of its territory, a great deal of its army, and most of its manufacturing capacity to Germans in 1941-1942. Losing a few cities (even assuming the bombers could get through—there were no ICBMs and Russia in 1945 had a pretty good air force and AA capabilities) would not cripple Russia. I would guess that it would just incentivize it to roll over the remainder of Europe. It’s not like Stalin ever cared about casualties.
Good point, but I think Bostrom’s point about risk aversion does much to ameliorate it. If the US had had a 50% chance of securing global hegemonicy, and a 50% chance of destruction from such a move, it probably would not have done it. A non-risk-averse, non-deontological AI, on the other hands, with its eye on the light cone, might consider the gamble worthwhile.
Quite right, hard to tell.
Plausible, though note that while the US (a large fraction of the world) may have dominated the world by that route, it would have been hard for any small group involved to do so.
And very hard for them to KEEP domination.
Do you think an AI project will obtain a decisive strategic advantage? Why/Why not?
Probably not: Some nerdy superintelligent AI systems will emerge but humans will try their utmost to shut off early enough. Humankind will become very creative to socialize AGI. The highest risk is that a well funded intelligence agency (e.g. NSA) will be first. Their AI system could make use of TAO knowledge to kill all competing projects. Being nerdy intelligent it could even manipulate competing AI projects that their AIs get “mental” illnesses. This AI will need quite a long time of learning and trust-building until it could take over world dominion.
Bostrums book is a wake-up call. In his book presentation Authors@Google (final six minutes Q&A) he claimed that only half a dozen scientists are working full time on the control problem worldwide. This is by far not enough to cope with future risks. More and effective funding is needed.
Bostrum does not want to destroy his straightforward FOOM-DOOM scenario. He does not discuss technical means of monitoring but only organizational ones.
I fully agree with Bostrum that too few people are working on the control problem. In 2007 Stephen Omohundro asked for synergistic research between psychologists, sociologists and computer system engineers on the control problem. Today we have to conclude that progress is limited.
We have many technical options at hand to prevent that an AI project can obtain decisive strategic advantage:
Prevent content overhang of powerful inside knowledge (taboos, secrecy, fighting organized internet crime).
Prevent control access by keeping life supporting infrastructures independent from internet.
Prevent hardware overhang by improving immunity to cyberattacks (hard crypto).
Law against backdoors in any system.
Transparency (AI development, cyberattack monitoring).
Develop weak-AI with superintelligent capability of monitoring AIs (thought police)
Develop fixed social conscience utility function.
“What is the best way to push it [risk of doom] down.” was Bostrums last sentence at his book presentation. This should be a further point of our discussion.
Well, no life form has achieved what Bostrom calls a decisive strategic advantage. Instead, they live their separate lives in various environmental niches.
One way to look at it is this: Suppose a dominant AGI emerged which was largely running the planet and expanding into the galaxy.
Would it then be impossible to engineer another AGI which survived modestly in some niche or flew off at near the speed of light in a new direction? No.
For the first AGI to be the only AGI, all other AGI development would have to cease without such “niche AGIs” ever being created.
An AGI could be extremely unobtrusive for tens of thousands of years at a time, and even be engaging in some form of self-improvement or replication.
“Sterilizing” matter of all of the “niche AGI” it contains could be quite an involved process.
If you are trying to sterilize, then the level you seek is not only AGI level, but also replicators, which can mobilize lots of resources within a niche.
That AGI does not need to stay the only one to solidly stay in power. Since it has been playing the game for longer, it would be reasonable for it to be able to keep tabs on other intelligent entities, and only interfere with their development if they became too powerful. You can still have other entities doing their own thing, there just has to be a predictable ceiling on how much power they can acquire—indeed, that is the idea behind FAI programming: Have the FAI solve some fundamental problems of society, but still leave a society composed of plenty of other intelligences.
This would be made easier if reality is virtualized (i.e. if the singleton AI handles building and maintaining computronium infrastructure, and the rest of society runs as programs using some of the resources it provides); you don’t need to monitor every piece of matter for what computations it might carry out, if you’ve limited how much computation power you give to specific entities, and prevented them from direct write access to physical reality, to begin with.
In the end, I think eventual decisive strategic advantage for a single AI is extremely likely; it’s certainly a stable solution, it might happen due to initial timing, and even if doesn’t happen right then, it can still happen later. It’s far from clear any other arrangement would be similarly stable over the extremely long time horizons of relevance here (which are the same as those for continued existence of intelligences derived from our civilization; in the presence of superintelligent AGIs, likely billions of years). In fact, the most likely alternative to me is that humanity falls into some other existential catastrophe that prevents us from developing AGI at all.
Ants are probably good example how could organisational intelligence (?) be advantage.
According to wiki ″Ants thrive in most ecosystems and may form 15–25% of the terrestrial animal biomass.″. See also google answer, wiki table or stackexchange.
Although we have to think careful—apex predators does not use to form large biomass. So it could be more complicated to define success of life form.
Problem of humanity is not only global replacer—something which erase all other lifeforms. It could be enough to replace us in our niche. Something which globally (from life viewpoint ) means nothing.
And we dont need to be totally erased to meet huge disaster. Decline population to several milions or tousands… (pets or AI) … is also unwanted.
We are afraid not decisive strategic advance over ants but over humans.
What do you think of Paul’s view? (‘Another view’ above)
One possibility to prevent smaller group gain strategic advantage is something like operation Opera.
And it was only about nukes (see Elon Musk statement)...
On selling versions of different AGIs to others, something that does not come up in the text but does in the above paragraphs:
An AGI is a weaponizable technology. Technology capable of running an instance of the code should not be for sale any more than you can buy a tank or an ICBM-it only gets done by international agreement.
Many AGI components should not be for sale individually, just as you cannot buy some parts of a tank, and access to them could be restricted even before they are ever assembled into an AGI or during the time when we only believe they might end up being part of one solution pathway.
Even a remote-controlled minivan is something that would require heavy licensing and ownership restrictions. Measures will be taken so that the owners of self-driving cars have great difficulty in reprogramming the car’s control systems.
If we develop an early AI communicating in a slightly stilted way, but with sufficient reasoning and NLP skills to largely replace teachers, accountants, stockbrokers or journalists, then leave it is a shared service like a search engine whose code we never see.
If AGI is used to tell self-driving cars where to drive and robots what to build as part of a master strategy to generate material abundance, maybe the cars can contain enough AI to navigate and pick up cargo, but bundling a complete AGI inside of a device like a car or a robot seems pretty hazardous.
Along with AI containment, the question of which devices get how much reasoning power also needs an answer. One AGI per car or per robot, like we see in sci fi sometimes, seems like a very wrong idea.
However, a singleton with a decisive strategic advantage does not rule out devices which are running separate AGIs. If devices are physically distant, then the speed of light may require such configurations.
Elements of these considerations need additional thought...
Bostrom does not purport to be an expert on international relations, but the chapter does correctly reference one of the critical turning-points of the Cold War-the failure of the Soviet Union to accept the Baruch Plan.
The Baruch Plan would have placed all nuclear power and nuclear weapons technology in the hands of a multilateral organization developed as part of The United Nations. The Soviet Union refused because communist nations were out-voted in the UN.
We can chalk this outcome up to the personality of Stalin, but apparently the Soviets did not feel that they were more secure in a world where nuclear technology was governed multilaterally.
Nations could consider revisiting The Baruch Plan again just for the purpose of governing nuclear weapons. From my perspective, however, we also need to consider a multilateral framework to govern AI and AGI technology.
This framework, which would have similarities and difference from The Baruch Plan, could begin as a means to govern the use of drones and autonomous vehicles in conventional warfare. Drones are going to be used by UN peacekeeping mission, and we should get a jump on selecting the rules.
As the framework is tuned and improved, perhaps it can be expanded. However, strengthening of multilateral institutions will be impossible absent a complete understanding of the perceived interests of all relevant parties.
While The Baruch Plan failed, today we may have time to test frameworks more carefully before they are debated by the international community, and we also have some additional intellectual tools.
Political considerations may dictate that a “managed roll-out” of AI technology is necessary, regardless of when these technologies are first discovered. This is as important a sub-topic of this discourse as any other.
People interested in this particular question may be interested in Peter Thiel’s discussions about Last Mover Advantage, available in many media if you google for it.
How useful do you think looking at bombs and space projects (box 5) is for picturing what will happen with AI? (p79-82)
While space projects are more like the Peacock’s tail (flamboyant displays, a theme we will return to later in the book), bombs are more like poison indicators (such as colors black red and yellow in snakes).
AI is actually powerful. As Elon puts it, with AI, you are summoning a demon.
Comparing AI with space projects could be beneficial on some dimensions, but it seems to me that comparison with bombs and other military technology, that are more like the ability to poison or summon demons, should take evidential precedence.
Did you find anything this week particularly unpersuasive?
On p82 Bostrom argues that even if one research group has access to another’s code, it would still take them months to change approach, which could lead to the leaders getting a full FOOM’s worth of headstart. But I don’t really understand why he thinks this. If I have access to your code, can’t I just run it as soon as I notice you’re running it? True, you could obfuscate it, or protect your production-box more carefully. But if I have that good monitoring systems, I’m probably a government, in which case I can afford to have a team dedicated to reverse-engineering your approach.
Some reasons that come to mind:
It might take you a while to come to the conclusion that your technology won’t overtake theirs.
You might have slightly different computational resources, and the code might be specific to that.
I don’t see anything about access to code on p82. Are you inferring that from “closely monitor”?
Yes, and good (implicit) point—perhaps Nick had in mind something slightly less close than access to their codebase.
Bostrom says the organizational route to superintelligence would require much of the world economy (p83). I don’t understand why that would be so. Does someone else?
That seems to depend on the size of the world economy. There may be an absolute minimum of organizational ability for some complex entities to exist—Toby Ord points out that if the human population was much smaller, we wouldn’t be able to create the infrastructure that allows for computers.
If the absolute structure of an easy to create Superintelligence requires hundreds of millions of agents in concert, it would appear that a large fraction of the economy is necessary for an observer of a world with a few billion people.
A more obvious note is that the closer you are to representing half the economy, the easier it is to reach the point in which you control 51% of it (which may coincide with strategic advantage in some cases).
What did you find most interesting this week?
Bostrom makes an interesting point that open literature tends to benefit small teams relative to large ones.
Also, he points out that it is easier for a government to nationalise the most successful projects than to ban all projects. The most successful projects are likely to have significant funding, hire well-known experts, go to conferences, etc. Small basement projects, on the other hand, could much more easily escape detection. Unfortunately, this is a pro-arms-race dynamic: it is easy for governments to try to outrace small teams with their own teams than to ban development.
The point about countries defecting from global collaborations is also interesting. It’s relatively hard to defect on a nuclear-sharing project: you actually need to build your own reactors, acquire fissile material, etc. But running your own AI project might only requires you to steal some thumb drives.
The “Monitoring” subsection. Bostrom makes good points about why state actors would be likely to seek to monopolize AGI development, if they took it to be a threat to their power monopolies. I hadn’t given the possibility sufficient consideration until this point. These actors do seem disinterested so far, but (as Bostrom points out) that might well be a pretense, and even if not it doesn’t mean they’ll remain so for too long to matter.
Bostrum underestimates complexity of learning, compare Robin Hanson’s criticism “I Still Don’t Get Foom” on his book.
Assume following small team scenario that could reach a decisive advantage: A hedge fond seeks world dominion and develops in secrecy a self-improving AI. Following skills shall reach superhuman capabilities:
cyberattack and cryptanalysis
semantic comprehension of tech and business documents
trading strategy
Latest when this AI reaches a decisive strategic advantage over other market players they will acknowledge this instantly. Insider trade investigations will soon reveal that this hedge fond was breaking the law.
This AI had not yet the time to learn all other skills needed for world dominion:
creating a globally operating robot guard
creativity to develop countermeasures against all thinkable switching-off scenarios
influencing humans (politicians, military brass and mass propaganda).
A human is capable to run a large company or a government earliest at the age of 30. To learn how to influence people has very few speedup options and books are of little help. The AI has to acquire real insight comprehension of human values and motivations to become a successful world leader.
A quick-and-dirty AI without carefully designed utility function and wisdom from humane values will evoke the utmost available power of the entire world united to switch off this AI.
LOL. Do you really think so?
Yes indeed I am convinced that 30 years of learning is a minimum for running a large company or a government. I compiled data from 155 government leaders of five countries. On the average they took office for their first term at the age of 54.3 years.
For my above statement allow me to substract 2 standard deviations (2 x 8.5 = 19 years). A government leader is therefore with 97.7% probability older than 35.3 years when he takes office for the first time. The probability of a government leader being younger than 30 years is 0.22%, calculated from the standard distribution. William Pitt the Younger became youngest UK Prime Minister in 1783 at the age of 24. He is the only one younger than 30 in my data set.
Lumifer, could you be so kind to compile a similar statistical evaluation of about top 150 companies of these five countries. I can help you with DAX 30. I am expecting that the average will be lower for following reasons:
Trustbuilding is more time consuming in politics than in business and involves more people.
Some startup pioneers started very young and their companies grew extremely quick.
Right tail of age distribution >65 years will be thin for companies.
After this we both know who of us might LOL. We both know that a doom scenario is a possible outcome of AI development. My intention is to understand what knowledge is needed to rule the world and how it is possible to hide this content to slow down the learning curve of an AI capable of taking over everything.
Your data shows what typically happens. That’s a bit different from what “a human is capable” of.
Technically speaking, a single counter-example overturns your claim.
The Manhattan Project [Tubes] managed to remain secret even though scientists and researchers themselves thrive on the open community which in turn feeds new ideas through innovation and response to continue the scientific dialectic. A spy in the works [Fuchs] saved years re the Soviets to get it’s bomb together. Then there was the Soviet sputnik which too surprised esp Amerika. And went paranoid because they thought in cold-war terms not scientific ones. Not at first anyway. Now AI research and breakthroughs-spyware aside-can keep researchers autonomous. Computers are shrinking. What took electro-shock-rock bands a full room of hardware to synthesize their sound is now done in a tiny nifty box top model the size of a thick book. Research will be done in secret [spyware aside]. After all terrorists nearly get away with their secret schemes so devoted scientists can and will do the same in the interest of glory which they fully deserve. We will be in for some surprises I hope