Historically have you found an example of a worldwide pause for a strategically relevant technology? Something negotiated by agreement that wasn’t immediately defected on?
Skimming https://en.wikipedia.org/wiki/Arms_control I see a couple interesting tidbits: the Washington Naval Conferencedid result in “an effective end to building new battleship fleets, and the few ships that were built were limited in size and armament”. This seems to fit your reference class!
And then after the Treaty ended in 1936 (with defections throughout), we get:
BB is battleship, there’s only 10 new ones, but 141! aircraft carriers. In tonnage this would fit the reference class of “after the pause, more ships than ever”. Also, the USN had discovered the carrier hard countered all other surface combatants. The AI equivalent might be post pause few new GPUs are built, it’s all neural accelerator ASICs.
Another view, close to 10x the scale in 4 years. This looks like a fit for your reference class.
It’s a tight fit another way. What you would expect in a timeline where all the world powers do agree to some kind of AI limitations. During that time period, similar to the post washington navy treaty, you might get an equivalent to:
The United States developed better technology to get better performance from their ships while still working within the weight limits, the United Kingdom exploited a loop-hole in the terms, the Italians misrepresented the weight of their vessels, and when up against the limits, Japan left the treaty.
You can imagine this happening many ways for AI, where each party to the pause finds a different route to evade it.
Eventually what would happen is breakout, where one of the superpowers reaches the capability level for exponential growth or achieves some other advantage that threatens the others. The other parties to the pause must unpause or die, and you would see something like the above. Looks like humans could double their warships in existence 3 times in ~4 years, presumably robots constructing robots and weapons will be faster.
I have now looked into this example, and talked to Bean at Naval Gazing about it.
I founddata for the total tonnage in many countries’ navies from 1865-2011. It seems to show overhang for the US navy during the interwar years, and maybe also for the Italian navy, but not for any of the other great powers.
Bean has convinced me that this data is not to be trusted. It does not distinguish between warships and auxiliary ships, or between ships in active duty and in reserve. It has some straightforward errors in the year the last battleship was decommissioned and the first nuclear submarine was commissioned. It seems to be off by a factor of 10 at some points. There seems to not be a dataset covering this time period that is even accurate to within a factor of 2 for me to do this analysis with.
Even if the data were reliable, it’s not clear how relevant they would be. It seems as though the total tonnage in the navy of an industrialized country is mostly determined by the international situation, rather than anything about the technology itself. If the trend is going up, that mostly means that the country is spending a larger fraction of its GDP on the navy.
I don’t think that this changes the conclusions of the post much. Overhang is uncommon. It happened partially for land speed records, but it was not sustained and this is not a technology that people put much effort into. Something that looks sort of similar might have happened with the US navy during the interwar period, but not for the navies of any of the other great powers, and the underlying dynamics seem to be different.
What are we trying to model here or find examples of?
Here’s what I think we’re trying to model: if a technology were isolated and for whatever reason, development was stopped, then during the ‘stopped’ period very little effort is being put into it.
After the ‘stopped’ period ends, development resumes and presumably progress is proportional to effort, with an unavoidable serial part of the process (from Amdahl’s law/ Gant charts show this) restricting the rate that progress could be made at.
For US Navy tonnage : without a washington Naval treaty, and a Great Depression and a policy of isolation, the US Navy would presumably have built warships at a steady rate. They did not, as shown in your data.
However, during this prewar period, other processes continued. Multiple countries continuously improved aircraft designs, with better aerodynamics (biplane to mono), carrier launching and landing, ever larger and more powerful engines, dive and torpedo bombing, and other innovations.
So even though very few ships are being built, aircraft are being improved. Now Pearl harbor, and unpause. All out effort, which shows in the data you linked.
But we don’t have to trust it, all that really matters is the aircraft carrier numbers, nothing else. As it turned out, the carrier was a hard counter to everything, even other carriers—the other ships in a carrier battle group are there to hunt submarines, supplement the carriers antiaircraft fire, and resupply the carriers. While there were direct gun battles in late ww2 in the Pacific theater, better admirals could probably have avoided every battle and just sank all the enemy ships with aircraft. Shooting down enemy aircraft was also way easier, it turned out, to do with aircraft.
So only the left column matters for the model, and you also need the 0 point. There were seven fleet aircraft carriers and one escort carrier at t=0, beginning of ww2.
If we count the escort carriers at 30% of a fleet carrier, and there were 27 new fleet carriers, then the total of “weighted” carriers went from 7.3 (prewar) to 61.2 (weighted, 1945).
8.38x increase in the most relevant parameter.
The data you don’t trust, 1940 tonnage is 1956867 and 1945 tonnage is 11267550. Increase of 5.75x.
In terms of “relevant” tonnage obviously aircraft carriers, due to their role as a hard counter with 1940s era technology, is all that matters.
From the perspective of Japanese admirals, over 4 years of war, they faced 8 times the warships produced during the entire prewar period. This is the issue, their battle plans could not scale to handle this. Had the Americans been building carriers at a steadier rate the entire time, the Japanese would never have attacked. The rate of increase turned out to be a strategic surprise that doomed the Japanese side of the war.
Predictions for AI:
It’s frankly hard to imagine a world where an AI pause is actually negotiated. Remember, if any major party says no, nobody can afford to pause. Say it happens:
During the pause, there would be some flops limit on model scale, above which requires likely prohibitively expensive precautions.
During the pause period, people would be experimenting with below threshold ML algorithms. They might build app stores where sub AGI algorithms can be easily licensed and combined into integrated systems, where “ML microservices” give an integrated system some of the benefits of a real AGI. Probably a network* of ML microservices could be built that can perceive, interpret the environment, compare the current state to a goal state, consider many possible actions using an environment/robotics sim that is a service, and then choose the best action. It might not even be that much less effective than a “real” AGI model that is integrated for some purposes like factory work.
The app stores would also likely contain key innovations that current AI stacks are missing. A series of software intercompatibility frameworks (anyone can drive a robot with any model), cloud hosted realistic evaluation environments (these are things like realistic robotic environment sims), composable robotics backends, formally proven stack components (so robotics can be certified as life safety using new AI models), and likely cloud hosted AI improving AI services. (this is where it is possible to pay for improvements to an AI stack using a service that is cloud hosted, this is possible even with sub AGI RL models).
Note that the things I just mentioned do not exist today, everyone on autonomous cars has rolled their own duplicate version of it, with only a little bit of sharing with platforms like ROS.
During the pause period, Moore’s law continues to whatever level it is able to continue (physics is obviously an ultimate limit to an extent), but the experience curve would continue to make transistors cheaper even after they cannot be made smaller. AI models are close to embarrassingly parallel so they benefit linearly from the number of transistors.
During the pause period, ASIC architectures for AI chips are developed. On previous examples like bitcoin mining and others, ASICs have proven to be substantially faster and more powerful efficient for the same quantity of transistors.
So after the pause period, all these innovations hit all at once, and the rate of AI increase becomes very high, potentially uncontrollable. You can see why: all the obstacles to letting true AGI models immediately control robots have been smoothed away, AGI models can be used to improve each other even if a different company owns the model (that cloud hoisted AI improving service), and the realistic evaluation environments allow the new AGI models to quickly be evaluated and improved on. The compute is also faster, and the pause was likely dropped due to a war, so the new AI effort is probably flat out.
So I’m convinced. With a gears level model I think the overhang idea has a very high probability of happening if a pause occurred. It’s just super rare for there to be any pause at all historically, and there probably will not be one for AI.
Oh one bit of confusion: say there was a 5 year pause between 2030 and 2035. Am I saying that in 2040, is the amount of AI progress the same as in a 2040 with no pause? No. Obviously the pause slowed down progress, the fastest progress is when people are working on AI at full speed from 2023 to 2040. But it does create a discontinuity : 2035 after the pause is a more interesting year than the “no pause” 2030-2035 period.
*the reason a network of separate models isn’t the same as an AGI is each model gets trained separately, and it would be illegal to train the models on information from when they were assembled into a system. Each model has to remain static, validated on separate benches. And the models must operate in a human interpretable way, humans would have to be able to clearly understand both the input and output. This is how current autonomous car stacks, except comma.ai’s, already work. Assuming each model can be up to 10^26 flops—that’s quite a bit of headroom, probably easily enough to do a lot of “AGI” like things using a network of 10-100 subsystems, each under 10^26. The human brain works this way.
>But we don’t have to trust it, all that really matters is the aircraft carrier numbers, nothing else. As it turned out, the carrier was a hard counter to everything, even other carriers—the other ships in a carrier battle group are there to hunt submarines, supplement the carriers antiaircraft fire, and resupply the carriers.
This is not true. Carriers were powerful, yes, but also vulnerable. I point to the loss of Glorious and Samar as cases where they ended up under the guns of battleships and it didn’t go great, and frankly Samar should have been so much worse than it was. And sinking ships with airplanes is quite difficult. The total number of battleships sunk at sea by carrier planes? Two, and in both cases, it took a lot of planes.
More broadly, the growth of the American carrier fleet was because there was a war on, and any time there was a US-Japan war, the US was going to be building a lot of ships. There were a lot of carriers because carriers had reached the point of being genuinely useful (and could be built reasonably quickly, the main reason the battleship program was curtailed) but it wasn’t like the USN would have had 30+ fleet carriers in 1945 in a world with neither the treaties nor the war.
The intricacies of tradeoffs between WW2 ship classes could be argued, and was argued, for decades in books.
You’re correct that you can create a scenario where the carrier doesn’t always win, and in the confusion of ww2 sensors and communications those scenarios occasionally happened.
You’re correct that aerial weapons at the time were less effective against battleships.
I don’t think these exceptions change the basic idea that the chance of winning the pacific theater fleet battles is proportional to the number and effectiveness of the carrier launched aircraft you can field. So the total combat power of the USN in WW2 is mostly proportional to the carrier number, and the rate of increase is exactly the post overhang example asked for.
Note also overhang does not mean catch up. The timeline with an artificial pause always has less potential progress than the normal timeline.
I doubt that would still hold for the modern world.
The moment one great power detects another is violating such an important, binding, ratified, arms treaty in a way that can’t be explained away as happenstance, they would very quickly launch a hot nuclear war.
Since it would signal that the other country’s leadership are supremely confident in their efforts, either out of lunatic overconfidence and/or because they’re genuinely approaching a breakout capability.
But even more likely, no country would dare to violate the treaty in such a way because they know the consequences would be vastly vastly greater than consequences would have been in the 1930s.
i.e. game theoretic outcomes and schelling points have changed significantly post WW2.
The Soviet Union did violate the Biological Weapons Convention, which seems like an example of “an important, binding, ratified arms treaty.” It did not lead to nuclear war.
It’s very misleading to cite that wikipedia article as an example as the actual text of the BWC only bans substances that are classified as ‘biological and toxin weapons’.
But not substances classified as ‘biodefense’, ‘defensive’, etc., capabilities.
And guess which parties the text assigns to be responsible for making that determination?
Which is the loophole that allows countries to operate ‘biodefense programs’.
i.e. I’m fairly certain the Soviet Union never in fact violated the Convention according to the letter of the law, since all it would have taken to comply was a single piece of paper from the politburo reclassifying their programs to ‘biodefense’ programs.
I’m seeing a lot of examples in this thread of the person arguing how past examples which seemingly apply don’t really apply because of some technicality. Of course the situation doesn’t exactly apply, we aren’t talking about bioweapons in the Soviet era here. The parallels in a hypothetical AI treaty are obvious here.
A key question is has the threat of anything prompted a nuclear exchange?
The answer is no.
Has the threat of even a supposed nuclear exchange from faulty sensors prompted a nuclear exchange?
No.
Nuclear weapons are very expensive to make, very hard to develop, only good at killing, and for all practical purposes pretty useless.
We still failed to several rogue countries from developing them. Of course many countries didn’t build them or are downsizing their stockpiles, but is that primarily because of treaties or because they’re very expensive and practically useless.
Try starting a nuclear exchange over China’s “cancer research” gpu clusters. Wonder how that will go.
Another key question is would an overhang exist. We don’t need to even compare this to jet rexords, we have evidence they exist in deep learning from the history deep learning! Many hardware upgrades lead to researchers quickly beating SoTA by just trying algorithms the hardwares let them. There is also a slower algorithmic overhang, just look at chess algos learning rates vs computes.
Maybe I’m reading between the lines too much, but a lot of these arguments would make sense if someone first decided governance is effective then decided on their positions.
Has there been cold war scenarios where either side should have launched a hot nuclear war ? Does the country launching the nuclear war actually win or do they die also in your scenario?
Do you know of a modern world example of mass suicide attacks? That would be similar to your reference class. Isis? Problem there is the leaders of isis didn’t strap bombs to themselves at the start of the war, which is the case for a nuclear exchange.
Suicide doesn’t have as long of a survival timeline as just letting another superpower win or an AI win. Suicide means the country launching the attack and it’s allies die in 24 hours, a country breaking AI treaties might not win for years. (Implicit assumption: only superpowers with large nuclear arsenals would break the treaty)
So the expected value of launching the nuclear war isn’t as good as bluffing and not carrying out the attack. Right? What do you disagree with on?
Do you dispute that launching a nuclear war against a country with loaded ICBMs will result in those missiles headed for your most valuable assets?
Do you dispute that most of your population will be dead or dying as soon as the missiles hit?
Do you dispute that if you hold fire, and the worst case scenario happens, it will take months to years for an AI to actually kill everyone?
Are you thinking of S risks, where death is preferable to eternal torture?
I think all these claims are incorrect.
First, estimates of damage from nukes are very likely to be hugely (and intentionally) overblown, the same way as pre-WWII estimates for strategic bombing were off by an order of magnitude.
Second, even ignoring that, current arsenals (only warheads deployed on ICBMs, other are irrelevant in this scenario) are not sufficient for counter-population strike. Destroying large cities does not destroy the nation.
Third and most importantly, leadership ordering the first strike will surely survive! They just move to some remote location before, and then claim that the other side attacked first.
Quick fact check: atomic scientists cites this paper which claims 360 million deaths between Russia and the USA. That’s 75% of the current population. This pattern of unacceptable losses goes back, this source says 274 million dead in 1964, the USSR + USA population at that time was 417 million. So 65% in 1964.
You are correct that technically 25% of the population survive, and the prewar leadership could try to have their friends and family hide and they might survive.
Keep in mind that the distribution of losses won’t be even. A powerful nation requires a huge pool of specialists with unique skills that not everyone is trained in. Disproportionately more specialists will be killed, especially engineers and technicians and soldiers and so on. The survivors will likely miss skillsets and obviously all the distribution system to even allow prewar populations to exist is gone, so the survivors will likely be forced to flee to subsist as refugees in neighboring countries.
This outcome is the complete destruction of the military and economic power of the nation—even if everyone isn’t dead, there is going to be essentially no GDP and no means to resist outsiders doing whatever they want. That sounds like national suicide to me, what do you think?
Note also in a scenario of increasing tensions over AI, all the parties would be scaling their nuclear arsenals and preparing measures to continue to fight until the other party is annihilated. This would mean more deployed warheads probably on more forms of delivery vehicle that arms limitation treaties currently restrict. (like stealth cruise missiles)
Again, that some estimates are given in papers doesn’t mean they are even roughly correct. But if they are—then no, that scenario is not suicide. There are some nations now which have lower GDP per capita than USA had two centuries ago.
As for defense—well, that definitely wouldn’t be a problem. Who and why will be willing to invade a big and very poor country, leaders of which claim they still have some nukes in reserve?
Nuclear exchanges won’t end the world, but they will make the nations that started them forever irrelevant. If the top 100 major cities in the U.S. were wiped out, the U.S. would become the next Roman Empire for all sakes and purposes, an echo of the past. That represents an immense loss in GDP and a complete destruction of the economy.
Also you can’t have it both ways,
Either a nuclear exchange is deadly and a reason to abide a treaty (and suicidal enough that leaders won’t actually do it) or it’s not and people won’t abide by it.
The first claim is true—but ruling a third world nation is still better than being dead and ruling nothing. If the leadership has Eliezer-level conviction that AI would kill everybody, then the choice is clear.
The second isn’t—destroying the ability to build AI is much easier, so the reason for abiding the treaty is not “we all die” but rather “we become much poorer and don’t get the AI anyway”.
I was trying to establish where we disagree and if the disagreement can be resolved by citing facts.
Could you help me by showing in quotes where you believe I “rambled”. I see each argument building on the next, can you show me how it rambles?
Here’s an outline:
Nuclear war is suicide
Suicide for national leaders hasn’t been practiced as a warfare strategy
Expected value of nuclear war is worse than dying to AI
Do you believe a nuclear war is suicide?
Are you thinking of S risks?
Update: I thought of a new one. Are you thinking government leaders will kill their nation and themselves altruistically? Since if they die, and the superpower building ai dies, other countries live and just have to deal with nuclear winter and fallout which is probably survivable. While the AI might kill everyone.
The game theoretic outcome, in the unlikely case where one nation is hellbent on achieving a breakout of a major arms treaty in modern conditions with thousands of WMDs stockpiled, is not something you even touched on in the parent.
I’m not sure how adding on to a historical parent example and extrapolating to the world of 2023 counts as ‘disagreement’.
If you disagree with the extrapolation in the child comment then you need to write down why before asking further questions.
Hence why asking dozens of questions right off the bat, when it’s unclear what you were disagreeing about, makes no sense.
So today we have Russia violating international law openly, China running concentration camps, and the US contributing to climate change without any serious intent to stop. Oh and the West seems to be in a proxy war with Russia and Iran with Israel.
This doesn’t look like a world of multiple parties afraid to offend the other, knowing it will be nuclear war if they don’t.
Can you point to examples that support your general idea that the world is closer to a world government and moving away from sovereign powers that compete and do what they want? I am not a geopolitical expert, I picked history because it seems to show a stable long term trend over decades. Or succinctly, “the outcome was X over N years, why would it change to Y?”
My disagreement is that I think history will repeat and nuclear threats will be dismissed. I think any AI treaties will be worked around like in the past and it will be a competition until the end.
I think the current events show that is what is happening today, with increasing competition that will lead to AI races.
This doesn’t look like a world of multiple parties afraid to offend the other, knowing it will be nuclear war if they don’t.
How does ‘look like’ relate to the actual views and decision making of the leadership of any country?
People can talk one way and walk the other way. And you don’t exactly seem like someone who has the inside scoop in major world capitals so even the talk you’ve heard is likely several steps removed.
So to summarize, you believe that the parties have or will reach secret international agreements with each other and will not compete over AI like in past international arms races? Has something changed in the world to make this happen now but not then? Like better communications tech?
Does any published evidence exist to support your belief or is it all secret?
So to summarize, you believe that the parties have or will reach secret international agreements with each other and will not compete over AI like in past international arms races?
What is being ‘summarized’?
I don’t think I’ve expressed such a belief in the previous comments?
This is the belief I gathered from your most recent reply, where you said:
How does ‘look like’ relate to the actual views and decision making of the leadership of any country?
People can talk one way and walk the other way. And you don’t exactly seem like someone who has the inside scoop in major world capitals so even the talk you’ve heard is likely several steps removed.
Ergo I believe you are saying I cannot use historical evidence, since it is 2023, and I can’t use “the superpowers are openly committing violations of international law and Russia and the USA are in a proxy war” as evidence since I don’t know what world leaders say to each other privately.
I’m not seeking out a further dialogue, just the actual arguments/reasoning/etc. behind your first reply, so that I can evaluate whether any of this has a logical basis.
Frankly, as of right now there’s almost nothing for me to base a decision upon since it’s been a pretty one-sided back and forth.
The “actual argument” is that MAD has existed since the 1960s, so about 65 years, and the United States/Soviet Union/China have done whatever they want except invade each other. Including many things another side hates. “Don’t do X or I will nuke” has not been an effective threat.
The most recent round of this was 2022, when Russia has threatened to use nuclear weapons many times and it has not been an effective threat.
On 21 September, while announcing a partial mobilization of conscripts, Putin said that Russia “will use all the means at our disposal” – widely interpreted as a threat to use nuclear weapons – in order to defend the country’s territory.[10] He warned that his threat was “not a bluff”, baselessly accused NATO of “nuclear blackmail” and of threatening to use nuclear weapons against Russia, and said Russia’s nuclear weapons were more advanced than NATO’s.[11][12] Russian Foreign Minister Sergey Lavrov did not rule out the use of nuclear weapons to defend annexed Ukrainian territories.[13] Several days later, former Russian president and Putin ally Dmitry Medvedev made a more explicit threat of a nuclear strike against Ukraine.[14]
I feel like we’re going in circles, so I will just lay it out step by step and you can determine where your getting stuck:
What leaders talk about in public is not identical to what they talk about in private
and even that is not identical to their actual views.
And even their actual views are not identical with their real world decisions. Because they are constrained by hundreds or thousands of actors in a complex system.
Unless you have some super special knowledge you can’t even complete link 1 of this chain.
Therefore this line of argumentation cannot be developed, limiting anyone to just guessing the inferential gaps.
We can of course make an unlimited number of guesses, but since we can’t outweigh each other, this line of discussion wouldn’t be able to lead anywhere.
I doubt that would still hold for the modern world.
The moment one great power detects another is violating such an important, binding, ratified, arms treaty in a way that can’t be explained away as happenstance, they would very quickly launch a hot nuclear war.
I don’t see where the private communications is relevant here to your claim. Your claim is brittle in that only one example is needed to falsify it, which the above example seems to do.
I realized it was pointless to argue if a public nuclear threat was ‘real’ or empty, since no one can know that until 20+ years from now, and then you would argue that it’s ancient history. So i went back to the root argument and falsified it. In some of my prior edits I have been trying to explain you’ve made a really fragile claim fundamentally. That the whole idea of needing “equal evidence” is only true if both claims are equally probable.
For example, “coin will come up A” and “coin will resolve B” have near equal probability. “UFOs are real” and “UFOs are not real” do not.
Given a prior of “no nuclear war for 65 years”, “a nuclear war will start on the first treaty violation” is similar to the UFO claim in that it is not probable.
So i went back to the root argument and falsified it, which ends the discussion.
What ‘root argument’? And to end the discussion, just leaving this deep comment chain suffices, no need for theatrics.
The first few times I could overlook some oddities, but now my patience is exhausted so I’m putting my foot down and insist you not jump around all over the place.
It at best appears like a bizarre thinking process and at worst as someone side-stepping whatever is inconvenient to them.
The past few especially seem to me like dodging the actual substance.
Historically have you found an example of a worldwide pause for a strategically relevant technology? Something negotiated by agreement that wasn’t immediately defected on?
Skimming https://en.wikipedia.org/wiki/Arms_control I see a couple interesting tidbits: the Washington Naval Conference did result in “an effective end to building new battleship fleets, and the few ships that were built were limited in size and armament”. This seems to fit your reference class!
And then after the Treaty ended in 1936 (with defections throughout), we get:
BB is battleship, there’s only 10 new ones, but 141! aircraft carriers. In tonnage this would fit the reference class of “after the pause, more ships than ever”. Also, the USN had discovered the carrier hard countered all other surface combatants. The AI equivalent might be post pause few new GPUs are built, it’s all neural accelerator ASICs.
Another view, close to 10x the scale in 4 years. This looks like a fit for your reference class.
It’s a tight fit another way. What you would expect in a timeline where all the world powers do agree to some kind of AI limitations. During that time period, similar to the post washington navy treaty, you might get an equivalent to:
The United States developed better technology to get better performance from their ships while still working within the weight limits, the United Kingdom exploited a loop-hole in the terms, the Italians misrepresented the weight of their vessels, and when up against the limits, Japan left the treaty.
You can imagine this happening many ways for AI, where each party to the pause finds a different route to evade it.
Eventually what would happen is breakout, where one of the superpowers reaches the capability level for exponential growth or achieves some other advantage that threatens the others. The other parties to the pause must unpause or die, and you would see something like the above. Looks like humans could double their warships in existence 3 times in ~4 years, presumably robots constructing robots and weapons will be faster.
I have now looked into this example, and talked to Bean at Naval Gazing about it.
I found data for the total tonnage in many countries’ navies from 1865-2011. It seems to show overhang for the US navy during the interwar years, and maybe also for the Italian navy, but not for any of the other great powers.
Bean has convinced me that this data is not to be trusted. It does not distinguish between warships and auxiliary ships, or between ships in active duty and in reserve. It has some straightforward errors in the year the last battleship was decommissioned and the first nuclear submarine was commissioned. It seems to be off by a factor of 10 at some points. There seems to not be a dataset covering this time period that is even accurate to within a factor of 2 for me to do this analysis with.
Even if the data were reliable, it’s not clear how relevant they would be. It seems as though the total tonnage in the navy of an industrialized country is mostly determined by the international situation, rather than anything about the technology itself. If the trend is going up, that mostly means that the country is spending a larger fraction of its GDP on the navy.
I don’t think that this changes the conclusions of the post much. Overhang is uncommon. It happened partially for land speed records, but it was not sustained and this is not a technology that people put much effort into. Something that looks sort of similar might have happened with the US navy during the interwar period, but not for the navies of any of the other great powers, and the underlying dynamics seem to be different.
What are we trying to model here or find examples of?
Here’s what I think we’re trying to model: if a technology were isolated and for whatever reason, development was stopped, then during the ‘stopped’ period very little effort is being put into it.
After the ‘stopped’ period ends, development resumes and presumably progress is proportional to effort, with an unavoidable serial part of the process (from Amdahl’s law/ Gant charts show this) restricting the rate that progress could be made at.
For US Navy tonnage : without a washington Naval treaty, and a Great Depression and a policy of isolation, the US Navy would presumably have built warships at a steady rate. They did not, as shown in your data.
However, during this prewar period, other processes continued. Multiple countries continuously improved aircraft designs, with better aerodynamics (biplane to mono), carrier launching and landing, ever larger and more powerful engines, dive and torpedo bombing, and other innovations.
So even though very few ships are being built, aircraft are being improved. Now Pearl harbor, and unpause. All out effort, which shows in the data you linked.
But we don’t have to trust it, all that really matters is the aircraft carrier numbers, nothing else. As it turned out, the carrier was a hard counter to everything, even other carriers—the other ships in a carrier battle group are there to hunt submarines, supplement the carriers antiaircraft fire, and resupply the carriers. While there were direct gun battles in late ww2 in the Pacific theater, better admirals could probably have avoided every battle and just sank all the enemy ships with aircraft. Shooting down enemy aircraft was also way easier, it turned out, to do with aircraft.
So only the left column matters for the model, and you also need the 0 point. There were seven fleet aircraft carriers and one escort carrier at t=0, beginning of ww2.
If we count the escort carriers at 30% of a fleet carrier, and there were 27 new fleet carriers, then the total of “weighted” carriers went from 7.3 (prewar) to 61.2 (weighted, 1945).
8.38x increase in the most relevant parameter.
The data you don’t trust, 1940 tonnage is 1956867 and 1945 tonnage is 11267550. Increase of 5.75x.
In terms of “relevant” tonnage obviously aircraft carriers, due to their role as a hard counter with 1940s era technology, is all that matters.
From the perspective of Japanese admirals, over 4 years of war, they faced 8 times the warships produced during the entire prewar period. This is the issue, their battle plans could not scale to handle this. Had the Americans been building carriers at a steadier rate the entire time, the Japanese would never have attacked. The rate of increase turned out to be a strategic surprise that doomed the Japanese side of the war.
Predictions for AI:
It’s frankly hard to imagine a world where an AI pause is actually negotiated. Remember, if any major party says no, nobody can afford to pause. Say it happens:
During the pause, there would be some flops limit on model scale, above which requires likely prohibitively expensive precautions.
During the pause period, people would be experimenting with below threshold ML algorithms. They might build app stores where sub AGI algorithms can be easily licensed and combined into integrated systems, where “ML microservices” give an integrated system some of the benefits of a real AGI. Probably a network* of ML microservices could be built that can perceive, interpret the environment, compare the current state to a goal state, consider many possible actions using an environment/robotics sim that is a service, and then choose the best action. It might not even be that much less effective than a “real” AGI model that is integrated for some purposes like factory work.
The app stores would also likely contain key innovations that current AI stacks are missing. A series of software intercompatibility frameworks (anyone can drive a robot with any model), cloud hosted realistic evaluation environments (these are things like realistic robotic environment sims), composable robotics backends, formally proven stack components (so robotics can be certified as life safety using new AI models), and likely cloud hosted AI improving AI services. (this is where it is possible to pay for improvements to an AI stack using a service that is cloud hosted, this is possible even with sub AGI RL models).
Note that the things I just mentioned do not exist today, everyone on autonomous cars has rolled their own duplicate version of it, with only a little bit of sharing with platforms like ROS.
During the pause period, Moore’s law continues to whatever level it is able to continue (physics is obviously an ultimate limit to an extent), but the experience curve would continue to make transistors cheaper even after they cannot be made smaller. AI models are close to embarrassingly parallel so they benefit linearly from the number of transistors.
During the pause period, ASIC architectures for AI chips are developed. On previous examples like bitcoin mining and others, ASICs have proven to be substantially faster and more powerful efficient for the same quantity of transistors.
So after the pause period, all these innovations hit all at once, and the rate of AI increase becomes very high, potentially uncontrollable. You can see why: all the obstacles to letting true AGI models immediately control robots have been smoothed away, AGI models can be used to improve each other even if a different company owns the model (that cloud hoisted AI improving service), and the realistic evaluation environments allow the new AGI models to quickly be evaluated and improved on. The compute is also faster, and the pause was likely dropped due to a war, so the new AI effort is probably flat out.
So I’m convinced. With a gears level model I think the overhang idea has a very high probability of happening if a pause occurred. It’s just super rare for there to be any pause at all historically, and there probably will not be one for AI.
Oh one bit of confusion: say there was a 5 year pause between 2030 and 2035. Am I saying that in 2040, is the amount of AI progress the same as in a 2040 with no pause? No. Obviously the pause slowed down progress, the fastest progress is when people are working on AI at full speed from 2023 to 2040. But it does create a discontinuity : 2035 after the pause is a more interesting year than the “no pause” 2030-2035 period.
*the reason a network of separate models isn’t the same as an AGI is each model gets trained separately, and it would be illegal to train the models on information from when they were assembled into a system. Each model has to remain static, validated on separate benches. And the models must operate in a human interpretable way, humans would have to be able to clearly understand both the input and output. This is how current autonomous car stacks, except comma.ai’s, already work. Assuming each model can be up to 10^26 flops—that’s quite a bit of headroom, probably easily enough to do a lot of “AGI” like things using a network of 10-100 subsystems, each under 10^26. The human brain works this way.
>But we don’t have to trust it, all that really matters is the aircraft carrier numbers, nothing else. As it turned out, the carrier was a hard counter to everything, even other carriers—the other ships in a carrier battle group are there to hunt submarines, supplement the carriers antiaircraft fire, and resupply the carriers.
This is not true. Carriers were powerful, yes, but also vulnerable. I point to the loss of Glorious and Samar as cases where they ended up under the guns of battleships and it didn’t go great, and frankly Samar should have been so much worse than it was. And sinking ships with airplanes is quite difficult. The total number of battleships sunk at sea by carrier planes? Two, and in both cases, it took a lot of planes.
More broadly, the growth of the American carrier fleet was because there was a war on, and any time there was a US-Japan war, the US was going to be building a lot of ships. There were a lot of carriers because carriers had reached the point of being genuinely useful (and could be built reasonably quickly, the main reason the battleship program was curtailed) but it wasn’t like the USN would have had 30+ fleet carriers in 1945 in a world with neither the treaties nor the war.
The intricacies of tradeoffs between WW2 ship classes could be argued, and was argued, for decades in books.
You’re correct that you can create a scenario where the carrier doesn’t always win, and in the confusion of ww2 sensors and communications those scenarios occasionally happened.
You’re correct that aerial weapons at the time were less effective against battleships.
I don’t think these exceptions change the basic idea that the chance of winning the pacific theater fleet battles is proportional to the number and effectiveness of the carrier launched aircraft you can field. So the total combat power of the USN in WW2 is mostly proportional to the carrier number, and the rate of increase is exactly the post overhang example asked for.
Note also overhang does not mean catch up. The timeline with an artificial pause always has less potential progress than the normal timeline.
I did not look at the Washington Naval Conference as a potential example. It seems like it might be relevant. Thank you !
I doubt that would still hold for the modern world.
The moment one great power detects another is violating such an important, binding, ratified, arms treaty in a way that can’t be explained away as happenstance, they would very quickly launch a hot nuclear war.
Since it would signal that the other country’s leadership are supremely confident in their efforts, either out of lunatic overconfidence and/or because they’re genuinely approaching a breakout capability.
But even more likely, no country would dare to violate the treaty in such a way because they know the consequences would be vastly vastly greater than consequences would have been in the 1930s.
i.e. game theoretic outcomes and schelling points have changed significantly post WW2.
The Soviet Union did violate the Biological Weapons Convention, which seems like an example of “an important, binding, ratified arms treaty.” It did not lead to nuclear war.
It’s very misleading to cite that wikipedia article as an example as the actual text of the BWC only bans substances that are classified as ‘biological and toxin weapons’.
But not substances classified as ‘biodefense’, ‘defensive’, etc., capabilities.
And guess which parties the text assigns to be responsible for making that determination?
Which is the loophole that allows countries to operate ‘biodefense programs’.
i.e. I’m fairly certain the Soviet Union never in fact violated the Convention according to the letter of the law, since all it would have taken to comply was a single piece of paper from the politburo reclassifying their programs to ‘biodefense’ programs.
I’m seeing a lot of examples in this thread of the person arguing how past examples which seemingly apply don’t really apply because of some technicality. Of course the situation doesn’t exactly apply, we aren’t talking about bioweapons in the Soviet era here. The parallels in a hypothetical AI treaty are obvious here.
A key question is has the threat of anything prompted a nuclear exchange?
The answer is no.
Has the threat of even a supposed nuclear exchange from faulty sensors prompted a nuclear exchange?
No.
Nuclear weapons are very expensive to make, very hard to develop, only good at killing, and for all practical purposes pretty useless.
We still failed to several rogue countries from developing them. Of course many countries didn’t build them or are downsizing their stockpiles, but is that primarily because of treaties or because they’re very expensive and practically useless.
Try starting a nuclear exchange over China’s “cancer research” gpu clusters. Wonder how that will go.
Another key question is would an overhang exist. We don’t need to even compare this to jet rexords, we have evidence they exist in deep learning from the history deep learning! Many hardware upgrades lead to researchers quickly beating SoTA by just trying algorithms the hardwares let them. There is also a slower algorithmic overhang, just look at chess algos learning rates vs computes.
Maybe I’m reading between the lines too much, but a lot of these arguments would make sense if someone first decided governance is effective then decided on their positions.
How are these points relevant to what was being discussed?
I don’t think the opposing stance for these two question was implied.
The history of (or lack of) nuclear exchanges seems relevant to discussing whether they will be effective at enforcing treaties.
What specifically do these questions relate to in the preceding comment, the one it’s replying to? Or to Jeffrey’s comment before that?
Has there been cold war scenarios where either side should have launched a hot nuclear war ? Does the country launching the nuclear war actually win or do they die also in your scenario?
Do you know of a modern world example of mass suicide attacks? That would be similar to your reference class. Isis? Problem there is the leaders of isis didn’t strap bombs to themselves at the start of the war, which is the case for a nuclear exchange.
Suicide doesn’t have as long of a survival timeline as just letting another superpower win or an AI win. Suicide means the country launching the attack and it’s allies die in 24 hours, a country breaking AI treaties might not win for years. (Implicit assumption: only superpowers with large nuclear arsenals would break the treaty)
So the expected value of launching the nuclear war isn’t as good as bluffing and not carrying out the attack. Right? What do you disagree with on?
Do you dispute that launching a nuclear war against a country with loaded ICBMs will result in those missiles headed for your most valuable assets?
Do you dispute that most of your population will be dead or dying as soon as the missiles hit?
Do you dispute that if you hold fire, and the worst case scenario happens, it will take months to years for an AI to actually kill everyone?
Are you thinking of S risks, where death is preferable to eternal torture?
I think all these claims are incorrect. First, estimates of damage from nukes are very likely to be hugely (and intentionally) overblown, the same way as pre-WWII estimates for strategic bombing were off by an order of magnitude. Second, even ignoring that, current arsenals (only warheads deployed on ICBMs, other are irrelevant in this scenario) are not sufficient for counter-population strike. Destroying large cities does not destroy the nation. Third and most importantly, leadership ordering the first strike will surely survive! They just move to some remote location before, and then claim that the other side attacked first.
Quick fact check: atomic scientists cites this paper which claims 360 million deaths between Russia and the USA. That’s 75% of the current population. This pattern of unacceptable losses goes back, this source says 274 million dead in 1964, the USSR + USA population at that time was 417 million. So 65% in 1964.
You are correct that technically 25% of the population survive, and the prewar leadership could try to have their friends and family hide and they might survive.
Keep in mind that the distribution of losses won’t be even. A powerful nation requires a huge pool of specialists with unique skills that not everyone is trained in. Disproportionately more specialists will be killed, especially engineers and technicians and soldiers and so on. The survivors will likely miss skillsets and obviously all the distribution system to even allow prewar populations to exist is gone, so the survivors will likely be forced to flee to subsist as refugees in neighboring countries.
This outcome is the complete destruction of the military and economic power of the nation—even if everyone isn’t dead, there is going to be essentially no GDP and no means to resist outsiders doing whatever they want. That sounds like national suicide to me, what do you think?
Note also in a scenario of increasing tensions over AI, all the parties would be scaling their nuclear arsenals and preparing measures to continue to fight until the other party is annihilated. This would mean more deployed warheads probably on more forms of delivery vehicle that arms limitation treaties currently restrict. (like stealth cruise missiles)
Again, that some estimates are given in papers doesn’t mean they are even roughly correct. But if they are—then no, that scenario is not suicide. There are some nations now which have lower GDP per capita than USA had two centuries ago.
As for defense—well, that definitely wouldn’t be a problem. Who and why will be willing to invade a big and very poor country, leaders of which claim they still have some nukes in reserve?
Nuclear exchanges won’t end the world, but they will make the nations that started them forever irrelevant. If the top 100 major cities in the U.S. were wiped out, the U.S. would become the next Roman Empire for all sakes and purposes, an echo of the past. That represents an immense loss in GDP and a complete destruction of the economy.
Also you can’t have it both ways,
Either a nuclear exchange is deadly and a reason to abide a treaty (and suicidal enough that leaders won’t actually do it) or it’s not and people won’t abide by it.
The first claim is true—but ruling a third world nation is still better than being dead and ruling nothing. If the leadership has Eliezer-level conviction that AI would kill everybody, then the choice is clear. The second isn’t—destroying the ability to build AI is much easier, so the reason for abiding the treaty is not “we all die” but rather “we become much poorer and don’t get the AI anyway”.
Did you skip over reading some part of the comment? Or do some parts seem confusing?
If you read and understood all of it, then writing down dozens of rambling questions doesn’t seem to make sense.
A suicidal nuclear war seems highly improbable.
I was trying to establish where we disagree and if the disagreement can be resolved by citing facts.
Could you help me by showing in quotes where you believe I “rambled”. I see each argument building on the next, can you show me how it rambles?
Here’s an outline:
Nuclear war is suicide
Suicide for national leaders hasn’t been practiced as a warfare strategy
Expected value of nuclear war is worse than dying to AI
Do you believe a nuclear war is suicide?
Are you thinking of S risks?
Update: I thought of a new one. Are you thinking government leaders will kill their nation and themselves altruistically? Since if they die, and the superpower building ai dies, other countries live and just have to deal with nuclear winter and fallout which is probably survivable. While the AI might kill everyone.
I don’t disagree?
The game theoretic outcome, in the unlikely case where one nation is hellbent on achieving a breakout of a major arms treaty in modern conditions with thousands of WMDs stockpiled, is not something you even touched on in the parent.
I’m not sure how adding on to a historical parent example and extrapolating to the world of 2023 counts as ‘disagreement’.
If you disagree with the extrapolation in the child comment then you need to write down why before asking further questions.
Hence why asking dozens of questions right off the bat, when it’s unclear what you were disagreeing about, makes no sense.
So today we have Russia violating international law openly, China running concentration camps, and the US contributing to climate change without any serious intent to stop. Oh and the West seems to be in a proxy war with Russia and Iran with Israel. This doesn’t look like a world of multiple parties afraid to offend the other, knowing it will be nuclear war if they don’t.
Can you point to examples that support your general idea that the world is closer to a world government and moving away from sovereign powers that compete and do what they want? I am not a geopolitical expert, I picked history because it seems to show a stable long term trend over decades. Or succinctly, “the outcome was X over N years, why would it change to Y?”
My disagreement is that I think history will repeat and nuclear threats will be dismissed. I think any AI treaties will be worked around like in the past and it will be a competition until the end.
I think the current events show that is what is happening today, with increasing competition that will lead to AI races.
How does ‘look like’ relate to the actual views and decision making of the leadership of any country?
People can talk one way and walk the other way. And you don’t exactly seem like someone who has the inside scoop in major world capitals so even the talk you’ve heard is likely several steps removed.
So to summarize, you believe that the parties have or will reach secret international agreements with each other and will not compete over AI like in past international arms races? Has something changed in the world to make this happen now but not then? Like better communications tech?
Does any published evidence exist to support your belief or is it all secret?
What is being ‘summarized’?
I don’t think I’ve expressed such a belief in the previous comments?
This is the belief I gathered from your most recent reply, where you said:
Ergo I believe you are saying I cannot use historical evidence, since it is 2023, and I can’t use “the superpowers are openly committing violations of international law and Russia and the USA are in a proxy war” as evidence since I don’t know what world leaders say to each other privately.
Ergo, based on what reasoning?
Can you actually lay out the argument, step by step, in writing?
I would be open to a dialogue if you want to discuss this further.
I’m not seeking out a further dialogue, just the actual arguments/reasoning/etc. behind your first reply, so that I can evaluate whether any of this has a logical basis.
Frankly, as of right now there’s almost nothing for me to base a decision upon since it’s been a pretty one-sided back and forth.
The “actual argument” is that MAD has existed since the 1960s, so about 65 years, and the United States/Soviet Union/China have done whatever they want except invade each other. Including many things another side hates. “Don’t do X or I will nuke” has not been an effective threat.
The most recent round of this was 2022, when Russia has threatened to use nuclear weapons many times and it has not been an effective threat.
https://en.m.wikipedia.org/wiki/Nuclear_risk_during_the_Russian_invasion_of_Ukraine
A threat over AI is another empty threat for game theoretic reasons which you can read up thread.
How do you even know that has been threatened, by the actual leadership of any country, for whatever you believe ‘X’ to include?
To be entirely clear and direct, you are still not writing down actual arguments, just your opinions and guesses on world affairs.
From the linked article:
I feel like we’re going in circles, so I will just lay it out step by step and you can determine where your getting stuck:
What leaders talk about in public is not identical to what they talk about in private
and even that is not identical to their actual views.
And even their actual views are not identical with their real world decisions. Because they are constrained by hundreds or thousands of actors in a complex system.
Unless you have some super special knowledge you can’t even complete link 1 of this chain.
Therefore this line of argumentation cannot be developed, limiting anyone to just guessing the inferential gaps.
We can of course make an unlimited number of guesses, but since we can’t outweigh each other, this line of discussion wouldn’t be able to lead anywhere.
You said :
https://www.state.gov/u-s-countermeasures-in-response-to-russias-violations-of-the-new-start-treaty/
I don’t see where the private communications is relevant here to your claim. Your claim is brittle in that only one example is needed to falsify it, which the above example seems to do.
Huh?
This comment is showing, on my screen, as a reply to my comment here.
Which was a response to your comment here.
This is worded as if it was intended as a reply for something else. What relation does it have to the previous two comments?
I realized it was pointless to argue if a public nuclear threat was ‘real’ or empty, since no one can know that until 20+ years from now, and then you would argue that it’s ancient history. So i went back to the root argument and falsified it. In some of my prior edits I have been trying to explain you’ve made a really fragile claim fundamentally. That the whole idea of needing “equal evidence” is only true if both claims are equally probable.
For example, “coin will come up A” and “coin will resolve B” have near equal probability. “UFOs are real” and “UFOs are not real” do not.
Given a prior of “no nuclear war for 65 years”, “a nuclear war will start on the first treaty violation” is similar to the UFO claim in that it is not probable.
What ‘root argument’? And to end the discussion, just leaving this deep comment chain suffices, no need for theatrics.
The first few times I could overlook some oddities, but now my patience is exhausted so I’m putting my foot down and insist you not jump around all over the place.
It at best appears like a bizarre thinking process and at worst as someone side-stepping whatever is inconvenient to them.
The past few especially seem to me like dodging the actual substance.