Conditioning does not necessarily follow time ordering. E.g., you can condition the odds of X on being in a world on track to develop robots by 2043 without having robots well in advance of X. Similarly, we can condition on a world where transformative AGI is trainable with 1e30 floating point operations then ask the likelihood that 1e30 floating point operations can be constructed and harnessed for TAGI. Remember too that in a world with rapidly advancing AI and robots, much of the demand will be for things other than TAGI.
I’m sympathetic to your point that it’s hard for brains to forecast these conditional probabilities. Certainly we may be wrong. But on the other hand, it’s also hard for brains to forecast things that involve smushing lots of probabilities together under the hood. I generally think that factoring things out into components helps, but I can understand if you disagree.
Here is the world I am most interested in, where the conditional probability seems least plausible:
We invent algorithms for transformative AGI
We invent a way for AGIs to learn faster than humans
AGI inference costs drop below $25/hr
We invent and scale cheap, quality robots
We massively scale production of chips and power
We avoid derailment by human regulation
We avoid derailment by AI-caused delay
In this world, what is the probability that we were “derailed” by wars, such as China invading Taiwan?
Reading the paper naively, it says that there is a 30% chance that we achieved all of this technical progress, in the 99th percentile of possible outcomes, despite China invading Taiwan. That doesn’t seem like a 30% chance to me. Additionally, if China invaded Taiwan, but it didn’t prevent us achieving all this technical progress, in what sense was it a derailment? The executive summary suggests:
Even if an invasion doesn’t spark war, the sanctions applied in retaliation will shut down most of the world’s AI chip production
No, it can’t possibly derail AI by shutting down chip production, in this conditional branch, because we already know from item 5 that we massively scaled chip production, and both things can’t be true at the same time.
Right. The idea is: “What are the odds that China invading Taiwan derails chip production conditional on a world where we were otherwise going to successfully scale chip production.”
I would not have guessed that! So in slightly more formal terms:
CHIPS = There are enough chips for TAGI by 2043
WAR = There is a war that catastrophically derails chip production by 2043
P(x) = subjective probability of x
ObjP(x) = objective probability of x
P(CHIPS and WAR) = 0% (by definition)
Then as I understand your method, it goes something like:
Estimate P(CHIPS given not WAR) = 46%
This means that in 46% of worlds, ObjP(CHIPS given not WAR) = 100%. Call these worlds CHIPPY worlds. In all other worlds ObjP(CHIPS given not WAR) = 0%.
Estimate P(not WAR given CHIPPY) = 70%.
The only option for CHIPS is “not WAR and CHIPPY”.
Calculate P(not WAR and CHIPPY) = 70% x 46% = 32.2%.
Therefore P(CHIPS) = 32.2%.
(probabilities may differ, this is just illustrative)
However, I don’t think the world is deterministic enough for step 2 to work—the objective probability could be 50% or some other value.
Conditioning does not necessarily follow time ordering. E.g., you can condition the odds of X on being in a world on track to develop robots by 2043 without having robots well in advance of X. Similarly, we can condition on a world where transformative AGI is trainable with 1e30 floating point operations then ask the likelihood that 1e30 floating point operations can be constructed and harnessed for TAGI. Remember too that in a world with rapidly advancing AI and robots, much of the demand will be for things other than TAGI.
I’m sympathetic to your point that it’s hard for brains to forecast these conditional probabilities. Certainly we may be wrong. But on the other hand, it’s also hard for brains to forecast things that involve smushing lots of probabilities together under the hood. I generally think that factoring things out into components helps, but I can understand if you disagree.
Here is the world I am most interested in, where the conditional probability seems least plausible:
We invent algorithms for transformative AGI
We invent a way for AGIs to learn faster than humans
AGI inference costs drop below $25/hr
We invent and scale cheap, quality robots
We massively scale production of chips and power
We avoid derailment by human regulation
We avoid derailment by AI-caused delay
In this world, what is the probability that we were “derailed” by wars, such as China invading Taiwan?
Reading the paper naively, it says that there is a 30% chance that we achieved all of this technical progress, in the 99th percentile of possible outcomes, despite China invading Taiwan. That doesn’t seem like a 30% chance to me. Additionally, if China invaded Taiwan, but it didn’t prevent us achieving all this technical progress, in what sense was it a derailment? The executive summary suggests:
No, it can’t possibly derail AI by shutting down chip production, in this conditional branch, because we already know from item 5 that we massively scaled chip production, and both things can’t be true at the same time.
Right. The idea is: “What are the odds that China invading Taiwan derails chip production conditional on a world where we were otherwise going to successfully scale chip production.”
I would not have guessed that! So in slightly more formal terms:
CHIPS = There are enough chips for TAGI by 2043
WAR = There is a war that catastrophically derails chip production by 2043
P(x) = subjective probability of x
ObjP(x) = objective probability of x
P(CHIPS and WAR) = 0% (by definition)
Then as I understand your method, it goes something like:
Estimate P(CHIPS given not WAR) = 46%
This means that in 46% of worlds, ObjP(CHIPS given not WAR) = 100%. Call these worlds CHIPPY worlds. In all other worlds ObjP(CHIPS given not WAR) = 0%.
Estimate P(not WAR given CHIPPY) = 70%.
The only option for CHIPS is “not WAR and CHIPPY”.
Calculate P(not WAR and CHIPPY) = 70% x 46% = 32.2%.
Therefore P(CHIPS) = 32.2%.
(probabilities may differ, this is just illustrative)
However, I don’t think the world is deterministic enough for step 2 to work—the objective probability could be 50% or some other value.
Bingo