Here is the world I am most interested in, where the conditional probability seems least plausible:
We invent algorithms for transformative AGI
We invent a way for AGIs to learn faster than humans
AGI inference costs drop below $25/hr
We invent and scale cheap, quality robots
We massively scale production of chips and power
We avoid derailment by human regulation
We avoid derailment by AI-caused delay
In this world, what is the probability that we were “derailed” by wars, such as China invading Taiwan?
Reading the paper naively, it says that there is a 30% chance that we achieved all of this technical progress, in the 99th percentile of possible outcomes, despite China invading Taiwan. That doesn’t seem like a 30% chance to me. Additionally, if China invaded Taiwan, but it didn’t prevent us achieving all this technical progress, in what sense was it a derailment? The executive summary suggests:
Even if an invasion doesn’t spark war, the sanctions applied in retaliation will shut down most of the world’s AI chip production
No, it can’t possibly derail AI by shutting down chip production, in this conditional branch, because we already know from item 5 that we massively scaled chip production, and both things can’t be true at the same time.
Right. The idea is: “What are the odds that China invading Taiwan derails chip production conditional on a world where we were otherwise going to successfully scale chip production.”
I would not have guessed that! So in slightly more formal terms:
CHIPS = There are enough chips for TAGI by 2043
WAR = There is a war that catastrophically derails chip production by 2043
P(x) = subjective probability of x
ObjP(x) = objective probability of x
P(CHIPS and WAR) = 0% (by definition)
Then as I understand your method, it goes something like:
Estimate P(CHIPS given not WAR) = 46%
This means that in 46% of worlds, ObjP(CHIPS given not WAR) = 100%. Call these worlds CHIPPY worlds. In all other worlds ObjP(CHIPS given not WAR) = 0%.
Estimate P(not WAR given CHIPPY) = 70%.
The only option for CHIPS is “not WAR and CHIPPY”.
Calculate P(not WAR and CHIPPY) = 70% x 46% = 32.2%.
Therefore P(CHIPS) = 32.2%.
(probabilities may differ, this is just illustrative)
However, I don’t think the world is deterministic enough for step 2 to work—the objective probability could be 50% or some other value.
Here is the world I am most interested in, where the conditional probability seems least plausible:
We invent algorithms for transformative AGI
We invent a way for AGIs to learn faster than humans
AGI inference costs drop below $25/hr
We invent and scale cheap, quality robots
We massively scale production of chips and power
We avoid derailment by human regulation
We avoid derailment by AI-caused delay
In this world, what is the probability that we were “derailed” by wars, such as China invading Taiwan?
Reading the paper naively, it says that there is a 30% chance that we achieved all of this technical progress, in the 99th percentile of possible outcomes, despite China invading Taiwan. That doesn’t seem like a 30% chance to me. Additionally, if China invaded Taiwan, but it didn’t prevent us achieving all this technical progress, in what sense was it a derailment? The executive summary suggests:
No, it can’t possibly derail AI by shutting down chip production, in this conditional branch, because we already know from item 5 that we massively scaled chip production, and both things can’t be true at the same time.
Right. The idea is: “What are the odds that China invading Taiwan derails chip production conditional on a world where we were otherwise going to successfully scale chip production.”
I would not have guessed that! So in slightly more formal terms:
CHIPS = There are enough chips for TAGI by 2043
WAR = There is a war that catastrophically derails chip production by 2043
P(x) = subjective probability of x
ObjP(x) = objective probability of x
P(CHIPS and WAR) = 0% (by definition)
Then as I understand your method, it goes something like:
Estimate P(CHIPS given not WAR) = 46%
This means that in 46% of worlds, ObjP(CHIPS given not WAR) = 100%. Call these worlds CHIPPY worlds. In all other worlds ObjP(CHIPS given not WAR) = 0%.
Estimate P(not WAR given CHIPPY) = 70%.
The only option for CHIPS is “not WAR and CHIPPY”.
Calculate P(not WAR and CHIPPY) = 70% x 46% = 32.2%.
Therefore P(CHIPS) = 32.2%.
(probabilities may differ, this is just illustrative)
However, I don’t think the world is deterministic enough for step 2 to work—the objective probability could be 50% or some other value.
Bingo