Perhaps there was something I misunderstood, but wouldn’t AI alignment work and AI capabilities slowdown still have extreme positive expected value even if the probability of unaligned AI is only 0.1-10%?
Let’s say the universe will exist for 15 billion more years until the big rip.
Let’s say we could decrease the odds of unaligned AI by 1% by “waiting” 20 years longer before creating AGI, we would lose out 20 years of extreme utility, which is roughly 0.00000001% of the total time (approximation of utility).
On net we gain 15 billion * 0.01 − 20 * 0.99 ≈ 150 million years of utility.
I do agree that if we start 20 years earlier, we could possibly also populate a little bit more of space, but that should be several orders of magnitudes smaller difference than 1%.
I’m genuinely curios to hear your thoughts on this.
You’re choosing a certain death for 32% of currently living humans. Or at least, the humans alive after [some medical research interval] at the time the AGI delay decision is made.
The [medical research interval] is the time it requires, withly massively parallel research, for a network of AGI systems to learn which medical interventions will prevent most forms of human death, from injury and aging. The economic motivation for a company to research this is obvious.
Delaying AGi is choosing to shift the time until [humans start living their MTBF given perfect bodies and only accidents and murder, which is thousands of years], 20 years into the future.
Note also that cryonics could be made to work, with clear and convincing evidence including revival of lab mammals, probably within a short time. That [research interval until working cryo] might be months.
Personally as a member of that subgroup, the improvement in odds ratio for misaligned AI for that 20 year period would need to be greater than 32%, or it isn’t worth it. Or essentially you’d have to show pDoom really was almost 1.0 to justify such a long delay.
Basically you would have to build AGIs and show they all inherently collaborate with each other to kill us by default. Too few people are convinced by EY, even if he is correct.
There’s another issue though, in that the benefits of AGI coming soon aren’t considered by the top comment on this thread, and assuming a symmetric or nearly symmetric structure of how much utility it produces, my own values suggest that the positives of AGI outweigh the potential for extinction, especially over longer periods, which is why I have said that capabilities work is net positive.
That would have been like, post WW2, a worldwide agreement not to build nukes. “Suuurrre” all the parties would say. “A weapon that let’s us win even if outnumbered, we don’t need THAt”.
And they would basically all defect on the agreement. The weapon is too powerful. Or one side would honor it and be facing neighbors with nuclear arms and none of their own.
Okay, so seems like our disagreement comes down to two different factors:
We have different value functions, I personally don’t value currently living human >> than future living humans, but I agree with the reasoning that to maximize your personal chance of living forever faster AI is better.
Getting AGI sooner will have much greater positive benefits than simply 20 years of peak happiness for everyone, but for example over billions of years the accumulative effect will be greater than value from a few hundreds of thousands of years of of AGI.
Further I find the idea of everyone agreeing to delaying AGI 20 years to be equally absurd as you suggest Gerald, I just thought is could be a helpful hypothetical scenario for discussing the subject.
Interesting take.
Perhaps there was something I misunderstood, but wouldn’t AI alignment work and AI capabilities slowdown still have extreme positive expected value even if the probability of unaligned AI is only 0.1-10%?
Let’s say the universe will exist for 15 billion more years until the big rip.
Let’s say we could decrease the odds of unaligned AI by 1% by “waiting” 20 years longer before creating AGI, we would lose out 20 years of extreme utility, which is roughly 0.00000001% of the total time (approximation of utility).
On net we gain 15 billion * 0.01 − 20 * 0.99 ≈ 150 million years of utility.
I do agree that if we start 20 years earlier, we could possibly also populate a little bit more of space, but that should be several orders of magnitudes smaller difference than 1%.
I’m genuinely curios to hear your thoughts on this.
You’re choosing a certain death for 32% of currently living humans. Or at least, the humans alive after [some medical research interval] at the time the AGI delay decision is made.
The [medical research interval] is the time it requires, withly massively parallel research, for a network of AGI systems to learn which medical interventions will prevent most forms of human death, from injury and aging. The economic motivation for a company to research this is obvious.
Delaying AGi is choosing to shift the time until [humans start living their MTBF given perfect bodies and only accidents and murder, which is thousands of years], 20 years into the future.
Note also that cryonics could be made to work, with clear and convincing evidence including revival of lab mammals, probably within a short time. That [research interval until working cryo] might be months.
Personally as a member of that subgroup, the improvement in odds ratio for misaligned AI for that 20 year period would need to be greater than 32%, or it isn’t worth it. Or essentially you’d have to show pDoom really was almost 1.0 to justify such a long delay.
Basically you would have to build AGIs and show they all inherently collaborate with each other to kill us by default. Too few people are convinced by EY, even if he is correct.
There’s another issue though, in that the benefits of AGI coming soon aren’t considered by the top comment on this thread, and assuming a symmetric or nearly symmetric structure of how much utility it produces, my own values suggest that the positives of AGI outweigh the potential for extinction, especially over longer periods, which is why I have said that capabilities work is net positive.
Also how would you agree on a 20 year delay?
That would have been like, post WW2, a worldwide agreement not to build nukes. “Suuurrre” all the parties would say. “A weapon that let’s us win even if outnumbered, we don’t need THAt”.
And they would basically all defect on the agreement. The weapon is too powerful. Or one side would honor it and be facing neighbors with nuclear arms and none of their own.
Okay, so seems like our disagreement comes down to two different factors:
We have different value functions, I personally don’t value currently living human >> than future living humans, but I agree with the reasoning that to maximize your personal chance of living forever faster AI is better.
Getting AGI sooner will have much greater positive benefits than simply 20 years of peak happiness for everyone, but for example over billions of years the accumulative effect will be greater than value from a few hundreds of thousands of years of of AGI.
Further I find the idea of everyone agreeing to delaying AGI 20 years to be equally absurd as you suggest Gerald, I just thought is could be a helpful hypothetical scenario for discussing the subject.