If we’re going to be talking about a subjective “okay” standard, what makes 20% okay, but 0.00002% not-okay?
Scale sensitivity.
From our perspective today, 20% max value and 0.00002% max value both emotionally mean “infinity”, so they are like the same thing. When we get to the 0.00002% max value, the difference between “all that we can ever have” and “we could have had a million times more” will feel differently.
(Intuition: How would you feel if you found out that your life could have been literally million times better, but someone decided for you that both options are good enough so it makes no sense to fret about the difference?)
Counter-intuition, if I’m playing Russian Roulette while holding a lottery ticket in my other hand, then staying alive but not winning the lottery is an “okay” outcome.
Believing that ‘a perfected human civilization spanning hundreds of galaxies’ is a loss condition of AI, rather than a win condition, is not entirely obviously wrong, but certainly doesn’t seem obviously right.
And if you argue ‘AI is extraordinarily likely to lead to a bad outcome for humans’ while including ‘hundreds of galaxies of humans’ as a ‘bad outcome’, that seems fairly disingenuous.
In economics, “we can model utility as logarithmic in wealth”, even after adding human capital to wealth, feels like a silly asymptotic approximation that obviously breaks down in the other direction as wealth goes to zero and modeled utility to negative infinity.
In cosmology, though, the difference between “humanity only gets a millionth of its light cone” and “humanity goes extinct” actually does feel bigger than the difference between “humanity only gets a millionth of its light cone” and “humanity gets a fifth of its light cone”; not infinitely bigger, but a lot more than you’d expect by modeling marginal utility as a constant as wealth goes to zero.
This is all subjective; others’ feelings may differ.
(I’m also open in theory to valuing an appropriately-complete successor to humanity equally to humanity 1.0, whether the successor is carbon or silicon or whatever, but I don’t see how “appropriately-complete” is likely so I’m ignoring the possibility above.)
Scale sensitivity.
From our perspective today, 20% max value and 0.00002% max value both emotionally mean “infinity”, so they are like the same thing. When we get to the 0.00002% max value, the difference between “all that we can ever have” and “we could have had a million times more” will feel differently.
(Intuition: How would you feel if you found out that your life could have been literally million times better, but someone decided for you that both options are good enough so it makes no sense to fret about the difference?)
Counter-intuition, if I’m playing Russian Roulette while holding a lottery ticket in my other hand, then staying alive but not winning the lottery is an “okay” outcome.
Believing that ‘a perfected human civilization spanning hundreds of galaxies’ is a loss condition of AI, rather than a win condition, is not entirely obviously wrong, but certainly doesn’t seem obviously right.
And if you argue ‘AI is extraordinarily likely to lead to a bad outcome for humans’ while including ‘hundreds of galaxies of humans’ as a ‘bad outcome’, that seems fairly disingenuous.
In economics, “we can model utility as logarithmic in wealth”, even after adding human capital to wealth, feels like a silly asymptotic approximation that obviously breaks down in the other direction as wealth goes to zero and modeled utility to negative infinity.
In cosmology, though, the difference between “humanity only gets a millionth of its light cone” and “humanity goes extinct” actually does feel bigger than the difference between “humanity only gets a millionth of its light cone” and “humanity gets a fifth of its light cone”; not infinitely bigger, but a lot more than you’d expect by modeling marginal utility as a constant as wealth goes to zero.
This is all subjective; others’ feelings may differ.
(I’m also open in theory to valuing an appropriately-complete successor to humanity equally to humanity 1.0, whether the successor is carbon or silicon or whatever, but I don’t see how “appropriately-complete” is likely so I’m ignoring the possibility above.)