In terms of “miracles”—do you think they look more like some empirical result or some new genius comes up with a productive angle? Though I am inclined, even as a normie, to believe that human geniuses are immeasurably disappointing, you have sown a lot of seeds—and alignmentpilled a lot of clever people—and presumably some spectacularly clever people. Maybe some new prodigy will show up. My timelines are short—like less than 10 years wouldn’t surprise me—but the kids you alignmentpilled in 2008-2015 will be reaching peak productivity in the next few years. If it’s going to happen, it might happen soon.
Both seem unlikely, probably about equally unlikely… maybe leaning slightly more towards the empirical side, but you wouldn’t even get those without an at least slightly genius looking into it, I think.
A list of potential miracles (including empirical “crucial considerations” [/wishful thinking] that could mean the problem is bypassed):
Possibility of a failed (unaligned) takeoff scenario where the AI fails to model humans accurately enough (i.e. realise smart humans could detect its “hidden” activity in a certain way). [This may only set things back a few months to years; or could lead to some kind of Butlerian Jihad if there is a sufficiently bad (but ultimately recoverable) global catastrophe (and then much more time for Alignment the second time around?)].
Valence realism being true. Binding problem vs AGI Alignment.
Omega experiencing every possible consciousness and picking the best? [Could still lead to x-risk in terms of a Hedonium Shockwave].
Moral Realism being true (and the AI discovering it and the true morality being human-compatible).
In terms of “miracles”—do you think they look more like some empirical result or some new genius comes up with a productive angle? Though I am inclined, even as a normie, to believe that human geniuses are immeasurably disappointing, you have sown a lot of seeds—and alignmentpilled a lot of clever people—and presumably some spectacularly clever people. Maybe some new prodigy will show up. My timelines are short—like less than 10 years wouldn’t surprise me—but the kids you alignmentpilled in 2008-2015 will be reaching peak productivity in the next few years. If it’s going to happen, it might happen soon.
Both seem unlikely, probably about equally unlikely… maybe leaning slightly more towards the empirical side, but you wouldn’t even get those without an at least slightly genius looking into it, I think.
A list of potential miracles (including empirical “crucial considerations” [/wishful thinking] that could mean the problem is bypassed):
Possibility of a failed (unaligned) takeoff scenario where the AI fails to model humans accurately enough (i.e. realise smart humans could detect its “hidden” activity in a certain way). [This may only set things back a few months to years; or could lead to some kind of Butlerian Jihad if there is a sufficiently bad (but ultimately recoverable) global catastrophe (and then much more time for Alignment the second time around?)].
Valence realism being true. Binding problem vs AGI Alignment.
Omega experiencing every possible consciousness and picking the best? [Could still lead to x-risk in terms of a Hedonium Shockwave].
Moral Realism being true (and the AI discovering it and the true morality being human-compatible).
Natural abstractions leading to Alignment by Default?
Rohin’s links here.
AGI discovers new physics and exits to another dimension (like the creatures in Greg Egan’s Crystal Nights).
Simulation/anthropics stuff.
Alien Information Theory being true!? (And the aliens having solved alignment).