To me it sounds like you’re dividing possible futures into extinction, dystopia, and utopia, and noticing that you can’t really envision the latter. In which case, I agree, and I think if any of us could, we’d be a lot closer to solving alignment than we actually are.
Where my intuition cuts differently is that I think most seemingly-dystopian futures, where humans exist but are disempowered and dissatisfied with our lives and the world, are unstable or at best metastable, and will eventually give way to one of the other two categories. I’m sure stable dystopias are possible, of course, but ending up in one seems like it would require getting almost all of the steps of alignment right, but failing the last step of the grail quest.
Yes, this means I think most of the non-extinction futures you’re considering are really extinction-but-with-a-longer-delay-and-lots-of-suffering futures. But I also think there’s a sizable set of utopian-futures-outside-my-capacity-to-concretely-envision such that my P(doom) isn’t actually close to 100%.
To me it sounds like you’re dividing possible futures into extinction, dystopia, and utopia, and noticing that you can’t really envision the latter. In which case, I agree, and I think if any of us could, we’d be a lot closer to solving alignment than we actually are.
Where my intuition cuts differently is that I think most seemingly-dystopian futures, where humans exist but are disempowered and dissatisfied with our lives and the world, are unstable or at best metastable, and will eventually give way to one of the other two categories. I’m sure stable dystopias are possible, of course, but ending up in one seems like it would require getting almost all of the steps of alignment right, but failing the last step of the grail quest.
Yes, this means I think most of the non-extinction futures you’re considering are really extinction-but-with-a-longer-delay-and-lots-of-suffering futures. But I also think there’s a sizable set of utopian-futures-outside-my-capacity-to-concretely-envision such that my P(doom) isn’t actually close to 100%.