I spent about 15 minutes reading and jumping around this post a bit confused about what the main idea was. I think I finally found it in this text toward the middle—extracting here in case it helps others understand (please let me know if I’m missing a more central part, Daphne_W):
And in the bottom regime [referring to a diagram in the post], where we find ourselves, where both bad and neutral outcomes are much more likely than good outcomes, increasing the probability of good outcomes is practically pointless. All that matters is shifting probability from bad worlds to neutral worlds.
This is a radical departure from the status quo. Most alignment researchers spend their effort developing tools that might make good worlds more likely. Some may be applicable to reducing the probability of bad worlds, but almost nothing is done to increase the probability of neutral worlds. This is understandable – promoting human extinction has a bad reputation, and it’s not something you should do unless you’re confident. You were ready to call yourself confident when it meant dying with dignity, though, and probabilities don’t change depending on how uncomfortable the resulting optimal policy makes you.
Meanwhile, because increasing the probability of neutral worlds is an underexplored field of research, it is likely that there are high-impact projects available. Building friendly AGI may be out of reach of a Manhattan project, but building neutral AGI might not.
I find this to be a depressing idea, but think it’s also interesting and potentially worthwhile.
Q5 is an exploration of my uncertainty in spite of me not being able to find faults with Clippy’s argument, as well as what I expect others’ hesitance might be. If Clippy’s argument is correct, then the section you highlight seems like the logical conclusion.
I spent about 15 minutes reading and jumping around this post a bit confused about what the main idea was. I think I finally found it in this text toward the middle—extracting here in case it helps others understand (please let me know if I’m missing a more central part, Daphne_W):
I find this to be a depressing idea, but think it’s also interesting and potentially worthwhile.
Both that and Q5 seem important to me.
Q5 is an exploration of my uncertainty in spite of me not being able to find faults with Clippy’s argument, as well as what I expect others’ hesitance might be. If Clippy’s argument is correct, then the section you highlight seems like the logical conclusion.
Interesting. I thought the main idea was contained in Question 5.
Mitchell_Porter’s summary seems to concur with the text I focused on.
So you thought it was more about the fact that people would reject extinction, even if the likely alternative were huge amounts of suffering?