Being risk-averse around people’s lives is only a good strategy when you’re trading off against something else that isn’t human lives. If you have the choice to save 400 lives with certainty, or a 90% chance to save 500 lives, choosing the former is essentially condemning 50 people to death. At that point, you’re just behaving suboptimally.
Being risk-averse works if you’re trading off other things. E.g, if you could release a new car now that you’re almost certain is safe, you might be risk-averse and call for more tests. As long as people won’t die from you delaying this car, being risk-averse is a reasonable strategy here.
Given your Point 1 from earlier, there is no reason to expect the effect size will always be greater. If the effect on reducing X-risks from co-ordination becomes small enough, or the risk of a particular X-risk becomes large enough, this changes the equation. If you believe, like many in this forum do, that AI represents the lion’s share of X-risk, focusing on AI directly is probably more effective. If you believe that x-risk is diversified, that there’s some chance from AI, some from pandemics, some from nuclear war, some from climate change, etc. then co-ordination makes more sense. Co-ordination has a small effect on all x-risks, direct work has a larger effect on a single x-risk.
The point I’m trying to make here is this. There are perfectly reasonable states of the world where “Improve co-ordination” is the best action to take to reduce x-risk. There are also perfectly reasonable states of the world where “Work directly on <Risk A>” is the best action to take to reduce x-risk. You won’t be able to find out which is which if you believe one is “always” the case.
What I would suggest is to ask “What would cause me to change my mind and believe improving co-ordination is NOT the best way to work on x-risk”, and then seek out whether those things are true or not. If you don’t believe they are, great, that’s fine.
That said, it wouldn’t be fair to ask you what would change your mind without presenting my own. On my end, what would convince me that improving co-ordination is more important than direct AI work:
AI is less dangerous than I expect, so that the x-risk profile is more diversified instead of mostly AI.
We already have more technical progress in AI safety than I believe we have, so we don’t need more and should focus on either co-ordination or the next most dangerous x-risk. (Which I believe is pandemics, which is both less dangerous than AI and more responsive to government co-ordination in my opinion)
AI is far more dangerous than I expect, to the point where the AI alignment problem is unsolvable and co-ordination is the only solution.
Being risk-averse around people’s lives is only a good strategy when you’re trading off against something else that isn’t human lives. If you have the choice to save 400 lives with certainty, or a 90% chance to save 500 lives, choosing the former is essentially condemning 50 people to death. At that point, you’re just behaving suboptimally.
Being risk-averse works if you’re trading off other things. E.g, if you could release a new car now that you’re almost certain is safe, you might be risk-averse and call for more tests. As long as people won’t die from you delaying this car, being risk-averse is a reasonable strategy here.
Given your Point 1 from earlier, there is no reason to expect the effect size will always be greater. If the effect on reducing X-risks from co-ordination becomes small enough, or the risk of a particular X-risk becomes large enough, this changes the equation. If you believe, like many in this forum do, that AI represents the lion’s share of X-risk, focusing on AI directly is probably more effective. If you believe that x-risk is diversified, that there’s some chance from AI, some from pandemics, some from nuclear war, some from climate change, etc. then co-ordination makes more sense. Co-ordination has a small effect on all x-risks, direct work has a larger effect on a single x-risk.
The point I’m trying to make here is this. There are perfectly reasonable states of the world where “Improve co-ordination” is the best action to take to reduce x-risk. There are also perfectly reasonable states of the world where “Work directly on <Risk A>” is the best action to take to reduce x-risk. You won’t be able to find out which is which if you believe one is “always” the case.
What I would suggest is to ask “What would cause me to change my mind and believe improving co-ordination is NOT the best way to work on x-risk”, and then seek out whether those things are true or not. If you don’t believe they are, great, that’s fine.
That said, it wouldn’t be fair to ask you what would change your mind without presenting my own. On my end, what would convince me that improving co-ordination is more important than direct AI work:
AI is less dangerous than I expect, so that the x-risk profile is more diversified instead of mostly AI.
We already have more technical progress in AI safety than I believe we have, so we don’t need more and should focus on either co-ordination or the next most dangerous x-risk. (Which I believe is pandemics, which is both less dangerous than AI and more responsive to government co-ordination in my opinion)
AI is far more dangerous than I expect, to the point where the AI alignment problem is unsolvable and co-ordination is the only solution.