As for Point 2, you can absolutely compare a certainty and a probability. If I offered you a certainty of $10, or a 10% chance of $1,000,000, would you take the $10 because you can’t compare certainties and probabilities, and I’m only ever going to offer you the deal once?
That then brings me to question 3. The button I would press would be the one that reduces total X-risk the most. If both buttons reduced X-risk by 1%, I would press the 100% one. If the 100% button reduced X-risk by 0.1%, and the 10% one reduced X-risk by 10%, I would pick the second one, for an expected value of 1% X-risk reduction. You have to take the effect size into account. We can disagree on what the effect sizes are, but you still need to consider them.
Interesting, I see what you mean reagrding probability and it makes sense. I guess perhaps, what is missing is that when it comes to questions of peoples lives we may have a stronger imperative to be more risk-averse.
I completely agree with you about effect size. I guess what I would say is that that given my point 1 from earlier about the variety of X-risks coordination would contirbute in solving then the effect size will always be greater. If we want to maximise utility its the best chance we have. The added bonuses are that it is comparatively tractable and immediate avoiding the recent criticicisms about longtermism, while simoultnously being a longtermist solution.
Regadless, it does seem that coordination problems are underdiscussed in the community, will try and make a decent main post once my academic committments clear up a bit.
Being risk-averse around people’s lives is only a good strategy when you’re trading off against something else that isn’t human lives. If you have the choice to save 400 lives with certainty, or a 90% chance to save 500 lives, choosing the former is essentially condemning 50 people to death. At that point, you’re just behaving suboptimally.
Being risk-averse works if you’re trading off other things. E.g, if you could release a new car now that you’re almost certain is safe, you might be risk-averse and call for more tests. As long as people won’t die from you delaying this car, being risk-averse is a reasonable strategy here.
Given your Point 1 from earlier, there is no reason to expect the effect size will always be greater. If the effect on reducing X-risks from co-ordination becomes small enough, or the risk of a particular X-risk becomes large enough, this changes the equation. If you believe, like many in this forum do, that AI represents the lion’s share of X-risk, focusing on AI directly is probably more effective. If you believe that x-risk is diversified, that there’s some chance from AI, some from pandemics, some from nuclear war, some from climate change, etc. then co-ordination makes more sense. Co-ordination has a small effect on all x-risks, direct work has a larger effect on a single x-risk.
The point I’m trying to make here is this. There are perfectly reasonable states of the world where “Improve co-ordination” is the best action to take to reduce x-risk. There are also perfectly reasonable states of the world where “Work directly on <Risk A>” is the best action to take to reduce x-risk. You won’t be able to find out which is which if you believe one is “always” the case.
What I would suggest is to ask “What would cause me to change my mind and believe improving co-ordination is NOT the best way to work on x-risk”, and then seek out whether those things are true or not. If you don’t believe they are, great, that’s fine.
That said, it wouldn’t be fair to ask you what would change your mind without presenting my own. On my end, what would convince me that improving co-ordination is more important than direct AI work:
AI is less dangerous than I expect, so that the x-risk profile is more diversified instead of mostly AI.
We already have more technical progress in AI safety than I believe we have, so we don’t need more and should focus on either co-ordination or the next most dangerous x-risk. (Which I believe is pandemics, which is both less dangerous than AI and more responsive to government co-ordination in my opinion)
AI is far more dangerous than I expect, to the point where the AI alignment problem is unsolvable and co-ordination is the only solution.
I agree with you on the first point completely.
As for Point 2, you can absolutely compare a certainty and a probability. If I offered you a certainty of $10, or a 10% chance of $1,000,000, would you take the $10 because you can’t compare certainties and probabilities, and I’m only ever going to offer you the deal once?
That then brings me to question 3. The button I would press would be the one that reduces total X-risk the most. If both buttons reduced X-risk by 1%, I would press the 100% one. If the 100% button reduced X-risk by 0.1%, and the 10% one reduced X-risk by 10%, I would pick the second one, for an expected value of 1% X-risk reduction. You have to take the effect size into account. We can disagree on what the effect sizes are, but you still need to consider them.
Interesting, I see what you mean reagrding probability and it makes sense. I guess perhaps, what is missing is that when it comes to questions of peoples lives we may have a stronger imperative to be more risk-averse.
I completely agree with you about effect size. I guess what I would say is that that given my point 1 from earlier about the variety of X-risks coordination would contirbute in solving then the effect size will always be greater. If we want to maximise utility its the best chance we have. The added bonuses are that it is comparatively tractable and immediate avoiding the recent criticicisms about longtermism, while simoultnously being a longtermist solution.
Regadless, it does seem that coordination problems are underdiscussed in the community, will try and make a decent main post once my academic committments clear up a bit.
Being risk-averse around people’s lives is only a good strategy when you’re trading off against something else that isn’t human lives. If you have the choice to save 400 lives with certainty, or a 90% chance to save 500 lives, choosing the former is essentially condemning 50 people to death. At that point, you’re just behaving suboptimally.
Being risk-averse works if you’re trading off other things. E.g, if you could release a new car now that you’re almost certain is safe, you might be risk-averse and call for more tests. As long as people won’t die from you delaying this car, being risk-averse is a reasonable strategy here.
Given your Point 1 from earlier, there is no reason to expect the effect size will always be greater. If the effect on reducing X-risks from co-ordination becomes small enough, or the risk of a particular X-risk becomes large enough, this changes the equation. If you believe, like many in this forum do, that AI represents the lion’s share of X-risk, focusing on AI directly is probably more effective. If you believe that x-risk is diversified, that there’s some chance from AI, some from pandemics, some from nuclear war, some from climate change, etc. then co-ordination makes more sense. Co-ordination has a small effect on all x-risks, direct work has a larger effect on a single x-risk.
The point I’m trying to make here is this. There are perfectly reasonable states of the world where “Improve co-ordination” is the best action to take to reduce x-risk. There are also perfectly reasonable states of the world where “Work directly on <Risk A>” is the best action to take to reduce x-risk. You won’t be able to find out which is which if you believe one is “always” the case.
What I would suggest is to ask “What would cause me to change my mind and believe improving co-ordination is NOT the best way to work on x-risk”, and then seek out whether those things are true or not. If you don’t believe they are, great, that’s fine.
That said, it wouldn’t be fair to ask you what would change your mind without presenting my own. On my end, what would convince me that improving co-ordination is more important than direct AI work:
AI is less dangerous than I expect, so that the x-risk profile is more diversified instead of mostly AI.
We already have more technical progress in AI safety than I believe we have, so we don’t need more and should focus on either co-ordination or the next most dangerous x-risk. (Which I believe is pandemics, which is both less dangerous than AI and more responsive to government co-ordination in my opinion)
AI is far more dangerous than I expect, to the point where the AI alignment problem is unsolvable and co-ordination is the only solution.