Yours is the kind of response I have to posts like the OP and also Holden’s “Empowerment and Catastrophic Risk,” though I wouldn’t place so much specific emphasis on e.g. decision theory.
It is important to analyze general features of the world and build up many outside views, but one must also exercise the Be Specific skill. If several outside views tell me to eat Osha Thai, but then I snack on some Synsepalum dulcificum and I know it makes sour food taste sweet, then I should update heavily against the results of my original model combination, even if the Osha Thai recommendation was a robust result of 20+ models under model combination. Similarly, even if you have a very strong outside view that your lottery ticket is not the winner, a simple observation that the number on your ticket matches the announced winner on live TV should allow you to update all the way to belief that you won.
To consider a case somewhat more analogous to risks and AI, there were lots of outside views in 1938 suggesting that one shouldn’t invest billions in an unprecedented technology that would increase our bombing power by several orders of magnitude, based on then-theoretical physics. Definitely an “unproven cause.” And yet there were strong reasons to suggest it would be possible, and could be a determining factor in WWII, even though lots of the initial research would end up being on the wrong path and so on.
Also see Eliezer’s comment here, and its supporting post here.
Yours is the kind of response I have to posts like the OP and also Holden’s “Empowerment and Catastrophic Risk,” though I wouldn’t place so much specific emphasis on e.g. decision theory.
It is important to analyze general features of the world and build up many outside views, but one must also exercise the Be Specific skill. If several outside views tell me to eat Osha Thai, but then I snack on some Synsepalum dulcificum and I know it makes sour food taste sweet, then I should update heavily against the results of my original model combination, even if the Osha Thai recommendation was a robust result of 20+ models under model combination. Similarly, even if you have a very strong outside view that your lottery ticket is not the winner, a simple observation that the number on your ticket matches the announced winner on live TV should allow you to update all the way to belief that you won.
To consider a case somewhat more analogous to risks and AI, there were lots of outside views in 1938 suggesting that one shouldn’t invest billions in an unprecedented technology that would increase our bombing power by several orders of magnitude, based on then-theoretical physics. Definitely an “unproven cause.” And yet there were strong reasons to suggest it would be possible, and could be a determining factor in WWII, even though lots of the initial research would end up being on the wrong path and so on.
Also see Eliezer’s comment here, and its supporting post here.