I would say that poor social epistemology and, poor social axiology and mediocre individual rationality are the big culprits that prevent many scientists from taking AI risk seriously.
By “social axiology” I mean that our society is just not consequentialist enough. We don’t solve problems that way, and even the debate about global warming is not really dealing well with the problem of how to quantify risks under uncertainty. We don’t try to improve the world in a systematic, rational way; rather it is done piecemeal.
I would say that poor social epistemology and, poor social axiology and mediocre individual rationality are the big culprits that prevent many scientists from taking AI risk seriously.
By “social axiology” I mean that our society is just not consequentialist enough. We don’t solve problems that way, and even the debate about global warming is not really dealing well with the problem of how to quantify risks under uncertainty. We don’t try to improve the world in a systematic, rational way; rather it is done piecemeal.