You should take them exactly the right amount of seriously, as useful ways to discuss questions that are highly imprecise. I mostly only have people in about five buckets, which are something like:
[...]
The Leike Zone (10%-90%), where mostly the wise responses don’t much change.
[...]
In this case, a lot of people have put a lot of thought into the question.
I feel that when people put a lot of thought into the question of P(doom) associated with AI, they should at least provide pairs of conditional estimates, P(Doom|X) and P(Doom|not X).
Especially, when X is a really significant factor which can both increase and decrease chances of various scenarios leading to doom.
A single number can’t really be well calibrated, especially when the situation is so complex. When one has a pair of numbers, one can at least try to compare P(Doom|X) and P(Doom|not X), see which one is larger and why, do some extra effort making sure that important parts of the overall picture are not missed.
(Of course, Leike’s choice, 10%-90%, which is saying, effectively, “I don’t know, but it’s significant”, is also entirely valid. In this case, the actual answer in terms of a pair of numbers is likely to be, “P(Doom|X) and P(Doom|not X) are both significant, and we don’t know which one is higher”. To start saying “P(Doom|X) > P(Doom|not X)” or “P(Doom|X) < P(Doom|not X)” would take a lot more effort.)
I feel that when people put a lot of thought into the question of P(doom) associated with AI, they should at least provide pairs of conditional estimates, P(Doom|X) and P(Doom|not X).
Especially, when X is a really significant factor which can both increase and decrease chances of various scenarios leading to doom.
A single number can’t really be well calibrated, especially when the situation is so complex. When one has a pair of numbers, one can at least try to compare P(Doom|X) and P(Doom|not X), see which one is larger and why, do some extra effort making sure that important parts of the overall picture are not missed.
(Of course, Leike’s choice, 10%-90%, which is saying, effectively, “I don’t know, but it’s significant”, is also entirely valid. In this case, the actual answer in terms of a pair of numbers is likely to be, “P(Doom|X) and P(Doom|not X) are both significant, and we don’t know which one is higher”. To start saying “P(Doom|X) > P(Doom|not X)” or “P(Doom|X) < P(Doom|not X)” would take a lot more effort.)