Zeroth approximation: even the experts don’t know, I am not an expert, so I know even less, thus I should not make any of my decisions based on singularity-related arguments.
That is making a decision based on singularity-related arguments. Denial isn’t a privileged case that you can invoke when convenient. An argument that arguments for or against X should be ignored is an argument relating to X.
First approximation: find a reference class of predictions that were supposed to come true within 50 years or so, unchanged for decades, and see when (some of them) are resolved.
That seems to be a suitable prior to adopt if you have reason to believe that the AI related predictions are representatively sampled from the class of 50 year predictions. It’s almost certainly better than going with “0.5”. In the absence of any evidence whatsoever about either AI itself or those making the predictions it would even be an appropriate prediction to use for decision making. Of course if I do happen to have any additional information and ignore it then I am just going to be assigning probabilities that are subjectively objectively false. It doesn’t matter how much I plead that I am just being careful.
That is making a decision based on singularity-related arguments. Denial isn’t a privileged case that you can invoke when convenient. An argument that arguments for or against X should be ignored is an argument relating to X.
That seems to be a suitable prior to adopt if you have reason to believe that the AI related predictions are representatively sampled from the class of 50 year predictions. It’s almost certainly better than going with “0.5”. In the absence of any evidence whatsoever about either AI itself or those making the predictions it would even be an appropriate prediction to use for decision making. Of course if I do happen to have any additional information and ignore it then I am just going to be assigning probabilities that are subjectively objectively false. It doesn’t matter how much I plead that I am just being careful.