This is not about alien aircraft, this is just a completely wrong way to approach updating. The set of observations/experiments being evaluated is filtered by what was actually observed and by the narrative around the hypothesis (which is in turn not independent from what was actually observed). There are other potential observations that didn’t happen, and that fact is also evidence, and yet more observations that did happen but aren’t genre-appopriate. By not updating on these other potential observations, the evidence is heavily filtered, and so updating on what remains is of no use at all in estimating how much weight to put on any given hypothesis.
All filtered evidence is good for is formulating hypotheses, or even just inspiring ideas that are not hypotheses. If you do formulate a hypothesis, it’s then necessary to carefully think about which potentially observable things would be predicted by it, compared to its alternatives, in worlds reframed from within the hypotheses (at which point the noise of fake evidence gets a reckoning, and absence of evidence manifests as evidence of absence). Even that risks privileging strange hypotheses, but at least we can fight that with priors. Very strange hypotheses don’t give useful predictions of observable things, so they probably shouldn’t even count as hypotheses in the context of credences and updating.
Agreed that paying attention to how evidence is filtered is super important. But, in principle, you can still derive conclusions from filtered evidence. It’s just really hard, especially if the filter is strong and hard to characterize (as is the case with UAPs).
Fair enough. (though...really you could in principle still handle filtered evidence in a formalish way. It just would require a bunch of additional complication regarding your priors and evidence on how the filter operates).
Yeah, I thought to note that in the comment that starts this thread; that’s not the kind of thing that seems practical when coordinating updating in an informal way. So more carefully, the intended scope of the comment is formal updating (computing of credences) that’s directed informally (choosing the potential observations and hypotheses to pay attention to).
This is not about alien aircraft, this is just a completely wrong way to approach updating. The set of observations/experiments being evaluated is filtered by what was actually observed and by the narrative around the hypothesis (which is in turn not independent from what was actually observed). There are other potential observations that didn’t happen, and that fact is also evidence, and yet more observations that did happen but aren’t genre-appopriate. By not updating on these other potential observations, the evidence is heavily filtered, and so updating on what remains is of no use at all in estimating how much weight to put on any given hypothesis.
All filtered evidence is good for is formulating hypotheses, or even just inspiring ideas that are not hypotheses. If you do formulate a hypothesis, it’s then necessary to carefully think about which potentially observable things would be predicted by it, compared to its alternatives, in worlds reframed from within the hypotheses (at which point the noise of fake evidence gets a reckoning, and absence of evidence manifests as evidence of absence). Even that risks privileging strange hypotheses, but at least we can fight that with priors. Very strange hypotheses don’t give useful predictions of observable things, so they probably shouldn’t even count as hypotheses in the context of credences and updating.
Agreed that paying attention to how evidence is filtered is super important. But, in principle, you can still derive conclusions from filtered evidence. It’s just really hard, especially if the filter is strong and hard to characterize (as is the case with UAPs).
Sure, but that’s not about formal-ish updating that frames this post, where you are writing down likelihood ratios and computing credences.
Fair enough. (though...really you could in principle still handle filtered evidence in a formalish way. It just would require a bunch of additional complication regarding your priors and evidence on how the filter operates).
Yeah, I thought to note that in the comment that starts this thread; that’s not the kind of thing that seems practical when coordinating updating in an informal way. So more carefully, the intended scope of the comment is formal updating (computing of credences) that’s directed informally (choosing the potential observations and hypotheses to pay attention to).