The precision-recall tradeoff definitely varies from one task to another. I split tasks into “precision-maxxing” (where false-positives are costlier than false-negatives) and “recall-maxxing” (where false-negatives are costlier than false-positives).
I disagree with your estimate of the relative costs in history and in medical research. The truth is that academia does surprisingly well at filtering out the good from the bad.
Suppose I select two medical papers at random — one from the set of good medical papers, and one from the set of crap medical papers. If a wizard offered to permanently delete both papers from reality, that would rarely be a good deal because the benefit of deleting the crap paper is negligible compared to the cost of losing the good paper.
But what if the wizard offered to delete M crap papers and one good paper? How large must M be before this is a good deal? The minimal acceptable M is CFN/CFP, so τ⋆=CFP/(CFP+CFN)=1/(1+M). I’d guess that M is at least 30, so τ⋆ is at most 3.5%.
Reviewing the examples in the post again, I think I was confused on first reading. I initially read the nuclear reactor example as being a completed version of the Michaelangelo example, but now I see it clearly includes the harms issue I was thinking about.
I also think that the Library of Babel example contains my search thoughts, just not separated out in the same way as in the Poorly Calibrated Heuristics section.
The precision-recall tradeoff definitely varies from one task to another. I split tasks into “precision-maxxing” (where false-positives are costlier than false-negatives) and “recall-maxxing” (where false-negatives are costlier than false-positives).
I disagree with your estimate of the relative costs in history and in medical research. The truth is that academia does surprisingly well at filtering out the good from the bad.
Suppose I select two medical papers at random — one from the set of good medical papers, and one from the set of crap medical papers. If a wizard offered to permanently delete both papers from reality, that would rarely be a good deal because the benefit of deleting the crap paper is negligible compared to the cost of losing the good paper.
But what if the wizard offered to delete M crap papers and one good paper? How large must M be before this is a good deal? The minimal acceptable M is CFN/CFP, so τ⋆=CFP/(CFP+CFN)=1/(1+M). I’d guess that M is at least 30, so τ⋆ is at most 3.5%.
Reviewing the examples in the post again, I think I was confused on first reading. I initially read the nuclear reactor example as being a completed version of the Michaelangelo example, but now I see it clearly includes the harms issue I was thinking about.
I also think that the Library of Babel example contains my search thoughts, just not separated out in the same way as in the Poorly Calibrated Heuristics section.
I’m going to chalk this one up to an oops!