What’s the motivation? In what case is lower accuracy for higher consistency a reasonable trade off? Especially consistency over time sounds like something that would discourage updating on new evidence.
Some examples are where people care more about fairness, such as criminal sentencing and enterprise software pricing.
However you’re right that implicit in the question was “without new information appearing”, although you’d want the answer to update the same way every time the same new information appeared.
If every study on depression would use it’s own metric for depression that’s optimal for the specific study it would be hard to learn from the studies and aggregate information from them. It’s much better when you have a metric that has consistency.
Consistent measurements allow reacting to how a metric changes over time which is often very useful for evaluating interventions.
This is true, but it doesn’t fit well with the given example of “When will [country] develop the nuclear bomb?”. The problem isn’t that people can’t agree what “nuclear bomb” means or who already has them. The problem is that people are working from different priors and extrapolating them in different ways.
What’s the motivation? In what case is lower accuracy for higher consistency a reasonable trade off? Especially consistency over time sounds like something that would discourage updating on new evidence.
I attempted to summarize some of the motivation for this here: https://www.lesswrong.com/posts/Df2uFGKtLWR7jDr5w/?commentId=tdbfBQ6xFRc7j8nBE
Some examples are where people care more about fairness, such as criminal sentencing and enterprise software pricing.
However you’re right that implicit in the question was “without new information appearing”, although you’d want the answer to update the same way every time the same new information appeared.
If every study on depression would use it’s own metric for depression that’s optimal for the specific study it would be hard to learn from the studies and aggregate information from them. It’s much better when you have a metric that has consistency.
Consistent measurements allow reacting to how a metric changes over time which is often very useful for evaluating interventions.
This is true, but it doesn’t fit well with the given example of “When will [country] develop the nuclear bomb?”. The problem isn’t that people can’t agree what “nuclear bomb” means or who already has them. The problem is that people are working from different priors and extrapolating them in different ways.