A problem with trying to isolated calibration is that on the true/false test, the subject could always assign 50% probability to both true and false and be right 50% of the time, achieving perfect calibration.
Interestingly, this problem can be avoided by taking the domain of possible answers to be the natural numbers, or n-dimensional Euclidean space, etc., over which no uniform distribution is possible, and then asking your test subject to specify a probability distribution over the whole space. This is potentially impractical, though, and I’m not certain it can’t be gamed in other ways.
Interestingly, this problem can be avoided by taking the domain of possible answers to be the natural numbers, or n-dimensional Euclidean space, etc., over which no uniform distribution is possible, and then asking your test subject to specify a probability distribution over the whole space. This is potentially impractical, though, and I’m not certain it can’t be gamed in other ways.