[Edit: I’m retracting this comment, as I made some incorrect assumptions about Scott’s claim.] This is wrong. It is well known that the only strictly proper scoring rule that depends only on the probability at the actually occurring value is the logarithmic scoring rule (if there are more than two alternatives), or translations and/or positive scaling of the same. In this case, that would be log(Normal(x | mu, sigma)), where x is the value that occurs, and mu and sigma^2 are the mean and variance of the normal distribution that fits the interval you defined at the given confidence level. This may be simplified to
-log(sigma^2) - (x—mu)^2 / sigma^2.
Your scoring rule is not a translation and/or positive scaling of the logarithmic scoring rule.
Throwing out an attempt to resolve the disagreement, sorry if this is actually what we are disagreeing about:
Am unknownigly using words that imply that I care about normal distributions? I am imagining getting honest reporting out of an agent trying to maximize expected score, but with arbitrary beliefs. I am only trying to get an honest reporting of the subjective 5th and 95th percentiles, and am not trying to get any other information.
I’m used to seeing normal (or log-normal) distributions fit to subjective confidence intervals—because the confidence intervals are being used to do some subjective probabilistic analysis. I assumed that was what you were doing, given that you were using the actual attained value x, and not just which of the three possibilities A:(x < left), B:(left < x < right), and C:(right < x) occurred.
Hmmm… you seem to have evaded the theorem about the only strictly proper local scoring rule being the logarithmic score, by only seeking to find the confidence interval, but using more information than just the region (A, B, or C) the outcome belongs to.
It would help to see a proof of the claim; do you have a reference or a link to a URL giving the proof?
Oh, a quick thing thats not a proof that may convince you it is true:
It works exactly the same way as saying that measuring the distance between reported value and true value incentivizes honest reporting of your median. (The point you think it the true value is above with probability 50%)
This scoring rule does not depend only on the probability at the actually occuring value. You dont even report the probability at any value. I am not trying to incentivize reporting of probabilities of specific value, I am trying to incentivize reporting an interval such that the person reporting the belief believes the point will lie in with probability 90%.
Your rule seems to be trying to do something else, but it will not incentivize me giving my 90% confidence interval in cases where my beliefs are not normally distributed.
[Edit: I’m retracting this comment, as I made some incorrect assumptions about Scott’s claim.] This is wrong. It is well known that the only strictly proper scoring rule that depends only on the probability at the actually occurring value is the logarithmic scoring rule (if there are more than two alternatives), or translations and/or positive scaling of the same. In this case, that would be log(Normal(x | mu, sigma)), where x is the value that occurs, and mu and sigma^2 are the mean and variance of the normal distribution that fits the interval you defined at the given confidence level. This may be simplified to
-log(sigma^2) - (x—mu)^2 / sigma^2.
Your scoring rule is not a translation and/or positive scaling of the logarithmic scoring rule.
Throwing out an attempt to resolve the disagreement, sorry if this is actually what we are disagreeing about:
Am unknownigly using words that imply that I care about normal distributions? I am imagining getting honest reporting out of an agent trying to maximize expected score, but with arbitrary beliefs. I am only trying to get an honest reporting of the subjective 5th and 95th percentiles, and am not trying to get any other information.
I’m used to seeing normal (or log-normal) distributions fit to subjective confidence intervals—because the confidence intervals are being used to do some subjective probabilistic analysis. I assumed that was what you were doing, given that you were using the actual attained value x, and not just which of the three possibilities A:(x < left), B:(left < x < right), and C:(right < x) occurred.
Hmmm… you seem to have evaded the theorem about the only strictly proper local scoring rule being the logarithmic score, by only seeking to find the confidence interval, but using more information than just the region (A, B, or C) the outcome belongs to.
It would help to see a proof of the claim; do you have a reference or a link to a URL giving the proof?
I dont have a reference. gjm’s comment gives a quick sketch.
Oh, a quick thing thats not a proof that may convince you it is true:
It works exactly the same way as saying that measuring the distance between reported value and true value incentivizes honest reporting of your median. (The point you think it the true value is above with probability 50%)
This scoring rule does not depend only on the probability at the actually occuring value. You dont even report the probability at any value. I am not trying to incentivize reporting of probabilities of specific value, I am trying to incentivize reporting an interval such that the person reporting the belief believes the point will lie in with probability 90%.
Your rule seems to be trying to do something else, but it will not incentivize me giving my 90% confidence interval in cases where my beliefs are not normally distributed.