consider a decision problem in which there are k choices, each of which has true estimated [expected value] of 0.
Lukeprog, if I’ve understood you correctly, then this is no good; this is a corner case. The question to be answered here is whether we should expect a “common sense” executive who favors plans with a high prior estimate to do better than a “technical” analyst who favors plans that perform well according to the formal estimation criteria. By assuming that all prior estimates are identical except for bias, this assumption ensures that the technical analyst will win. This, however, begs the question. One could just as easily assume that there is large variation in the true expected values, and that the formal criteria will always produce an estimate of 0, in which case the common sense executive will always win.
Am I missing something? I like the topic; I would enjoy reading about which approach we should expect to perform better in a typical situation.
That’s fine; you’re more than welcome to illustrate the problem, and your analysis does in fact do that. It does it very well; your writing, as always, is very lucid.
However, you finish the article by claiming that Bayesian analysis can correct for the problem, and this is something that (I don’t think) you even begin to show. Bayesian analysis solves the corner case, but does it bring any traction at all on a typical case?
I think it’s worse than that: Karnofsky’s problem is that he has to compare moderate-mean low-variance estimates to large-mean large-variance estimates, but lukeprog’s solution is for comparing the estimate to the result in cases where the variance is equal across the board.
Put another way, the higher the variance in the true payoffs, the less relevant the curse. This is the flipside of: the more accurate the estimates, the less relevant the curse.
Lukeprog, if I’ve understood you correctly, then this is no good; this is a corner case. The question to be answered here is whether we should expect a “common sense” executive who favors plans with a high prior estimate to do better than a “technical” analyst who favors plans that perform well according to the formal estimation criteria. By assuming that all prior estimates are identical except for bias, this assumption ensures that the technical analyst will win. This, however, begs the question. One could just as easily assume that there is large variation in the true expected values, and that the formal criteria will always produce an estimate of 0, in which case the common sense executive will always win.
Am I missing something? I like the topic; I would enjoy reading about which approach we should expect to perform better in a typical situation.
I think the case where all the choices has a “true expected value” of 0 is picked out merely to illustrate the problem.
Yes.
That’s fine; you’re more than welcome to illustrate the problem, and your analysis does in fact do that. It does it very well; your writing, as always, is very lucid.
However, you finish the article by claiming that Bayesian analysis can correct for the problem, and this is something that (I don’t think) you even begin to show. Bayesian analysis solves the corner case, but does it bring any traction at all on a typical case?
I think it’s worse than that: Karnofsky’s problem is that he has to compare moderate-mean low-variance estimates to large-mean large-variance estimates, but lukeprog’s solution is for comparing the estimate to the result in cases where the variance is equal across the board.
Put another way, the higher the variance in the true payoffs, the less relevant the curse. This is the flipside of: the more accurate the estimates, the less relevant the curse.