I think a lot of the work here is being done by the assumption that the standard deviation of the estimate error depends on (and, moreover, is linear in terms of!) the estimate itself.
Why would we assume this?
Holden suggests that this is reasonable since it keeps the probability of the “right estimate” being zero constant. This seems bizarre to me: no matter how huge my estimate is, there’s a constant chance that it’s completely off? And surely, one would expect the chance of the “correct” estimate being zero to be much higher in cases where your own estimate is close to zero.
The only other motivation I can think of is that we ought to be more suspicious of higher estimates. Hence, if we come up with a high estimate, we should be suitably suspicious and assume our estimate had a high error.
But surely this “suspicion of high estimates” is precisely encoded in our prior; that is, before estimation we believe that a higher value is less likely, and our prior distribution reflects that. But then why are we adding an additional fudge factor in the form of an increased estimate error?
Even if we were convinced that we should assign higher standard deviation to the error on higher estimates, it seems far from obvious that this should be linear in our estimate!
Additionally, I’m not sure that a normal distribution is the correct distribution for healthcare charities, at least. The DCP2 data strongly suggests that the underlying distribution is log-normal. Even though DCP2 has been shown to be a lot less reliable than we’d like (here; some great work by GiveWell, if I may say so!), the sample size is large enough that we wouldn’t expect errors to change the underlying distribution.
I think a lot of the work here is being done by the assumption that the standard deviation of the estimate error depends on (and, moreover, is linear in terms of!) the estimate itself.
Why would we assume this?
Holden suggests that this is reasonable since it keeps the probability of the “right estimate” being zero constant. This seems bizarre to me: no matter how huge my estimate is, there’s a constant chance that it’s completely off? And surely, one would expect the chance of the “correct” estimate being zero to be much higher in cases where your own estimate is close to zero.
The only other motivation I can think of is that we ought to be more suspicious of higher estimates. Hence, if we come up with a high estimate, we should be suitably suspicious and assume our estimate had a high error. But surely this “suspicion of high estimates” is precisely encoded in our prior; that is, before estimation we believe that a higher value is less likely, and our prior distribution reflects that. But then why are we adding an additional fudge factor in the form of an increased estimate error?
Even if we were convinced that we should assign higher standard deviation to the error on higher estimates, it seems far from obvious that this should be linear in our estimate!
Additionally, I’m not sure that a normal distribution is the correct distribution for healthcare charities, at least. The DCP2 data strongly suggests that the underlying distribution is log-normal. Even though DCP2 has been shown to be a lot less reliable than we’d like (here; some great work by GiveWell, if I may say so!), the sample size is large enough that we wouldn’t expect errors to change the underlying distribution.