Should you still donate everything to the top scoring charity even if you know that the decision is likely based on noise?
If the charities are this close then you only expect to do very slightly better by giving only to the better scoring one. So it doesn’t matter much whether you give to one, the other, or both.
Ideally, you run your charity-evaluator function on huge selection of charities, and the one for which your charity-evaluator function gives the largest value, is in some sense the best, regardless of the noise.
More practically, imagine an imperfect evaluation function that due to a bug in it’s implementation multiplies by a Very Huge Number value of a charity whose description includes some string S which the evaluation function mis-processes in some dramatic way. Now, if the selection of charities is sufficiently big as to include at least one charity with such S in it’s description, you are essentially donating at random. Or worse than random, because the people that run in their head the computation resulting in production of such S tend to not be the ones you can trust.
Normally, I would expect people who know about human biases to not assume that evaluation would resemble the ideal and to understand that the output of some approximate evaluation will not have the exact properties of expected value.
If the charities are this close then you only expect to do very slightly better by giving only to the better scoring one. So it doesn’t matter much whether you give to one, the other, or both.
Systematic errors are the problem.
Ideally, you run your charity-evaluator function on huge selection of charities, and the one for which your charity-evaluator function gives the largest value, is in some sense the best, regardless of the noise.
More practically, imagine an imperfect evaluation function that due to a bug in it’s implementation multiplies by a Very Huge Number value of a charity whose description includes some string S which the evaluation function mis-processes in some dramatic way. Now, if the selection of charities is sufficiently big as to include at least one charity with such S in it’s description, you are essentially donating at random. Or worse than random, because the people that run in their head the computation resulting in production of such S tend to not be the ones you can trust.
Normally, I would expect people who know about human biases to not assume that evaluation would resemble the ideal and to understand that the output of some approximate evaluation will not have the exact properties of expected value.