That expected value is negative is a state of knowledge. We need careful studies to show whether a technique/medicine/etc is effective precisely because without such a study our state of knowledge shows that the expected value of the technique is negative. At the same time, we expect the new state of knowledge after the study to show that either the technique is useful, or that it’s not.
That’s one of the traps of woo: you often can’t efficiently demonstrate that it’s effective, and through intuition probably related to conservation of expected evidence you insist that if you don’t have a better method to show its effectiveness, the best available method should be enough, because it’s ridiculous to hold the claim to higher standard of proof on one side than on another. But you have to, the prior belief plays its part, the threshold to changing a decision may be too far away to cross by simple arguments. The intuitive thrust of the principle doesn’t carry over to expected utility because of the threshold, it may well be that you have a technique for which there is a potential test that could demonstrate that it’s effective, but the test is unavailable, and without performing the test the expected value of the technique remains negative.
I’m afraid I’m struggling to connect this to your originalobjections. Would you mind clarifying?
ETA: By way of attempting to clarify my issue with your objection, I think the lottery example differs from this situation in two important ways. AFAICT, the uselessness of evidence that a single person has won the lottery is a result of:
the fact that we usually know the odds of winning the lottery are very low, so evidence has little ability to shift our priors; and
that in addition to the evidence of the single winner, we also have evidence of incredibly many losers, so the sum of evidence does not favour a conclusion of profitability.
The analogy is this: using speculative self-help techniques corresponds to playing a lottery, in both cases you expect negative outcome, and in both cases making one more observation, even if it’s observation of success, even if you experience it personally, means very little for the estimation of expected outcome. There is no analogy in lottery for studies that support the efficacy of self-help techniques (or some medicine).
1) the range of conceivably effective self-help techniques is very large relative to the number of actually effective techniques
2) a technique that is negative-expected-value can look positive with small n
3) consequently, using small-n trials on lots of techniques is an inefficient way to look for effective ones, and is itself negative-expected-value, just like looking for the correct lottery number by playing the lottery.
In this analogy, it is the whole self-help space, not the one technique, that is like a lottery.
That expected value is negative is a state of knowledge. We need careful studies to show whether a technique/medicine/etc is effective precisely because without such a study our state of knowledge shows that the expected value of the technique is negative. At the same time, we expect the new state of knowledge after the study to show that either the technique is useful, or that it’s not.
That’s one of the traps of woo: you often can’t efficiently demonstrate that it’s effective, and through intuition probably related to conservation of expected evidence you insist that if you don’t have a better method to show its effectiveness, the best available method should be enough, because it’s ridiculous to hold the claim to higher standard of proof on one side than on another. But you have to, the prior belief plays its part, the threshold to changing a decision may be too far away to cross by simple arguments. The intuitive thrust of the principle doesn’t carry over to expected utility because of the threshold, it may well be that you have a technique for which there is a potential test that could demonstrate that it’s effective, but the test is unavailable, and without performing the test the expected value of the technique remains negative.
I’m afraid I’m struggling to connect this to your original objections. Would you mind clarifying?
ETA: By way of attempting to clarify my issue with your objection, I think the lottery example differs from this situation in two important ways. AFAICT, the uselessness of evidence that a single person has won the lottery is a result of:
the fact that we usually know the odds of winning the lottery are very low, so evidence has little ability to shift our priors; and
that in addition to the evidence of the single winner, we also have evidence of incredibly many losers, so the sum of evidence does not favour a conclusion of profitability.
Neither of these seem to be applicable here.
The analogy is this: using speculative self-help techniques corresponds to playing a lottery, in both cases you expect negative outcome, and in both cases making one more observation, even if it’s observation of success, even if you experience it personally, means very little for the estimation of expected outcome. There is no analogy in lottery for studies that support the efficacy of self-help techniques (or some medicine).
It sounds like you’re saying:
1) the range of conceivably effective self-help techniques is very large relative to the number of actually effective techniques
2) a technique that is negative-expected-value can look positive with small n
3) consequently, using small-n trials on lots of techniques is an inefficient way to look for effective ones, and is itself negative-expected-value, just like looking for the correct lottery number by playing the lottery.
In this analogy, it is the whole self-help space, not the one technique, that is like a lottery.
Am I on the right track?