I’m afraid I’m struggling to connect this to your originalobjections. Would you mind clarifying?
ETA: By way of attempting to clarify my issue with your objection, I think the lottery example differs from this situation in two important ways. AFAICT, the uselessness of evidence that a single person has won the lottery is a result of:
the fact that we usually know the odds of winning the lottery are very low, so evidence has little ability to shift our priors; and
that in addition to the evidence of the single winner, we also have evidence of incredibly many losers, so the sum of evidence does not favour a conclusion of profitability.
The analogy is this: using speculative self-help techniques corresponds to playing a lottery, in both cases you expect negative outcome, and in both cases making one more observation, even if it’s observation of success, even if you experience it personally, means very little for the estimation of expected outcome. There is no analogy in lottery for studies that support the efficacy of self-help techniques (or some medicine).
1) the range of conceivably effective self-help techniques is very large relative to the number of actually effective techniques
2) a technique that is negative-expected-value can look positive with small n
3) consequently, using small-n trials on lots of techniques is an inefficient way to look for effective ones, and is itself negative-expected-value, just like looking for the correct lottery number by playing the lottery.
In this analogy, it is the whole self-help space, not the one technique, that is like a lottery.
I’m afraid I’m struggling to connect this to your original objections. Would you mind clarifying?
ETA: By way of attempting to clarify my issue with your objection, I think the lottery example differs from this situation in two important ways. AFAICT, the uselessness of evidence that a single person has won the lottery is a result of:
the fact that we usually know the odds of winning the lottery are very low, so evidence has little ability to shift our priors; and
that in addition to the evidence of the single winner, we also have evidence of incredibly many losers, so the sum of evidence does not favour a conclusion of profitability.
Neither of these seem to be applicable here.
The analogy is this: using speculative self-help techniques corresponds to playing a lottery, in both cases you expect negative outcome, and in both cases making one more observation, even if it’s observation of success, even if you experience it personally, means very little for the estimation of expected outcome. There is no analogy in lottery for studies that support the efficacy of self-help techniques (or some medicine).
It sounds like you’re saying:
1) the range of conceivably effective self-help techniques is very large relative to the number of actually effective techniques
2) a technique that is negative-expected-value can look positive with small n
3) consequently, using small-n trials on lots of techniques is an inefficient way to look for effective ones, and is itself negative-expected-value, just like looking for the correct lottery number by playing the lottery.
In this analogy, it is the whole self-help space, not the one technique, that is like a lottery.
Am I on the right track?