I think I see different requirements for things working that for things not working.
People getting actively angry about something looks like a (weak) indicator that they are afraid there might be something to it. They do not get angry at 9/11 conspiracists, but laugh about them. They do get angry at the other party.
If you have someone trying to do something and be unsuccessful that tells you even less. I would not see it as indicator of not working. It is just no particular evidence.
But maybe I have the notion of what ‘weak’ means wrong.
If you have someone trying to do something and be unsuccessful that tells you even less. I would not see it as indicator of not working. It is just no particular evidence.
It almost has to be evidence; even if it’s just evidence that the person isn’t doing it right, then you push this case through your prior for how often they do things wrong. Unless your prior is very high, you’re still getting half the impact or more of the evidence. Although you could argue from the selection effect of “correct in your expectations of success” that you’re almost guaranteed to only notice cases where they are doing it wrong, not cases where it doesn’t work (because you don’t expect it to work where it won’t).
A person using the techniques and not having increased success—weak evidence
People hating PUA—trivial evidence
A bunch of people active on a PUA forum having increased success in aggregagate—moderate evidence (would be strong but selection on dependent variable)
I think I see different requirements for things working that for things not working.
People getting actively angry about something looks like a (weak) indicator that they are afraid there might be something to it. They do not get angry at 9/11 conspiracists, but laugh about them. They do get angry at the other party.
If you have someone trying to do something and be unsuccessful that tells you even less. I would not see it as indicator of not working. It is just no particular evidence.
But maybe I have the notion of what ‘weak’ means wrong.
It almost has to be evidence; even if it’s just evidence that the person isn’t doing it right, then you push this case through your prior for how often they do things wrong. Unless your prior is very high, you’re still getting half the impact or more of the evidence. Although you could argue from the selection effect of “correct in your expectations of success” that you’re almost guaranteed to only notice cases where they are doing it wrong, not cases where it doesn’t work (because you don’t expect it to work where it won’t).
My take -
A person using the techniques and not having increased success—weak evidence
People hating PUA—trivial evidence
A bunch of people active on a PUA forum having increased success in aggregagate—moderate evidence (would be strong but selection on dependent variable)