What evidence? PUAs may use a lot of trial and error in developing their techniques, but do their tests count as valid experimental evidence, or just anecdotes? Where are their control groups? What is their null hypothesis? Was subject selection randomized? Were the data gathered and analyzed by independent parties?
Would you accept this kind of evidence if we were talking about physics? Would you accept this kind of evidence if we were evaluating someone who claimed to have psychic powers?
One of the reasons this topic is of interest to rationalists is that it is an example of an area where rational evidence is available but scientific evidence is in short supply. It is not in general rational to postpone judgment until scientific evidence is available. Learning how to make maximal use of rational evidence without succumbing to the pitfalls of cognitive biases is a topic of much interest to many LWers.
Yes, that’s true. I’ve been phrasing my more recent comments in terms of scientific evidence, because several people I’ve been butting heads with have made assertions about PUA that seemed to imply it had a scientific-level base of supporting evidence.
I’m still not sure though what the rational evidence is that I’m supposed to be updating on. Numerous other self improvement programs make similar claims, based on similar reasoning, and offer similar anecdotal evidence. So I consider such evidence to be equally likely to appear regardless of whether PUA’s claims are true or false, leaving me with nothing but my priors.
What evidence? PUAs may use a lot of trial and error in developing their techniques, but do their tests count as valid experimental evidence, or just anecdotes? Where are their control groups? What is their null hypothesis? Was subject selection randomized? Were the data gathered and analyzed by independent parties?
Well, as I said, if you study the discourse in the PUA community at its best in a non-biased and detached way, desensitized to the language and attitudes you might find instinctively off-putting, you’ll actually find the epistemological standards surprisingly high. But you just have to see that for yourself.
A good comparison for the PUA milieu would be a high-quality community of hobbyist amateurs who engage in some technical work with passion and enthusiasm. In their discussions, they probably won’t apply the same formal standards of discourse and evidence that are used in academic research and corporate R&D, but it’s nevertheless likely that they know what they’re talking about and their body of established knowledge is as reliable as any other—and even though there are no formal qualifications for joining, those bringing bullshit rather than insight will soon be identified and ostracized.
Now, if you don’t know at first sight whether you’re dealing with such an epistemologically healthy community, the first test would be to see how its main body of established knowledge conforms to your own experiences and observations. (In a non-biased way, of course, which is harder when it comes to the PUA stuff than some ordinary technical skill.) In my case, and not just mine, the result was a definite pass. The further test is to observe the actual manner of discourse practiced and its epistemological quality. Again, it’s harder to do when biased reactions to various signals of disrespectability are standing in the way.
Would you accept this kind of evidence if we were talking about physics?
Even in physics, not all evidence comes from reproducible experiments. Sometimes you just have to make the best out of observations gathered at random opportune moments, for example when it comes to unusual astronomical or geophysical events.
Would you accept this kind of evidence if we were evaluating someone who claimed to have psychic powers?
You’re biasing your skepticism way upward now. The correct level of initial skepticism with which to meet the PUA stuff is the skepticism you apply to people claiming to have solved difficult problems in a way consistent with the existing well-established scientific knowledge—not the much higher level appropriate for those whose claims contradict it.
The correct level of initial skepticism with which to meet the PUA stuff is the skepticism you apply to people claiming to have solved difficult problems in a way consistent with the existing well-established scientific knowledge—not the much higher level appropriate for those whose claims contradict it.
That’s a good point—the priors for PUA, though low, are nowhere near as low as for psychic phenomena. But that just means that you need a smaller amount of evidence to overcome those priors—it doesn’t lower the bar for what qualifies as valid evidence.
What evidence? PUAs may use a lot of trial and error in developing their techniques, but do their tests count as valid experimental evidence, or just anecdotes? Where are their control groups? What is their null hypothesis? Was subject selection randomized? Were the data gathered and analyzed by independent parties?
Would you accept this kind of evidence if we were talking about physics? Would you accept this kind of evidence if we were evaluating someone who claimed to have psychic powers?
One of the reasons this topic is of interest to rationalists is that it is an example of an area where rational evidence is available but scientific evidence is in short supply. It is not in general rational to postpone judgment until scientific evidence is available. Learning how to make maximal use of rational evidence without succumbing to the pitfalls of cognitive biases is a topic of much interest to many LWers.
Yes, that’s true. I’ve been phrasing my more recent comments in terms of scientific evidence, because several people I’ve been butting heads with have made assertions about PUA that seemed to imply it had a scientific-level base of supporting evidence.
I’m still not sure though what the rational evidence is that I’m supposed to be updating on. Numerous other self improvement programs make similar claims, based on similar reasoning, and offer similar anecdotal evidence. So I consider such evidence to be equally likely to appear regardless of whether PUA’s claims are true or false, leaving me with nothing but my priors.
kodos96:
Well, as I said, if you study the discourse in the PUA community at its best in a non-biased and detached way, desensitized to the language and attitudes you might find instinctively off-putting, you’ll actually find the epistemological standards surprisingly high. But you just have to see that for yourself.
A good comparison for the PUA milieu would be a high-quality community of hobbyist amateurs who engage in some technical work with passion and enthusiasm. In their discussions, they probably won’t apply the same formal standards of discourse and evidence that are used in academic research and corporate R&D, but it’s nevertheless likely that they know what they’re talking about and their body of established knowledge is as reliable as any other—and even though there are no formal qualifications for joining, those bringing bullshit rather than insight will soon be identified and ostracized.
Now, if you don’t know at first sight whether you’re dealing with such an epistemologically healthy community, the first test would be to see how its main body of established knowledge conforms to your own experiences and observations. (In a non-biased way, of course, which is harder when it comes to the PUA stuff than some ordinary technical skill.) In my case, and not just mine, the result was a definite pass. The further test is to observe the actual manner of discourse practiced and its epistemological quality. Again, it’s harder to do when biased reactions to various signals of disrespectability are standing in the way.
Even in physics, not all evidence comes from reproducible experiments. Sometimes you just have to make the best out of observations gathered at random opportune moments, for example when it comes to unusual astronomical or geophysical events.
You’re biasing your skepticism way upward now. The correct level of initial skepticism with which to meet the PUA stuff is the skepticism you apply to people claiming to have solved difficult problems in a way consistent with the existing well-established scientific knowledge—not the much higher level appropriate for those whose claims contradict it.
That’s a good point—the priors for PUA, though low, are nowhere near as low as for psychic phenomena. But that just means that you need a smaller amount of evidence to overcome those priors—it doesn’t lower the bar for what qualifies as valid evidence.