what sort of p-values would you need to see in that paper in order to believe with, say, 50% probability that the effect measured is real?
A lot more than one study. But the other issue is, if the measured effect is real, the correct response is, “I see that I am confused,” not, “They must have psychic powers!” Unexpected stuff happening in an experiment is a rather more probable prior than people having psychic powers. Moreover, you’d need very precise experiments to figure out exactly what’s going on. This experiment may in fact show more about the invalidity of priming—if priming works in either direction, and people don’t have psychic powers, then something other than the priming is probably responsible for their reactions.
It is a very, very common error in psychology to take limited experimental evidence with “statistical significance” and then infer extremely complicated and precise notions of human psychology from those, despite the fact that such precise explanations would require vastly more evidence. (A hypothetical example would be: Men respond to red letters at a faster rate than women, as compared with blue letters. Therefore, the color red must have had some specific importance in hunting during our evolutionary past. The latter observation is an arbitrarily privileged hypothesis with no real evidenciary support over thousands of other hypotheses. I’ve seen more absurd jumps, as well as people simply ignoring evidence that did not conform to their prior theory.)
A lot more than one study. But the other issue is, if the measured effect is real, the correct response is, “I see that I am confused,” not, “They must have psychic powers!” Unexpected stuff happening in an experiment is a rather more probable prior than people having psychic powers. Moreover, you’d need very precise experiments to figure out exactly what’s going on. This experiment may in fact show more about the invalidity of priming—if priming works in either direction, and people don’t have psychic powers, then something other than the priming is probably responsible for their reactions.
It is a very, very common error in psychology to take limited experimental evidence with “statistical significance” and then infer extremely complicated and precise notions of human psychology from those, despite the fact that such precise explanations would require vastly more evidence. (A hypothetical example would be: Men respond to red letters at a faster rate than women, as compared with blue letters. Therefore, the color red must have had some specific importance in hunting during our evolutionary past. The latter observation is an arbitrarily privileged hypothesis with no real evidenciary support over thousands of other hypotheses. I’ve seen more absurd jumps, as well as people simply ignoring evidence that did not conform to their prior theory.)
Voted up because I agree that there’s way too much inference from very limited psychological experiments.