Basically, we have an almost religious reverence for high-powered decent-effect-size low-p-value statistical evidence, and we fail notice when these experiments acquire their bayes factors because they measure something incredibly narrow and is therefore unlikely to generalise to whatever gears-level models you’re entertaining.
It has the same problem as deferring to experts. The communication suffers a bandwidth problem[1], and we frequently delude ourselves into believing we have adopted their models as long as we copy their probabilities on a very slim sample of queries they’ve answered.
I know of no good write-up of the bandwidth problem in social epistemology but Owen Cotton-Barrat talks about it here (my comment) and Dennett refers to it as the “Daddy Is a Doctor” phenomenon.
To the contrary, johnswentworth’s point is not that the experiments have low external validity but that they have low internal validity. It’s that there are confounds.
Ironically, one of my quibbles with the post is that the verbiage implies measurement error is the problem. Not measuring what you think you’re measuring is about content validity, but the post is actually about how omitted variables (i.e., confounders) are a problem for inferences. “You are not Complaining About What You Think You Are Complaining About.”
Basically, we have an almost religious reverence for high-powered decent-effect-size low-p-value statistical evidence, and we fail notice when these experiments acquire their bayes factors because they measure something incredibly narrow and is therefore unlikely to generalise to whatever gears-level models you’re entertaining.
It has the same problem as deferring to experts. The communication suffers a bandwidth problem[1], and we frequently delude ourselves into believing we have adopted their models as long as we copy their probabilities on a very slim sample of queries they’ve answered.
I know of no good write-up of the bandwidth problem in social epistemology but Owen Cotton-Barrat talks about it here (my comment) and Dennett refers to it as the “Daddy Is a Doctor” phenomenon.
To the contrary, johnswentworth’s point is not that the experiments have low external validity but that they have low internal validity. It’s that there are confounds.
Ironically, one of my quibbles with the post is that the verbiage implies measurement error is the problem. Not measuring what you think you’re measuring is about content validity, but the post is actually about how omitted variables (i.e., confounders) are a problem for inferences. “You are not Complaining About What You Think You Are Complaining About.”