but there’s a 50% chance the test detects the difference and a 50% chance it doesn’t
Rationality is not a binary variable, but continuous. It is NOT the case that the test has a chance of detecting something or nothing: the test will output a value on some scale. If the test is not powerful enough to detect the difference, it will show up as the difference being not statistically significant—the difference will be swamped by noise, but not just fully appear or fully disappear in any given instance.
You’ll either get a lot of results that average 20% less or a lot of results that aren’t less at all
Nope—that would only be true if rationality were a boolean variable. It is not.
That doesn’t follow. For instance, imagine that one group is irrational because their brains freeze up at any problem that contains the number 8, and some tests contain the number 8 and some don’t. They’ll fail the former tests, but be indistinguishable from the first group on the latter tests.
I can imagine a lot of things that have no relationship to reality.
In any case, you were talking about a test that has a 50% chance of detecting the difference, presumably returning either 0% or 20% but never 10%. Your example does not address this case—it’s about different tests producing different results.
You were responding to Stefan. As such, it doesn’t matter whether you can imagine a test that works that way; it matters whether his uncertainty over whether the test works includes the possibility of it working that way.
Your example does not address this case—it’s about different tests producing different results.
If you don’t actually know that they freeze up at the sight of the number 8, and you are 50% likely to produce a test that contains the number 8, then the test has a 50% chance of working, by your own reasoning—actually, it has a 0% or 100% chance of working, but since you are uncertain about whether it works, you can fold the uncertainty into your estimate of how good the test is and claim 50%.
Rationality is not a binary variable, but continuous. It is NOT the case that the test has a chance of detecting something or nothing: the test will output a value on some scale. If the test is not powerful enough to detect the difference, it will show up as the difference being not statistically significant—the difference will be swamped by noise, but not just fully appear or fully disappear in any given instance.
Nope—that would only be true if rationality were a boolean variable. It is not.
That doesn’t follow. For instance, imagine that one group is irrational because their brains freeze up at any problem that contains the number 8, and some tests contain the number 8 and some don’t. They’ll fail the former tests, but be indistinguishable from the first group on the latter tests.
I can imagine a lot of things that have no relationship to reality.
In any case, you were talking about a test that has a 50% chance of detecting the difference, presumably returning either 0% or 20% but never 10%. Your example does not address this case—it’s about different tests producing different results.
You were responding to Stefan. As such, it doesn’t matter whether you can imagine a test that works that way; it matters whether his uncertainty over whether the test works includes the possibility of it working that way.
If you don’t actually know that they freeze up at the sight of the number 8, and you are 50% likely to produce a test that contains the number 8, then the test has a 50% chance of working, by your own reasoning—actually, it has a 0% or 100% chance of working, but since you are uncertain about whether it works, you can fold the uncertainty into your estimate of how good the test is and claim 50%.