One class of variance in cognitive test results is probably, effectively, pseudorandomness.
Suppose there’s a problem, and there are five plausible solutions you might try, two of which will work. Then your performance is effectively determined by the order in which you end up trying solutions. And if your skills and knowledge don’t give you a strong reason to prefer any of them, then it’ll presumably be determined in a pseudorandom way: whichever comes to mind first. Maybe being cold subconsciously reminds you of when you were thinking about stuff connected to Solution B, or discourages you from thinking about Solution C. Thus, you could get a reliably reproducible result that temperature affects your performance on a given test, even if it has no “real” effect on how well your mind works and wouldn’t generalize to other tests.
This should be addressable by simply taking more, different, cognitive tests to confirm any effect you think you’ve found.
I think the proposed method could still work though. A substantial fraction of the pseudorandomness may be consistent on the individual person level.
The type of pseudorandomness you describe here ought to be independent at the level of individual items, so it ought to be part of the least-reliable variance component (not part of the general trait measured and not stable over time). It’s possible to use statistics to estimate how big an effect it has on the scores, and it’s possible to drive it arbitrarily far down in effect simply by making the test longer.
One class of variance in cognitive test results is probably, effectively, pseudorandomness.
Suppose there’s a problem, and there are five plausible solutions you might try, two of which will work. Then your performance is effectively determined by the order in which you end up trying solutions. And if your skills and knowledge don’t give you a strong reason to prefer any of them, then it’ll presumably be determined in a pseudorandom way: whichever comes to mind first. Maybe being cold subconsciously reminds you of when you were thinking about stuff connected to Solution B, or discourages you from thinking about Solution C. Thus, you could get a reliably reproducible result that temperature affects your performance on a given test, even if it has no “real” effect on how well your mind works and wouldn’t generalize to other tests.
This should be addressable by simply taking more, different, cognitive tests to confirm any effect you think you’ve found.
I think the proposed method could still work though. A substantial fraction of the pseudorandomness may be consistent on the individual person level.
The type of pseudorandomness you describe here ought to be independent at the level of individual items, so it ought to be part of the least-reliable variance component (not part of the general trait measured and not stable over time). It’s possible to use statistics to estimate how big an effect it has on the scores, and it’s possible to drive it arbitrarily far down in effect simply by making the test longer.