I am just throwing out a few thoughts here though this example can serve as an excellent case study.
The first is availability and selection bias. It is only the students that actually get an apparent benefit that come to your mind as examples and it is only the students that actually get a benefit from learning that will continue to do so. Why learn even more when through testing you see that there is virtually no benefit?
Another effect is that repeatedly testing on the SAT may be the only part of the intervention actually having an effect. This might be through gaining confidence in test taking and/or reducing test anxiety. Also there might be some weird psychological effect going on where people subconsciously do worse until they receive tutoring to justify the expense.
And a last thing is status quo bias. For any piece of evidence that says that conventional wisdom is wrong, people will tend to disregard the evidence and this shows as an apparent chasm. That is why I personally tend to dislike the sentence “The burden of proof is on you.”.
I like your points, but it does seem awfully surprising that there would not be an improvement on, for example, the reading section which has a number of questions that are little more than vocabulary tests. Vocabulary is easy to study and if you don’t know it you have little chance of figuring it out.
Or for the math section, someone who hasn’t taken geometry before will do very poorly on any geometry questions on the test. (I think this is an unobjectionable claim.) So it would be surprising if people who studied geometry at all suddenly get all the possible benefit of studying—studying doesn’t seem like it should be a binary thing where you hop from no knowledge/poor performance to full knowledge/as-good-as-you-could-get performance.
Note that the two articles cited by the OP’s link are not randomized controlled trials and are both actually based off the same survey data.
I, too, join the OP in confusion and mild skepticism of the research.
As I am not a citizen of the US, I have no idea what the exact makeup is of the SAT, so I have to keep guessing. I take the contrary position for the sake of argument.
As far as the SAT measures knowledge it would be very surprising to see no effect of learning. What we are examining though is the effect of additional test preparation beyond the usual curriculum. So I would argue that the marginal benefit of additional preparation is extremely low as all the necessary knowledge is conferred by the usual curriculum—especially by the students considering taking the SAT and taking additional care to ace them.
I seem to recall that SAT and g are highly correlated. Insofar as the SAT is a g-heavy IQ test I am not at all surprised that additional preparation confers virtually no advantage, just as we know of no way to reliably increase g. Insofar the SAT measures ability like geometry it would be surprising if learning those subjects gives no benefit as long as the required knowledge goes beyond what is already taught in high school.
There is only a moderate correlation between income and taking of test prep
Under-performing minorities are more likely to take test prep than whites
In other words, quite a few people taking test prep are ones likely to be going to poor, under-performing school systems. Either test prep companies are incompetent or our school system is doing a lot better than I had expected, even on the low end!
I am just throwing out a few thoughts here though this example can serve as an excellent case study.
The first is availability and selection bias. It is only the students that actually get an apparent benefit that come to your mind as examples and it is only the students that actually get a benefit from learning that will continue to do so. Why learn even more when through testing you see that there is virtually no benefit?
Another effect is that repeatedly testing on the SAT may be the only part of the intervention actually having an effect. This might be through gaining confidence in test taking and/or reducing test anxiety. Also there might be some weird psychological effect going on where people subconsciously do worse until they receive tutoring to justify the expense.
And a last thing is status quo bias. For any piece of evidence that says that conventional wisdom is wrong, people will tend to disregard the evidence and this shows as an apparent chasm. That is why I personally tend to dislike the sentence “The burden of proof is on you.”.
I like your points, but it does seem awfully surprising that there would not be an improvement on, for example, the reading section which has a number of questions that are little more than vocabulary tests. Vocabulary is easy to study and if you don’t know it you have little chance of figuring it out.
Or for the math section, someone who hasn’t taken geometry before will do very poorly on any geometry questions on the test. (I think this is an unobjectionable claim.) So it would be surprising if people who studied geometry at all suddenly get all the possible benefit of studying—studying doesn’t seem like it should be a binary thing where you hop from no knowledge/poor performance to full knowledge/as-good-as-you-could-get performance.
Note that the two articles cited by the OP’s link are not randomized controlled trials and are both actually based off the same survey data.
I, too, join the OP in confusion and mild skepticism of the research.
As I am not a citizen of the US, I have no idea what the exact makeup is of the SAT, so I have to keep guessing. I take the contrary position for the sake of argument.
As far as the SAT measures knowledge it would be very surprising to see no effect of learning. What we are examining though is the effect of additional test preparation beyond the usual curriculum. So I would argue that the marginal benefit of additional preparation is extremely low as all the necessary knowledge is conferred by the usual curriculum—especially by the students considering taking the SAT and taking additional care to ace them.
I seem to recall that SAT and g are highly correlated. Insofar as the SAT is a g-heavy IQ test I am not at all surprised that additional preparation confers virtually no advantage, just as we know of no way to reliably increase g. Insofar the SAT measures ability like geometry it would be surprising if learning those subjects gives no benefit as long as the required knowledge goes beyond what is already taught in high school.
While a good point, the OP’s link says that:
There is only a moderate correlation between income and taking of test prep
Under-performing minorities are more likely to take test prep than whites
In other words, quite a few people taking test prep are ones likely to be going to poor, under-performing school systems. Either test prep companies are incompetent or our school system is doing a lot better than I had expected, even on the low end!