Hello. I’ve been a lurker here for quite some time now, but this is the first time I am making an appearance. I would like to consult everyone here regarding what I perceive to be irrationality on my part. I hope that you will be patient towards me and refrain from downvoting out of irritation, as I would prefer not to have my comment hidden, since that would greatly reduce my chances of getting feedback.
The issue is this: while I am fully aware that anecdotes do not constitute data, I have a very difficult time believing that test preparation only has a modest positive effect (if any) on SAT scores, even though this has been noted by several studies. Such a finding is completely incongruent with my personal experiences growing up in an East Asian country where most students (regardless of their socioeconomic status) attend cram schools or hire after-school private tutors—many of these students have managed to perform much better than they otherwise would have, due to the extra lessons and revision. (They usually go through several iterations of testing themselves using old SAT papers before sitting for the actual test, and there are often very significant gains—sometimes as high as several hundred points—in their performances.)
So, my request to my fellow LW readers is this: Please help me reconcile this jarring chasm between research findings and my own personal experiences. There might be some information I am missing that is contributing to my inability to believe the findings. Is there anyone here who has done extensive reading on this topic?
I would bet that the gains from practicing the SATs are relatively high for people who have strong verbal abilities but are not proficient in English.
Also, since the SATs are optimized for U.S. students it might be the case that there are question types which U.S. students have seen thousands of times and so because of diminishing marginal returns would gain little by additional study, yet a Chinese student might never have encountered a similar problem and so would massively gain from practicing it.
SAT is very g-loaded, so it would be susceptible to the same practice effect that IQ tests in general are susceptible to. If you look at SAT/IQ tables, the 20-40 point increase Tyler cites corresponds to about 1.5-3 IQ points. This is consistent with the typical magnitude (< 5 points) of the practice effects on IQ test scores. Your “hundreds of points” are wildly inconsistent with this. The only way I could see that happen is if quite a bit of the SAT would test for skills that can be practiced but don’t correlate with g. Not very likely.
The way to reconcile your experience with the evidence is to note that the score on a low-stakes practice test is just not comparable to the score on the real thing (with or without test prep). It’s not that implausible to believe that, say, 10% of people (more than enough to account for your anecdotes) will score at least 100 points less on an early practice test than they could score on the real thing at the same level of preparation. It’s hard to trick your brain into believing that something is high-stakes when it isn’t.
ETA: On reflection, the low-stakes hypothesis probably doesn’t account for too much of the puzzle. In particular, it doesn’t explain any gain between consecutive low-stakes practice tests. I think James Miller’s explanation takes the cake. The SAT-g correlation is likely a lot lower for a population not proficient in English.
The only way I could see that happen is if quite a bit of the SAT would test for skills that can be practiced but don’t correlate with g. Not very likely.
Not likely?? It’s certain!
If you know the scoring rules and their implications like when to guess and when to leave it blank, that can get you points you might miss from leaving it blank and reduce your penalty on things you’d have gotten wrong.
If you know better how to manage your time, then you won’t end up rushed.
Simply having done it before reduces the stress of the situation and can enable better focus.
Being familiar with the style of questions asked will help a lot—you’ll know to expect certain odd phrasings that can trip up a naive test-taker, and in some cases you will barely need to parse, simply pattern-match. ‘Yup, this is that kind of question.’
And that’s setting aside just studying the words they’re likely to ask you about.
None of these have all that much to do with g, and I can see them producing a swing of 40 points easily, perhaps more at the lower end (you know, in the case where there are hundreds of points to gain).
This isn’t to say that intense SAT prep is a huge difference on average—it could end up inducing more freakout instead of less, it could induce someone to stay up late and not be rested, it may primarily be used by those who would already do well, it may be used as a crutch by those wouldn’t… all sorts of confounding things. But the idea that there is no significant component of the SAT that’s practicable non-g is hard to believe.
You are right that a test being g-loaded is not inconsistent with test takers receiving significant gains from repeatedly taking this test. This is why formal IQ tests used by experts are not available to the general public because practicing them can significantly raise scores and so give an inaccurate estimate of IQ.
The way to reconcile your experience with the evidence is to note that the score on a low-stakes practice test is just not comparable to the score on the real thing (with or without test prep).
This is one possibility. Thanks for bringing it up.
I am just throwing out a few thoughts here though this example can serve as an excellent case study.
The first is availability and selection bias. It is only the students that actually get an apparent benefit that come to your mind as examples and it is only the students that actually get a benefit from learning that will continue to do so. Why learn even more when through testing you see that there is virtually no benefit?
Another effect is that repeatedly testing on the SAT may be the only part of the intervention actually having an effect. This might be through gaining confidence in test taking and/or reducing test anxiety. Also there might be some weird psychological effect going on where people subconsciously do worse until they receive tutoring to justify the expense.
And a last thing is status quo bias. For any piece of evidence that says that conventional wisdom is wrong, people will tend to disregard the evidence and this shows as an apparent chasm. That is why I personally tend to dislike the sentence “The burden of proof is on you.”.
I like your points, but it does seem awfully surprising that there would not be an improvement on, for example, the reading section which has a number of questions that are little more than vocabulary tests. Vocabulary is easy to study and if you don’t know it you have little chance of figuring it out.
Or for the math section, someone who hasn’t taken geometry before will do very poorly on any geometry questions on the test. (I think this is an unobjectionable claim.) So it would be surprising if people who studied geometry at all suddenly get all the possible benefit of studying—studying doesn’t seem like it should be a binary thing where you hop from no knowledge/poor performance to full knowledge/as-good-as-you-could-get performance.
Note that the two articles cited by the OP’s link are not randomized controlled trials and are both actually based off the same survey data.
I, too, join the OP in confusion and mild skepticism of the research.
As I am not a citizen of the US, I have no idea what the exact makeup is of the SAT, so I have to keep guessing. I take the contrary position for the sake of argument.
As far as the SAT measures knowledge it would be very surprising to see no effect of learning. What we are examining though is the effect of additional test preparation beyond the usual curriculum. So I would argue that the marginal benefit of additional preparation is extremely low as all the necessary knowledge is conferred by the usual curriculum—especially by the students considering taking the SAT and taking additional care to ace them.
I seem to recall that SAT and g are highly correlated. Insofar as the SAT is a g-heavy IQ test I am not at all surprised that additional preparation confers virtually no advantage, just as we know of no way to reliably increase g. Insofar the SAT measures ability like geometry it would be surprising if learning those subjects gives no benefit as long as the required knowledge goes beyond what is already taught in high school.
There is only a moderate correlation between income and taking of test prep
Under-performing minorities are more likely to take test prep than whites
In other words, quite a few people taking test prep are ones likely to be going to poor, under-performing school systems. Either test prep companies are incompetent or our school system is doing a lot better than I had expected, even on the low end!
Where do they get the old test papers? In America, it is common for the cram schools to supply “old test papers” that are actually considerably more difficult.
There are publishers here that publish collections of old test papers (along with the solutions) that were administered over the past 10 years, and many students practise by testing themselves using those.
This paper has some interesting information and proposes the idea that the tests are designed to be resistant to short-term test prep:
A principal justification for the use of the SAT I and ACT in the admissions process is that such tests are designed to be insensitive to the high school curriculum and to short- term test preparation. If short term preparatory activities prior to taking the SAT I or ACT can have the effect of significantly boosting the scores of students above those they would have received without the preparation, both the validity and reliability of the tests as indicators of college readiness might be called into question.
It also alludes to there being multiple studies that have failed to find large effect sizes from test-prep courses. However, I’m not sure I quite believe that test prep is that ineffective. I never did test-prep for the SAT because I got a good enough score my first time to get into the university I wanted, but I did use test-prep books for AP tests, a few times for classes that I didn’t even take, and I got 5⁄5 on most of those. I’m sure those were intended to be “cram-resistant” too, but they clearly weren’t.
It’s possible that the confusion here on LW comes from the systemic bias of the majority of people here being of above-average intelligence, though I don’t know what the mechanism for that would be. I’ll just add that “I notice that I am confused” as well, so something that we believe must be false...
My vague recollection of the SAT and ACT was that they were designed to test your ability to perform certain cognitive tasks, but didn’t test whether you had any particular knowledge. There may have been exceptions on grammar questions, but I think that in the math section, you were given any formulas you might need—whereas AP tests test knowledge extensively.
There may be a point at which you’ve taken so many test SAT/ACTs that, for lack of a better analogy, you’ve strengthened those cognitive muscles enough for it to make a difference, but cramming seems like an obviously ineffective way to handle this. AP tests, on the other hand, have a limited scope of facts for you to produce on command.
There was definitely a vocabulary component to the version of the SAT that I took (in the late Nineties). I seem to recall hearing that a later version of the test had dropped that, though, probably over cultural bias concerns.
Hello. I’ve been a lurker here for quite some time now, but this is the first time I am making an appearance. I would like to consult everyone here regarding what I perceive to be irrationality on my part. I hope that you will be patient towards me and refrain from downvoting out of irritation, as I would prefer not to have my comment hidden, since that would greatly reduce my chances of getting feedback.
The issue is this: while I am fully aware that anecdotes do not constitute data, I have a very difficult time believing that test preparation only has a modest positive effect (if any) on SAT scores, even though this has been noted by several studies. Such a finding is completely incongruent with my personal experiences growing up in an East Asian country where most students (regardless of their socioeconomic status) attend cram schools or hire after-school private tutors—many of these students have managed to perform much better than they otherwise would have, due to the extra lessons and revision. (They usually go through several iterations of testing themselves using old SAT papers before sitting for the actual test, and there are often very significant gains—sometimes as high as several hundred points—in their performances.)
So, my request to my fellow LW readers is this: Please help me reconcile this jarring chasm between research findings and my own personal experiences. There might be some information I am missing that is contributing to my inability to believe the findings. Is there anyone here who has done extensive reading on this topic?
I would bet that the gains from practicing the SATs are relatively high for people who have strong verbal abilities but are not proficient in English.
Also, since the SATs are optimized for U.S. students it might be the case that there are question types which U.S. students have seen thousands of times and so because of diminishing marginal returns would gain little by additional study, yet a Chinese student might never have encountered a similar problem and so would massively gain from practicing it.
SAT is very g-loaded, so it would be susceptible to the same practice effect that IQ tests in general are susceptible to. If you look at SAT/IQ tables, the 20-40 point increase Tyler cites corresponds to about 1.5-3 IQ points. This is consistent with the typical magnitude (< 5 points) of the practice effects on IQ test scores. Your “hundreds of points” are wildly inconsistent with this. The only way I could see that happen is if quite a bit of the SAT would test for skills that can be practiced but don’t correlate with g. Not very likely.
The way to reconcile your experience with the evidence is to note that the score on a low-stakes practice test is just not comparable to the score on the real thing (with or without test prep). It’s not that implausible to believe that, say, 10% of people (more than enough to account for your anecdotes) will score at least 100 points less on an early practice test than they could score on the real thing at the same level of preparation. It’s hard to trick your brain into believing that something is high-stakes when it isn’t.
ETA: On reflection, the low-stakes hypothesis probably doesn’t account for too much of the puzzle. In particular, it doesn’t explain any gain between consecutive low-stakes practice tests. I think James Miller’s explanation takes the cake. The SAT-g correlation is likely a lot lower for a population not proficient in English.
Not likely?? It’s certain!
If you know the scoring rules and their implications like when to guess and when to leave it blank, that can get you points you might miss from leaving it blank and reduce your penalty on things you’d have gotten wrong.
If you know better how to manage your time, then you won’t end up rushed.
Simply having done it before reduces the stress of the situation and can enable better focus.
Being familiar with the style of questions asked will help a lot—you’ll know to expect certain odd phrasings that can trip up a naive test-taker, and in some cases you will barely need to parse, simply pattern-match. ‘Yup, this is that kind of question.’
And that’s setting aside just studying the words they’re likely to ask you about.
None of these have all that much to do with g, and I can see them producing a swing of 40 points easily, perhaps more at the lower end (you know, in the case where there are hundreds of points to gain).
This isn’t to say that intense SAT prep is a huge difference on average—it could end up inducing more freakout instead of less, it could induce someone to stay up late and not be rested, it may primarily be used by those who would already do well, it may be used as a crutch by those wouldn’t… all sorts of confounding things. But the idea that there is no significant component of the SAT that’s practicable non-g is hard to believe.
This comment is very insightful—you managed to articulate a lot of non-g factors that would explain my own observations. Thank you.
You are right that a test being g-loaded is not inconsistent with test takers receiving significant gains from repeatedly taking this test. This is why formal IQ tests used by experts are not available to the general public because practicing them can significantly raise scores and so give an inaccurate estimate of IQ.
Edited because I misread the above comment.
This is one possibility. Thanks for bringing it up.
I am just throwing out a few thoughts here though this example can serve as an excellent case study.
The first is availability and selection bias. It is only the students that actually get an apparent benefit that come to your mind as examples and it is only the students that actually get a benefit from learning that will continue to do so. Why learn even more when through testing you see that there is virtually no benefit?
Another effect is that repeatedly testing on the SAT may be the only part of the intervention actually having an effect. This might be through gaining confidence in test taking and/or reducing test anxiety. Also there might be some weird psychological effect going on where people subconsciously do worse until they receive tutoring to justify the expense.
And a last thing is status quo bias. For any piece of evidence that says that conventional wisdom is wrong, people will tend to disregard the evidence and this shows as an apparent chasm. That is why I personally tend to dislike the sentence “The burden of proof is on you.”.
I like your points, but it does seem awfully surprising that there would not be an improvement on, for example, the reading section which has a number of questions that are little more than vocabulary tests. Vocabulary is easy to study and if you don’t know it you have little chance of figuring it out.
Or for the math section, someone who hasn’t taken geometry before will do very poorly on any geometry questions on the test. (I think this is an unobjectionable claim.) So it would be surprising if people who studied geometry at all suddenly get all the possible benefit of studying—studying doesn’t seem like it should be a binary thing where you hop from no knowledge/poor performance to full knowledge/as-good-as-you-could-get performance.
Note that the two articles cited by the OP’s link are not randomized controlled trials and are both actually based off the same survey data.
I, too, join the OP in confusion and mild skepticism of the research.
As I am not a citizen of the US, I have no idea what the exact makeup is of the SAT, so I have to keep guessing. I take the contrary position for the sake of argument.
As far as the SAT measures knowledge it would be very surprising to see no effect of learning. What we are examining though is the effect of additional test preparation beyond the usual curriculum. So I would argue that the marginal benefit of additional preparation is extremely low as all the necessary knowledge is conferred by the usual curriculum—especially by the students considering taking the SAT and taking additional care to ace them.
I seem to recall that SAT and g are highly correlated. Insofar as the SAT is a g-heavy IQ test I am not at all surprised that additional preparation confers virtually no advantage, just as we know of no way to reliably increase g. Insofar the SAT measures ability like geometry it would be surprising if learning those subjects gives no benefit as long as the required knowledge goes beyond what is already taught in high school.
While a good point, the OP’s link says that:
There is only a moderate correlation between income and taking of test prep
Under-performing minorities are more likely to take test prep than whites
In other words, quite a few people taking test prep are ones likely to be going to poor, under-performing school systems. Either test prep companies are incompetent or our school system is doing a lot better than I had expected, even on the low end!
Are the American methods of test preparation different from the methods you observed to be effective?
This is a good question—unfortunately, I am unable to answer this as I have no exposure to American schools.
Where do they get the old test papers? In America, it is common for the cram schools to supply “old test papers” that are actually considerably more difficult.
There are publishers here that publish collections of old test papers (along with the solutions) that were administered over the past 10 years, and many students practise by testing themselves using those.
This paper has some interesting information and proposes the idea that the tests are designed to be resistant to short-term test prep:
It also alludes to there being multiple studies that have failed to find large effect sizes from test-prep courses. However, I’m not sure I quite believe that test prep is that ineffective. I never did test-prep for the SAT because I got a good enough score my first time to get into the university I wanted, but I did use test-prep books for AP tests, a few times for classes that I didn’t even take, and I got 5⁄5 on most of those. I’m sure those were intended to be “cram-resistant” too, but they clearly weren’t.
It’s possible that the confusion here on LW comes from the systemic bias of the majority of people here being of above-average intelligence, though I don’t know what the mechanism for that would be. I’ll just add that “I notice that I am confused” as well, so something that we believe must be false...
My vague recollection of the SAT and ACT was that they were designed to test your ability to perform certain cognitive tasks, but didn’t test whether you had any particular knowledge. There may have been exceptions on grammar questions, but I think that in the math section, you were given any formulas you might need—whereas AP tests test knowledge extensively.
There may be a point at which you’ve taken so many test SAT/ACTs that, for lack of a better analogy, you’ve strengthened those cognitive muscles enough for it to make a difference, but cramming seems like an obviously ineffective way to handle this. AP tests, on the other hand, have a limited scope of facts for you to produce on command.
There was definitely a vocabulary component to the version of the SAT that I took (in the late Nineties). I seem to recall hearing that a later version of the test had dropped that, though, probably over cultural bias concerns.