I find myself bristling at this article, but I think it might be for braverydebate reasons. That is, I think I have a well-calibrated sense of what mathematical ability looks like, how we can measure it, and so on, and this article seems to be targeted at people who are miscalibrated in one particular way.
An example:
Just looking at my math SAT score, people would think very unlikely that I would come close to being the strongest calculus student in my year.
Really? Which people would think that? The Math SAT is so simple that I studied for it the last time I took it because it had been too many years since I had originally learned the material. For highly numerate people, the Math SAT is mostly an error-counting competition; I was particularly lucky that the year that I took it, one mistake would only knock you down from 800 to 780. The verbal is more suited to the actual range of students, where you can miss several questions and still get an 800, because there aren’t a large number of perfect scores running around. (The range of the Math SAT is too small, basically.)
And so I think if you told someone familiar with math education “hey, here’s a calculus class of 65 students drawn from a population of 650 students at a magnet school, and here are their math SAT scores. What’s your posterior on any student being the top student in that class?”, you would find that they didn’t adjust their (presumably uniform) prior all that much on learning the SAT scores. And if you had told them “well, the ‘top student’ isn’t the actual top student on intelligence and conscientiousness, but just on intelligence” they might have actually preferred the lower SAT scores (given that the lowest Math SAT score in your class presumably cleared 650, probably even 700), as they’re evidence for lower conscientiousness.
One is that it is in fact the case that a significant fraction of people less knowledgable than you would accord significantly greater significance to the math SAT than you would.
The other is that my observation has been that the most mathematically talented people who I know have usually have scored 800 on the math SAT … You seem to be claiming that a ceiling effect) makes the test a bad measurement instrument, which certainly is true to some extent, and which is a priori plausible, but you may have been less lucky than you think.
The other is that my observation has been that the most mathematically talented people who I know have usually have scored 800 on the math SAT … You seem to be claiming that a ceiling effect) makes the test a bad measurement instrument, which certainly is true to some extent, and which is a priori plausible, but you may have been less lucky than you think.
Specifically, a bad measurement instrument at differentiating very high levels of mathematical ability. It works as well as you would expect when the measurement error doesn’t hit the ceiling or floor.
I should be clearer about my ‘luck’ claim: what a raw score of “all but one right, one wrong” gets you depends on the percentage of students who got “all right” that year, which depends on that year’s test difficulty. Some years it’s 760, some years it’s 780, and so on. (If I remember correctly, I got both of those processed scores from taking it two times and getting the same raw score.) I do not think the underlying raw score of “all but one right, one wrong” is due to luck (in the sense of my underlying skill creates a family of rate parameters for Poisson distributions that are summed together to get a total error count, and while any sample from that distribution is stochastic the distribution is very narrow).
See my comment here. I agree to some extent, but the correlation between cognitive ability and math SAT scores is positive for all levels of cognitive ability and SAT math scores, including the highest ones (even if it becomes substantially smaller).
Added: To operationalize the situation, I would guess that the frequency with which mathematicians who have won famous prizes (Abel Prize, Fields Medal, etc.) would miss no questions at all (say, as 18 year olds) would be noticeably higher than the corresponding frequency for professors at top 50 math departments. I’ll give evidence in subsequent posts.
To operationalize the situation, I would guess that the frequency with which mathematicians who have won famous prizes (Abel Prize, Fields Medal, etc.) would miss no questions at all (say, as 18 year olds) would be noticeably higher than the corresponding frequency for professors at top 50 math departments.
I agree that I expect it would be higher, though I would describe my expectation as “modest,” which probably overlaps with “noticeable.”
I find myself bristling at this article, but I think it might be for bravery debate reasons. That is, I think I have a well-calibrated sense of what mathematical ability looks like, how we can measure it, and so on, and this article seems to be targeted at people who are miscalibrated in one particular way.
An example:
Really? Which people would think that? The Math SAT is so simple that I studied for it the last time I took it because it had been too many years since I had originally learned the material. For highly numerate people, the Math SAT is mostly an error-counting competition; I was particularly lucky that the year that I took it, one mistake would only knock you down from 800 to 780. The verbal is more suited to the actual range of students, where you can miss several questions and still get an 800, because there aren’t a large number of perfect scores running around. (The range of the Math SAT is too small, basically.)
And so I think if you told someone familiar with math education “hey, here’s a calculus class of 65 students drawn from a population of 650 students at a magnet school, and here are their math SAT scores. What’s your posterior on any student being the top student in that class?”, you would find that they didn’t adjust their (presumably uniform) prior all that much on learning the SAT scores. And if you had told them “well, the ‘top student’ isn’t the actual top student on intelligence and conscientiousness, but just on intelligence” they might have actually preferred the lower SAT scores (given that the lowest Math SAT score in your class presumably cleared 650, probably even 700), as they’re evidence for lower conscientiousness.
I guess there are two things to say here.
One is that it is in fact the case that a significant fraction of people less knowledgable than you would accord significantly greater significance to the math SAT than you would.
The other is that my observation has been that the most mathematically talented people who I know have usually have scored 800 on the math SAT … You seem to be claiming that a ceiling effect) makes the test a bad measurement instrument, which certainly is true to some extent, and which is a priori plausible, but you may have been less lucky than you think.
Specifically, a bad measurement instrument at differentiating very high levels of mathematical ability. It works as well as you would expect when the measurement error doesn’t hit the ceiling or floor.
I should be clearer about my ‘luck’ claim: what a raw score of “all but one right, one wrong” gets you depends on the percentage of students who got “all right” that year, which depends on that year’s test difficulty. Some years it’s 760, some years it’s 780, and so on. (If I remember correctly, I got both of those processed scores from taking it two times and getting the same raw score.) I do not think the underlying raw score of “all but one right, one wrong” is due to luck (in the sense of my underlying skill creates a family of rate parameters for Poisson distributions that are summed together to get a total error count, and while any sample from that distribution is stochastic the distribution is very narrow).
See my comment here. I agree to some extent, but the correlation between cognitive ability and math SAT scores is positive for all levels of cognitive ability and SAT math scores, including the highest ones (even if it becomes substantially smaller).
Added: To operationalize the situation, I would guess that the frequency with which mathematicians who have won famous prizes (Abel Prize, Fields Medal, etc.) would miss no questions at all (say, as 18 year olds) would be noticeably higher than the corresponding frequency for professors at top 50 math departments. I’ll give evidence in subsequent posts.
I agree that I expect it would be higher, though I would describe my expectation as “modest,” which probably overlaps with “noticeable.”