Harvard… When you’re looking at like 30 applications for each seat, 10 SAT points could be the difference between success and failure for a few applicants.
Only if Harvard cares a lot about SAT scores. According to this graph, the value of SATs is pretty flat between the 93rd and 96th percentiles. Moreover, at other Ivies, SAT scores are penalized in this range. source, page 7(8)
This graph is not a direct measure of the role of SATs, because they can’t force all else to be equal. The paper argues that some schools really do penalize SAT scores in some regimes. I do not buy the argument, but the graph convinces me that I don’t know how it works. Many people respond to the graph that it is the aggregation of two populations admitted under different scoring rules, both of which value SATs, but I do not think that explains the graph.
Only if Harvard cares a lot about SAT scores. According to this graph, the value of SATs is pretty flat between the 93rd and 96th percentiles. Moreover, at other Ivies, SAT scores are penalized in this range. source, page 7(8)
Your graph doesn’t show that the average applicant won’t benefit from 10 points. It shows that overall, SAT scores make a big difference (from ~0 to 0.2, with not even bothering to show anyone below the 88th percentile).
This graph is not a direct measure of the role of SATs, because they can’t force all else to be equal.
The paper I cited earlier for logistic regressions used models controlling for other things. Given the benefits to athletes, legacies, and minorities, benefits necessary presumably because they cannot compete as well on other factors (like SAT scores), it’s not necessarily surprising if aggregating these populations can lead to a raw graph like those you show. Note that the most meritocratic school which places the least emphasis on ‘holistic’ admissions (enabling them to discriminate in various ways) is MIT, and their curve looks dramatically different from, say, Princeton.
Yes, if large SAT changes matter, then there must be some small changes that matter. But it is possible that other points on the scale where they don’t, or are harmful. I’m sorry if I failed to indicate that I meant only this limited point.
If a school admits two populations, then the histogram of SATs of its students might look like a camel. But why should the graph of chance of admission? I suppose Harvard’s graph makes sense if students apply when their assessment of their ability to get in crosses some threshold. Then applying screens off SATs, at least in some normal regime.* But at Yale and especially Princeton, rising SATs in the middle regime predicts greater mistaken belief in ability to get in. Legacies (but not athletes or AA) might explain the phenomenon by only applying to one elite school, but I don’t think legacies alone are big enough to cause the graph.
Here are the lessons I take away from the graphs that I would apply if I had been doing the regressions and wanted to explain the graphs. First, schools have different admissions policies, even schools as similar as Harvard and Yale. Averaging them together, as in the paper, may make things appear smoother than they really are. Second, given the nonlinear effect of SATs, it is good that the regression used buckets rather than assuming a linear effect. Third, since the bizarre downward slope is over the course of less than 100 points, the 100 point buckets of the regression may be too coarse to see it. Fourth, they could have shown graphs, too. It would have been so much more useful to graph SAT scores of athletes and probability of admission as a function of SAT scores of athletes. The main value of regressions is using the words “model” and “p-value.” Fifth, the other use of the regression model is that it lets them consider interactions, which do seem to say that there is not much interaction between SATs and other factors, that the marginal value of an SAT point does not depend on race, legacy status, or athlete status (except for the tiny <1000 category). But the coarseness of the buckets and the aggregating of schools does not allow me to draw much of a conclusion from this.
* Actually, the whole point of this thread is that you can’t completely screen off. But I want to elaborate on “normal regime.” At the high end, screening breaks down because if, say, 1500 SAT is enough to cross the threshold, everyone with 1500+ SAT applies and there is no screening phenomenon. At the low end, I don’t see why screening would break down. Why would someone with SAT<1000 apply to an elite school without really good reason? Yet lots of people apply with such low scores and don’t get in.
But it is possible that other points on the scale where they don’t, or are harmful.
Sure, there could be non-monotonicity.
If a school admits two populations, then the histogram of SATs of its students might look like a camel. But why should the graph of chance of admission?...Fifth, the other use of the regression model is that it lets them consider interactions, which do seem to say that there is not much interaction between SATs and other factors, that the marginal value of an SAT point does not depend on race, legacy status, or athlete status (except for the tiny <1000 category).
Imagine that Harvard lets in equal numbers of ‘athletes’ and ‘nerds’, the 2 groups are different populations with different means, and they do something like pick the top 10% in each group by score. Clearly there’s going to be a bimodal histogram of SAT scores: you have a lump of athlete scores in the 1000s, say, and a lump of nerd scores in the 1500s. Sure. 2 equal populations, different means, of course you’re going to see a bimodal.
Now imagine Harvard gets more 10x more nerd applicants than athletic applicants; since each group gets the same number of spots, a random nerd will have 1⁄10 the admission chance as an athlete. Poor nerds. But Harvard kept the admission procedure the same as before. So what happens when you look at admission probability if all you know is the SAT score? Well, if you look at the 1500s applicants, you’ll notice that an awful lot of them aren’t admitted; and if you look at the 1000s applicants, you’ll notice that an awful lot of them getting in. Does Harvard hate SAT scores? No, of course not: we specified they were picking mostly the high scorers, and indeed, if we classify each applicant into nerd or athlete categories and then looked at admission rates by score, we’d see that yes, increasing SAT scores is always good: the nerd with a 1200 better apply to other colleges, and the athlete with 1400 might as well start learning how to yacht.
So even though in aggregate in our little model, high SAT scores look like a bad thing, for each group higher SAT scores are better.
But the coarseness of the buckets and the aggregating of schools does not allow me to draw much of a conclusion from this.
Yes, I don’t think we could make a conclusive argument against the claim that SAT scores may not help at all levels, not without digging deep into all the papers running logistic regressions; but I regard that claim as pretty darn unlikely in the first place.
At the low end, I don’t see why screening would break down. Why would someone with SAT<1000 apply to an elite school without really good reason? Yet lots of people apply with such low scores and don’t get in.
They could be self-delusive, doing it to appease a delusive parent (‘My Johnnie Yu must go to Harvard and become a doctor!’), gambling that a tiny chance of admission is worth the effort, doing it on a dare, expecting that legacies or other things are more helpful than they actually are...
Sure, maybe you can make a model that outputs Harvard or Princeton’s results, but how do you explain the difference between Harvard and Princeton? It is easier to get into Princeton as either a jock or a nerd, but at 98th SAT percentile, it is harder to get into Princeton than Harvard. These are the smart jocks or dumb nerds. Maybe Harvard has first dibs on the smart jocks so that the student body is more bimodal at other schools. But why would admissions be more bimodal? Does Princeton not bother to admit the smart jocks? That’s the hypothesis in the paper: an SAT penalty. Or maybe Princeton rejects the dumb nerds. It would be one thing if Princeton, as a small school, admitted fewer nerds and just had higher standards for nerds. But they don’t at the high end. What’s going on? Here’s a hypothesis: Harvard (like Caltech) could admit nerds based on other achievements that only correlate with SATs, while Princeton has high pure-SAT standards.
I don’t think an SAT penalty is very plausible, but nothing I’ve heard sounds plausible. Mostly people make vague models like yours that I don’t think explain all the observations. The hypothesis that Princeton in contrast to Harvard does not count SAT for jocks beyond a graduation threshold at least does not sound insane.
not without digging deep into all the papers running logistic regressions
I take graphs over regressions, any day. Regressions fit a model. They yield very little information. Sometimes it’s exactly the information you want, as in the calculation you originally brought in the regression for. But with so little information there is no possibility of exploration or model checking.
By the way, the paper you cite is published at a journal with a data access provision.
Sure, maybe you can make a model that outputs Harvard or Princeton’s results, but how do you explain the difference between Harvard and Princeton?
Dunno. I’ve already pointed out the quasi-Simpsons Paradox effect that could produce a lot of different shapes even while SAT score increases always help. Maybe Princeton favors musicians or something. If the only reason to look into the question is your incredulity and interest in the unlikely possibility that increase in SAT score actually hurts some applicants, I don’t care nearly enough to do more than speculate.
By the way, the paper you cite is published at a journal with a data access provision.
I have citations in my DNB FAQ on how such provisions are honored mostly in the breach… I wonder what the odds that you could get the data and that it would be complete and useful.
Only if Harvard cares a lot about SAT scores. According to this graph, the value of SATs is pretty flat between the 93rd and 96th percentiles. Moreover, at other Ivies, SAT scores are penalized in this range. source, page 7(8)
This graph is not a direct measure of the role of SATs, because they can’t force all else to be equal. The paper argues that some schools really do penalize SAT scores in some regimes. I do not buy the argument, but the graph convinces me that I don’t know how it works. Many people respond to the graph that it is the aggregation of two populations admitted under different scoring rules, both of which value SATs, but I do not think that explains the graph.
Your graph doesn’t show that the average applicant won’t benefit from 10 points. It shows that overall, SAT scores make a big difference (from ~0 to 0.2, with not even bothering to show anyone below the 88th percentile).
The paper I cited earlier for logistic regressions used models controlling for other things. Given the benefits to athletes, legacies, and minorities, benefits necessary presumably because they cannot compete as well on other factors (like SAT scores), it’s not necessarily surprising if aggregating these populations can lead to a raw graph like those you show. Note that the most meritocratic school which places the least emphasis on ‘holistic’ admissions (enabling them to discriminate in various ways) is MIT, and their curve looks dramatically different from, say, Princeton.
Yes, if large SAT changes matter, then there must be some small changes that matter. But it is possible that other points on the scale where they don’t, or are harmful. I’m sorry if I failed to indicate that I meant only this limited point.
If a school admits two populations, then the histogram of SATs of its students might look like a camel. But why should the graph of chance of admission? I suppose Harvard’s graph makes sense if students apply when their assessment of their ability to get in crosses some threshold. Then applying screens off SATs, at least in some normal regime.* But at Yale and especially Princeton, rising SATs in the middle regime predicts greater mistaken belief in ability to get in. Legacies (but not athletes or AA) might explain the phenomenon by only applying to one elite school, but I don’t think legacies alone are big enough to cause the graph.
Here are the lessons I take away from the graphs that I would apply if I had been doing the regressions and wanted to explain the graphs. First, schools have different admissions policies, even schools as similar as Harvard and Yale. Averaging them together, as in the paper, may make things appear smoother than they really are. Second, given the nonlinear effect of SATs, it is good that the regression used buckets rather than assuming a linear effect. Third, since the bizarre downward slope is over the course of less than 100 points, the 100 point buckets of the regression may be too coarse to see it. Fourth, they could have shown graphs, too. It would have been so much more useful to graph SAT scores of athletes and probability of admission as a function of SAT scores of athletes. The main value of regressions is using the words “model” and “p-value.” Fifth, the other use of the regression model is that it lets them consider interactions, which do seem to say that there is not much interaction between SATs and other factors, that the marginal value of an SAT point does not depend on race, legacy status, or athlete status (except for the tiny <1000 category). But the coarseness of the buckets and the aggregating of schools does not allow me to draw much of a conclusion from this.
* Actually, the whole point of this thread is that you can’t completely screen off. But I want to elaborate on “normal regime.” At the high end, screening breaks down because if, say, 1500 SAT is enough to cross the threshold, everyone with 1500+ SAT applies and there is no screening phenomenon. At the low end, I don’t see why screening would break down. Why would someone with SAT<1000 apply to an elite school without really good reason? Yet lots of people apply with such low scores and don’t get in.
Sure, there could be non-monotonicity.
Imagine that Harvard lets in equal numbers of ‘athletes’ and ‘nerds’, the 2 groups are different populations with different means, and they do something like pick the top 10% in each group by score. Clearly there’s going to be a bimodal histogram of SAT scores: you have a lump of athlete scores in the 1000s, say, and a lump of nerd scores in the 1500s. Sure. 2 equal populations, different means, of course you’re going to see a bimodal.
Now imagine Harvard gets more 10x more nerd applicants than athletic applicants; since each group gets the same number of spots, a random nerd will have 1⁄10 the admission chance as an athlete. Poor nerds. But Harvard kept the admission procedure the same as before. So what happens when you look at admission probability if all you know is the SAT score? Well, if you look at the 1500s applicants, you’ll notice that an awful lot of them aren’t admitted; and if you look at the 1000s applicants, you’ll notice that an awful lot of them getting in. Does Harvard hate SAT scores? No, of course not: we specified they were picking mostly the high scorers, and indeed, if we classify each applicant into nerd or athlete categories and then looked at admission rates by score, we’d see that yes, increasing SAT scores is always good: the nerd with a 1200 better apply to other colleges, and the athlete with 1400 might as well start learning how to yacht.
So even though in aggregate in our little model, high SAT scores look like a bad thing, for each group higher SAT scores are better.
Reminds me of Simpson’s paradox.
Yes, I don’t think we could make a conclusive argument against the claim that SAT scores may not help at all levels, not without digging deep into all the papers running logistic regressions; but I regard that claim as pretty darn unlikely in the first place.
They could be self-delusive, doing it to appease a delusive parent (‘My Johnnie Yu must go to Harvard and become a doctor!’), gambling that a tiny chance of admission is worth the effort, doing it on a dare, expecting that legacies or other things are more helpful than they actually are...
Sure, maybe you can make a model that outputs Harvard or Princeton’s results, but how do you explain the difference between Harvard and Princeton? It is easier to get into Princeton as either a jock or a nerd, but at 98th SAT percentile, it is harder to get into Princeton than Harvard. These are the smart jocks or dumb nerds. Maybe Harvard has first dibs on the smart jocks so that the student body is more bimodal at other schools. But why would admissions be more bimodal? Does Princeton not bother to admit the smart jocks? That’s the hypothesis in the paper: an SAT penalty. Or maybe Princeton rejects the dumb nerds. It would be one thing if Princeton, as a small school, admitted fewer nerds and just had higher standards for nerds. But they don’t at the high end. What’s going on? Here’s a hypothesis: Harvard (like Caltech) could admit nerds based on other achievements that only correlate with SATs, while Princeton has high pure-SAT standards.
I don’t think an SAT penalty is very plausible, but nothing I’ve heard sounds plausible. Mostly people make vague models like yours that I don’t think explain all the observations. The hypothesis that Princeton in contrast to Harvard does not count SAT for jocks beyond a graduation threshold at least does not sound insane.
I take graphs over regressions, any day.
Regressions fit a model. They yield very little information. Sometimes it’s exactly the information you want, as in the calculation you originally brought in the regression for. But with so little information there is no possibility of exploration or model checking.
By the way, the paper you cite is published at a journal with a data access provision.
Dunno. I’ve already pointed out the quasi-Simpsons Paradox effect that could produce a lot of different shapes even while SAT score increases always help. Maybe Princeton favors musicians or something. If the only reason to look into the question is your incredulity and interest in the unlikely possibility that increase in SAT score actually hurts some applicants, I don’t care nearly enough to do more than speculate.
I have citations in my DNB FAQ on how such provisions are honored mostly in the breach… I wonder what the odds that you could get the data and that it would be complete and useful.