Testing is optimized for predicting current and future competence in a subject:
What if this finds that the best predictors for most employers consists of two tests (1) a <10 minute IQ test, and (2) a test showing a student’s willingness to submit to authority and learn what he was told, in the manner under which he was told without getting too far ahead or behind his classmates?
I am willing to bet money (up to €100, I am a student) that these will not, in fact, be the best predictors, just as grades in college have been proven not to be good predictors of how well a Google employee does, IQ tests have been proven not to be a good predictor of how well a salesman does, and higher salaries have proven to, past a certain point, be detrimental to productivity for employees whose jobs require creativity, imagination or otherwise intellectual exertion.
My hypothesis is that a <10 min IQ test will be so inaccurate as to be useless, and that the current school system already favors people who do what they are told as they are told without getting too far ahead or behind, with the destructive and wasteful results we all know: they are the ones it grades best, and yet, time and again, tested against the workplace, those grades have proven little.
IQ test scores are massively correlated with workplace performance.
I find it plausible that among the set of people Google has hired grades don’t predict workplace success, but I bet if Google were to randomly select employees from among the U.S. population (or even from just among computer science majors) grades would be hugely correlated with performance.
What are the terms of the bet? I would be willing to bet €100 depending on the terms. Full disclosure: I read a lot of academic research on IQ to help me write this book, one of whose chapters is about IQ.
There is a lot of silly stuff written about IQ. No offense, but do you think you have the needed background to separate the high from the low quality IQ scholarship?
I only read Talent is Overrated, which cited a few studies to the effect that IQ was a bad predictor of professional performance beyond the short term; to become a world-class performer at some specific skill, you need to spend 20000 hours of your life deliberately practicing it (trying things outside your comfort zone, failing to get it right, figuring out what to improve and how, getting it right, trying something harder, rinse and repeat, and it’s a pain in the neck). Most people, no matter how smart, tend to stop improving at their jobs after around one year and a half of practice: this includes highly skilled professionals like doctors and lawyers.
And, on the anecdotal side, in my own experience and what I could get from the people that frequent Lesswrong and TVTropes, a high IQ commonly results in huge childhood issues, a miserable social life, and a boatpload of akrasia, which seriously hamper whatever base utility it might have. Perhaps there’s a selection bias and only the incompetent brains have the time to hang out here, while competent smart people are too busy actually getting things done?
in my own experience and what I could get from the people that frequent Lesswrong and TVTropes, a high IQ commonly results in [...] a miserable social life, and a boatpload of akrasia [...] Perhaps there’s a selection bias and only the incompetent brains have the time to hang out here[?]
There is massive selection bias going on here.
I don’t think it has as much to do with free time as with target audience, though. LW attracts a few different clusters of people, but the ones you’re seeing in this context are those who feel their thinking is flawed in some way, and who believe they have a decent chance of fixing it with a cognitive science toolset and vocabulary. The site’s native idiom and interaction style—basically a founder effect—imposes a few more filters, underrepresenting some problems and overrepresenting others. Akrasia and social problems are precisely the issues I’d expect to see a lot of, given those constraints.
TV Tropes… well, that might have more to do with free time. Almost everyone likes media, but if you want to make many original contributions, you need unusual knowledge of media and an analytical attitude towards it. Moreover, the media best represented there tend to be the most time-consuming ones—TV, anime, doorstopper fantasy novels. I don’t think I need to go into too much detail regarding the people most likely to share those requirements.
Offtopic post, but a discussion I wish to pursue nonetheless:
Regarding TVT: it used to be so. Nowadays school study media and the “literary canon” are beginning to find their way in… and all those tropes with silly names, built from mass media, are proving their usefulness as tools of analysis. Of course, getting a movie adaptation or a TV miniseries is one of the best ways to draw troper attention to a work, but classics always end up getting those with some regularity. So let’s just say that the user base has widened. Oh, and many classics are as doorstoppery as modern fantasy sagas, especially stuff from the XIXth century, when novels where published as long-running serials in magazines and authors were paid by the word. When people call Eliezer Yudkowsky a terrible writer because of MoR’s lack of tightness or his using it as a vehicle for ideas and lectures, I feel half-tempted to point at the likes of Victor Hugo or Alexandre Dumas or Dickens or Benito Pérez Galdós, just to name a few… surely if those are the traits of terrible writing, it means that those books have room for improvement, if only by way of abridging them?
Regarding LW: The filters the site imposes on its demographics (language and mode of interaction) worry me: what’s the point of translating MoR to Spanish or French, if afterwards hispanophones and francophones don’t get to read their sequences and have a site to be a community in? I’m trying to seed a rationalist community at my school, and the language barrier is proving to be troublesome; I often find myself not even knowing how a bias would be called in Spanish.
what’s the point of translating MoR to Spanish or French, if afterwards hispanophones and francophones don’t get to read their sequences and have a site to be a community in?
There have been a fewSequencestranslationprojects discussed, proposed, or started, in various languages, plus a few more for MoR. Of these, only one (diegocaleiro’s, in Portugese) seems aimed at a persistent community hub. I might have missed some, though.
With regard to the specific issue of bias names, if you don’t have access to cognitive science faculty, I think a good place to start might be finding a translation of Daniel Kahneman’s Thinking, Fast and Slow; Kahneman is a major source for the Sequences, his book’s been widely translated, and there should be a lot of overlap. After that, you might want to try digging into XiXiDu’s resource list. Though there are a few original LW coinages floating around, and I’m afraid you’re on your own with those.
Well thank goodness I’m on a holiday and can actually afford to read all that. Nevertheless, such a violent flipflop of opinions would make me dangerously closer to epistemic learned helplessness, i.e. giving up on learning new stuff because of not feeling confident that I can tell good arguments from bad ones when they come from experts after getting burned one time too many.
That is a dangerous state for a rationalist to be in. so would you please be a dear and have a look at that book to figure out how precisely it junk-scienced me and the rest of its readers? It would really help me out and I’d really be grateful for that.
Also, I have a vested interest in believing in the “only deliberate practice matters” thesis. I used to feel having a high IQ obliged me to get excellent results with little effort, and every time that didn’t happen I felt disappointed in myself and that I didn’t live up to some weird standard. Nowadays I only believe in working as hard as possible for as long as possible, and it serves me much better.
That is a dangerous state for a rationalist to be in. so would you please be a dear and have a look at that book to figure out how precisely it junk-scienced me and the rest of its readers? It would really help me out and I’d really be grateful for that.
I’m afraid I haven’t read that exact popularization, but if it’s drawing on Ericsson’s research as it sounds like, the explanation is easy enough: Ericsson’s points are valid largely because the studies are correlational, do not control for underlying factors or Matthew effects, and suffer from heavy range restriction in he’s already looking at people who are selected or self-selected to be elites.
(ie. suppose someone studied MIT physicists with a mean IQ of 150 and discovered that in this group of physicists, Conscientiousness predicted better than IQ which would go on to win Nobels. This is a possible result, and what this has actually demonstrated is “you have to be incredibly brainy to be a MIT physicist in the first place, but once you’ve gotten that, then other things are also important; which is another way of saying that if we look at the general population, like all the people from IQ 60 to 150, IQ is the overwhelming most important trait” but this is easily popularized to “IQ doesn’t matter!”)
Also, I have a vested interest in believing in the “only deliberate practice matters” thesis. I used to feel having a high IQ obliged me to get excellent results with little effort, and every time that didn’t happen I felt disappointed in myself and that I didn’t live up to some weird standard. Nowadays I only believe in working as hard as possible for as long as possible, and it serves me much better.
As a practical matter, it’s probably a good idea to believe hard work matters and deliberate practice matters. If you could somehow improve your intelligence, then it might be important to hold correct beliefs about IQ being far more important than practice; but unfortunately, IQ is pretty much fixed and all that’s left is to make the best use possible. Given people matched on IQ and other traits, and Conscientiousness will be pretty important.
However, in other contexts, it’s very important to hold the correct beliefs about the relative value of intelligence and ‘just work harder’ - if we were discussing iodization or immigration or whether someone should go into debt for college, for example. Many population-level questions will rest far more on intelligence than other traits.
Understanding which context we’re in can be a hard balancing act, and especially difficult when reading papers making statistical claims (did they control for IQ? Should they control? Or for education? Or for range restriction? Or for reliability of their metrics?); I try to be consistent and clear in any discussions of Conscientiousness or IQ which context we’re in and which we should value or ignore, but I don’t think I always succeed.
Nowadays I only believe in working as hard as possible for as long as possible, and it serves me much better.
Imagine a world where 50% of your results are genetically determined and 50% of your results are hard work. What would be the best strategy for success in that world, assuming that you already have decent genes? It would be working hard. Not working 50% hard, but working 100% hard.
Seems like you found the right strategy for the wrong reasons. You can keep the strategy; you don’t have to blindly reverse your decisions.
I dunno, at a certain point the marginal utility of one unit of hard work will be less than the marginal utility of one unit of leisure, and it’s well possible that the point at which that happens depends on how genetically good you are.
You should work 100% hard on whatever you’re working on when you’re working on, but there might still be cases where you should think about the nature/nurture ratio to get the best outcome.
If outcomes are all about hard work, doggedly aiming for a rare high-reward position that requires a large amount of skill, like a quantitative analyst on Wall Street or a professional athlete, can be a good high risk / high reward strategy. But the more you know outcomes to be affected by genetic talent, the faster you’d want to recognize that some goals are beyond you and direct your 100% effort elsewhere if you find your genetic talent lacking, because then the people who also put in 100% effort but have more genetic talent than you will take all the positions no matter how much effort you put in.
I’d be curious to see that research that proves IQ test scores are correlated with workplace performance. All jobs or just some? How strongly correlated? How is workplace performance measured? Is it discussed in your book?
A good starting point might be this 2004 review by Schmidt & Hunter. It tabulates correlations of 0.23 to 0.58 between “general mental ability” and “supervisory ratings of job performance”, with the correlation increasing monotonically with a five-level rating of job complexity.
Edit: I should probably add Schmidt & Hunter’s note that those averages “are corrected for measurement error in the dependent variable and for range restriction but not for measurement error in the GMA measure; hence, these are estimates of operational validities, not construct-level correlations.”
It’s very useful and, in the U.S., illegal. Employers, I suspect, often favor applicants with degrees from prestigious colleges as a way to get around the direct ban on the use of IQ tests.
Given recent Supreme Court decisions, I’m not sure how much longer this will be a true statement of US law.
The most recent case held that fear of disparate impact lawsuits was not a justification for a disparate treatment (throwing out a promotion test because no minority passed). In a concurrence (agree on result, not reasoning), one justice noted that disparate impact treatment law for employment is in considerable tension with the constitutional doctrine on equal protection—ConLaw equal protection does not have a disparate treatment component, while you correctly note that private employment discrimination law does.
There are two possible resolutions of this legal tension: a) Employers are screwed—any choice subjects them to either (1) a valid disparate impact claim or (2) a valid disparate treatment claim. b) Disparate impact claims will be tightened substantially—to the point that one would likely need to prove a facially neutral test was selected primarily for its disparate impact. This is functionally equivalent to lowering the legally required evidence to show disparate treatment.
My read of the legal landscape is that (b) is massively more likely than (a). (> .99). The resolution of that tension might even make private disparate impact claims practically impossible to prove. Or disparate impact liability might even be ruled unconstitutional (in the absence of disparate treatment evidence).
Although IQ tests were initially outlawed by the courts on disparate impact grounds, I believe that Congress enacted these IQ test restrictions directly into civil rights laws so they can’t be overturned by courts except on constitutional grounds, which seem unlikely.
The test makers in the Ricci v. DeStefano case you cite went out of their way to have their tests not be just IQ tests.
I’ve looked briefly, and I don’t see a federal statutory cite that explicitly prohibits IQ testing. Most employer decisions not to administer an IQ can be almost completely explained by a risk-averse unwillingness to defend IQ tests. Defending lawsuits costs money, even if one wins, and litigation is always a risk.
In short, I think employer behavior avoiding IQ tests is expect-value maximizing for individual employers—for tragedy-of-the-commons reasons, this might be bad for employers overall. Most importantly, I don’t think there is a federal law specifically prohibiting IQ testing for employment, it’s just expensive for an employer to show that higher IQ is helpful for the specific position the potential employee is seeking.
That article is consistent with my brief legal research, and increases my confidence that Congress has not explicitly prohibited IQ tests by statute, regardless of whether a potential employee could show disparate impact through use of the IQ test. I think this quote from the article is a reasonable statement of current law:
If one reads between the lines of Griggs and all other disparate impact type cases, one may intuit that the real holding is that any moron can do prole jobs such as repairing electrical power lines, and therefore any type of hiring criteria that tends to discriminate against blacks is going to be illegal. But the rarefied world of upper-middle-class jobs are obvious not included.
I make no comment on whether the current state of the law optimizes what it claims, or what it should optimize in some moral sense.
One possible source of information here is what goes in other countries that do use IQ tests for this end.
In England I’ve witnessed de facto IQ tests being used for grammar schooladmissions, and job applications to banks & auditors. So some English headmasters & professional service companies’ HR departments think they’re useful, which does not prove they are useful, but is good evidence of it.
I suspect the tests are more useful for the schools than the graduate employers, since there’s less scope for cheating with the school tests (which are virtually always administered like exams AFAIK) than the employment tests (the first round of which can sometimes be taken on any Internet-connected computer, such as the one you’re sitting at in a campus café while your mate hangs over the back of your seat and has a jolly time alternating between calling out answers and ribbing you for feeling the need to cheat), and the employers are sampling a more range-restricted population (stereotypically, 21- or 22-year-olds with 2:1 or 1st class quantitative degrees from impressive-sounding universities).
I don’t doubt that IQ tests can be useful, but I’m sceptical that one shorter than 10 minutes can be so robust as to make anything else (save a test for willingness to submission) unnecessary.
I was speaking more to your Campbelling point than JM’s shortness point. I’ll quickly address the latter now.
I’ve seen papers mention developing or using short versions of IQ tests that give scores correlating well (r > 0.8) with scores on the original full tests. I don’t know whether these “short” tests last <10 minutes, but it wouldn’t surprise me. Also, the US General Social Survey has a 10-item (I think?) vocabulary test that works as a passable proxy for IQ, even though it must take only 5 minutes or so to administer. Given these two existing kinds of test, I wouldn’t be surprised if a 9-minute IQ test had enough reliability to be almost as good as a full-length IQ test.
Note that JM wasn’t saying his two suggested test types would make other tests unnecessary. I agree with you that short tests for IQ & submissiveness wouldn’t make all other predictors of job performance superfluous. (Conscientiousness tests & structured interviews, if I remember rightly, both predict job performance even after controlling for IQ.) JM just said they’d be better predictors than anything else, which wouldn’t surprise me much. AFAIK, a full-length IQ test is the best general predictor of job performance, and the same would presumably be true for a 9-minute IQ test were it reliable enough (q.v. the previous paragraph).
[Edit: Christ, I really like that “wouldn’t surprise me” phrase, don’t I?]
Also, the US General Social Survey has a 10-item (I think?) vocabulary test that works as a passable proxy for IQ, even though it must take only 5 minutes or so to administer.
That sounds even more amenable to that problem (if the items are the same in all editions of the test): if employers started to use such a test to sort prospective employees, within a year applicants would probably start memorizing those particular 10 words and >80% of them would ace the test.
From what I’ve seen in certain similar situations, some people would prepare for such a test by trying to memorize every single word in the wordbank, even if there are several thousands of them; OTOH, if the bank is large enough, then the extent to which they’d manage to do that would probably correlate with IQ strongly enough. (Unless some candidates use SRSs and other don’t.)
People have already pointed out that IQ tests are considered to be illegal discrimination, and have been ignoring that, but I think a good case can be made that it is proper to treat them as illegal discrimination. Imagine that a test genuinely detects ability to perform at the job but is also correlated with race independently of whether people of that race can do the job. Under those circumstances, if you use an IQ test to test employees, you’ll 1) avoid stupid employees and 2) avoid employees of one race in favor of equally competent people of another.
The overall result will be beneficial to you, since avoiding stupid employees benefits you while avoiding employees of one race has neutral effects on you. But the overall effect on people of that race will be devastating.
This will never be corrected by the market. The employers, after all, benefit unless there are so few available employees that cutting out the good ones of the wrong race reduces the pool of potential employees enough to affect the overall employee quality to a significant degree. How could we prevent this other than by banning IQ tests?
The overall result will be beneficial to you, since avoiding stupid employees benefits you while avoiding employees of one race has neutral effects on you. But the overall effect on people of that race will be devastating.
This ‘devastating’ effect sounds drastic. It also seems to directly require the assumption that the performance of people of the particular race perform devastatingly poor on IQ tests. That’s a rather dubious assumption and when asked to ‘imagine’ a world where this idea is so, any conclusions we draw only apply in that imaginary universe, not this one. This is not a ‘good case’.
When arguing that IQ tests should be illegal discrimination due to drastic racial bias of IQ tests the reasoning from the premise to conclusion is straightforward. The bulk of a ‘good case’ must be in providing evidence for the premise. Maybe there is sufficient evidence to make that claim about IQ tests but imagining it is not sufficient.
This ‘devastating’ effect sounds drastic. It also seems to directly require the assumption that the performance of people of the particular race perform devastatingly poor on IQ tests.
No, it doesn’t. It only requires the assumption that 1) such people score slightly more poorly, and 2) employers begin choosing employees from the tail end of the distribution. A slight difference in scores on the test will reduce the percentage at the tail end out of proportion to the size of the difference. I don’t remember the formal name for this, but it’s very well known.
Edit: Also, consider what percentage of the population needs to be unemployed for the problem to be considered serious. 10% is bad. If test bias causes an additional 5 or 10% to be unemployed, that’s going to be a huge effect even if 5% or 10% is only a small part of the entire population. Nobody says that the unemployment rate needs to get up to 25% or 50% in order for unemployment to be a serious problem.
No, it doesn’t. It only requires the assumption that 1) such people score slightly more poorly, and 2) employers begin choosing employees from the tail end of the distribution. A slight difference in scores on the test will reduce the percentage at the tail end out of proportion to the size of the difference. I don’t remember the formal name for this, but it’s very well known.
I will not quibble about what constitutes ‘devastatingly poor’ beyond observing that I of course agree that the difference need not be numerically large in order to be devastating. I will again point out that you are assuming that there is sufficient bias in the IQ test that if used (instead of whatever other form of measurement or selection could be adopted) the change will be devastating. That is a significant claim, particularly when it is quite possible that subjective evaluations of intelligence by interviewers are more biased than IQ tests. I expect IQ may be somewhat racially biased. I know for certain that unstructured human arbitration is racially biased. I don’t know for sure to which is worse but do observe that it is sufficiently controversial that some evidence is required.
You asked the reader to imagine a world where some assumption holds—which is fine as far as it goes. It becomes an error in reasoning when you jump from “imagine...” to “a good case” without providing evidence that the imaginary scenario applies to reality. Even the bare statement “I assert that IQ tests are more racially biased than whatever is used in their stead” would have made the case at least coherent, albeit still weak.
The argument was in response to the implicit assumption that even if the tests are biased, we shouldn’t worry about that as long as they can predict performance. “Imagine” takes the place of the word “if” in there and the intended conclusion is “yes, you should worry about whether the tests are biased, because if they are biased, that would be bad.” To make that conclusion I do not need to provide evidence that the imaginary scenario applies to reality.
If I say “imagine that you jumped off a cliff. You’ll get smashed. Maybe you should avoid jumping off cliffs”, I don’t need to provide evidence that you jump off cliffs, because the conclusion is in the form of a conditional that already conditions on whether the imaginary scenario is real.
What if this finds that the best predictors for most employers consists of two tests (1) a <10 minute IQ test, and (2) a test showing a student’s willingness to submit to authority and learn what he was told, in the manner under which he was told without getting too far ahead or behind his classmates?
I am willing to bet money (up to €100, I am a student) that these will not, in fact, be the best predictors, just as grades in college have been proven not to be good predictors of how well a Google employee does, IQ tests have been proven not to be a good predictor of how well a salesman does, and higher salaries have proven to, past a certain point, be detrimental to productivity for employees whose jobs require creativity, imagination or otherwise intellectual exertion.
My hypothesis is that a <10 min IQ test will be so inaccurate as to be useless, and that the current school system already favors people who do what they are told as they are told without getting too far ahead or behind, with the destructive and wasteful results we all know: they are the ones it grades best, and yet, time and again, tested against the workplace, those grades have proven little.
IQ test scores are massively correlated with workplace performance.
I find it plausible that among the set of people Google has hired grades don’t predict workplace success, but I bet if Google were to randomly select employees from among the U.S. population (or even from just among computer science majors) grades would be hugely correlated with performance.
How much are you betting?
I can cite my studies, if you want. It may take me some time to find them (an afternoon or two). Can you cite yours?
What are the terms of the bet? I would be willing to bet €100 depending on the terms. Full disclosure: I read a lot of academic research on IQ to help me write this book, one of whose chapters is about IQ.
There is a lot of silly stuff written about IQ. No offense, but do you think you have the needed background to separate the high from the low quality IQ scholarship?
I only read Talent is Overrated, which cited a few studies to the effect that IQ was a bad predictor of professional performance beyond the short term; to become a world-class performer at some specific skill, you need to spend 20000 hours of your life deliberately practicing it (trying things outside your comfort zone, failing to get it right, figuring out what to improve and how, getting it right, trying something harder, rinse and repeat, and it’s a pain in the neck). Most people, no matter how smart, tend to stop improving at their jobs after around one year and a half of practice: this includes highly skilled professionals like doctors and lawyers.
And, on the anecdotal side, in my own experience and what I could get from the people that frequent Lesswrong and TVTropes, a high IQ commonly results in huge childhood issues, a miserable social life, and a boatpload of akrasia, which seriously hamper whatever base utility it might have. Perhaps there’s a selection bias and only the incompetent brains have the time to hang out here, while competent smart people are too busy actually getting things done?
There is massive selection bias going on here.
I don’t think it has as much to do with free time as with target audience, though. LW attracts a few different clusters of people, but the ones you’re seeing in this context are those who feel their thinking is flawed in some way, and who believe they have a decent chance of fixing it with a cognitive science toolset and vocabulary. The site’s native idiom and interaction style—basically a founder effect—imposes a few more filters, underrepresenting some problems and overrepresenting others. Akrasia and social problems are precisely the issues I’d expect to see a lot of, given those constraints.
TV Tropes… well, that might have more to do with free time. Almost everyone likes media, but if you want to make many original contributions, you need unusual knowledge of media and an analytical attitude towards it. Moreover, the media best represented there tend to be the most time-consuming ones—TV, anime, doorstopper fantasy novels. I don’t think I need to go into too much detail regarding the people most likely to share those requirements.
Offtopic post, but a discussion I wish to pursue nonetheless:
Regarding TVT: it used to be so. Nowadays school study media and the “literary canon” are beginning to find their way in… and all those tropes with silly names, built from mass media, are proving their usefulness as tools of analysis. Of course, getting a movie adaptation or a TV miniseries is one of the best ways to draw troper attention to a work, but classics always end up getting those with some regularity. So let’s just say that the user base has widened. Oh, and many classics are as doorstoppery as modern fantasy sagas, especially stuff from the XIXth century, when novels where published as long-running serials in magazines and authors were paid by the word. When people call Eliezer Yudkowsky a terrible writer because of MoR’s lack of tightness or his using it as a vehicle for ideas and lectures, I feel half-tempted to point at the likes of Victor Hugo or Alexandre Dumas or Dickens or Benito Pérez Galdós, just to name a few… surely if those are the traits of terrible writing, it means that those books have room for improvement, if only by way of abridging them?
Regarding LW: The filters the site imposes on its demographics (language and mode of interaction) worry me: what’s the point of translating MoR to Spanish or French, if afterwards hispanophones and francophones don’t get to read their sequences and have a site to be a community in? I’m trying to seed a rationalist community at my school, and the language barrier is proving to be troublesome; I often find myself not even knowing how a bias would be called in Spanish.
There have been a few Sequences translation projects discussed, proposed, or started, in various languages, plus a few more for MoR. Of these, only one (diegocaleiro’s, in Portugese) seems aimed at a persistent community hub. I might have missed some, though.
With regard to the specific issue of bias names, if you don’t have access to cognitive science faculty, I think a good place to start might be finding a translation of Daniel Kahneman’s Thinking, Fast and Slow; Kahneman is a major source for the Sequences, his book’s been widely translated, and there should be a lot of overlap. After that, you might want to try digging into XiXiDu’s resource list. Though there are a few original LW coinages floating around, and I’m afraid you’re on your own with those.
That is junk science.
The correlation between IQ and workplace performance is extremely robust and very well established.
See Intelligence: Knowns and Unknowns
Well thank goodness I’m on a holiday and can actually afford to read all that. Nevertheless, such a violent flipflop of opinions would make me dangerously closer to epistemic learned helplessness, i.e. giving up on learning new stuff because of not feeling confident that I can tell good arguments from bad ones when they come from experts after getting burned one time too many.
That is a dangerous state for a rationalist to be in. so would you please be a dear and have a look at that book to figure out how precisely it junk-scienced me and the rest of its readers? It would really help me out and I’d really be grateful for that.
Also, I have a vested interest in believing in the “only deliberate practice matters” thesis. I used to feel having a high IQ obliged me to get excellent results with little effort, and every time that didn’t happen I felt disappointed in myself and that I didn’t live up to some weird standard. Nowadays I only believe in working as hard as possible for as long as possible, and it serves me much better.
I’m afraid I haven’t read that exact popularization, but if it’s drawing on Ericsson’s research as it sounds like, the explanation is easy enough: Ericsson’s points are valid largely because the studies are correlational, do not control for underlying factors or Matthew effects, and suffer from heavy range restriction in he’s already looking at people who are selected or self-selected to be elites.
(ie. suppose someone studied MIT physicists with a mean IQ of 150 and discovered that in this group of physicists, Conscientiousness predicted better than IQ which would go on to win Nobels. This is a possible result, and what this has actually demonstrated is “you have to be incredibly brainy to be a MIT physicist in the first place, but once you’ve gotten that, then other things are also important; which is another way of saying that if we look at the general population, like all the people from IQ 60 to 150, IQ is the overwhelming most important trait” but this is easily popularized to “IQ doesn’t matter!”)
As a practical matter, it’s probably a good idea to believe hard work matters and deliberate practice matters. If you could somehow improve your intelligence, then it might be important to hold correct beliefs about IQ being far more important than practice; but unfortunately, IQ is pretty much fixed and all that’s left is to make the best use possible. Given people matched on IQ and other traits, and Conscientiousness will be pretty important.
However, in other contexts, it’s very important to hold the correct beliefs about the relative value of intelligence and ‘just work harder’ - if we were discussing iodization or immigration or whether someone should go into debt for college, for example. Many population-level questions will rest far more on intelligence than other traits.
Understanding which context we’re in can be a hard balancing act, and especially difficult when reading papers making statistical claims (did they control for IQ? Should they control? Or for education? Or for range restriction? Or for reliability of their metrics?); I try to be consistent and clear in any discussions of Conscientiousness or IQ which context we’re in and which we should value or ignore, but I don’t think I always succeed.
Well thank you very much!
.. sigh This may be the first time I find some actual use for compartmentalization, and I can’t do it anymore...
Imagine a world where 50% of your results are genetically determined and 50% of your results are hard work. What would be the best strategy for success in that world, assuming that you already have decent genes? It would be working hard. Not working 50% hard, but working 100% hard.
Seems like you found the right strategy for the wrong reasons. You can keep the strategy; you don’t have to blindly reverse your decisions.
I dunno, at a certain point the marginal utility of one unit of hard work will be less than the marginal utility of one unit of leisure, and it’s well possible that the point at which that happens depends on how genetically good you are.
You should work 100% hard on whatever you’re working on when you’re working on, but there might still be cases where you should think about the nature/nurture ratio to get the best outcome.
If outcomes are all about hard work, doggedly aiming for a rare high-reward position that requires a large amount of skill, like a quantitative analyst on Wall Street or a professional athlete, can be a good high risk / high reward strategy. But the more you know outcomes to be affected by genetic talent, the faster you’d want to recognize that some goals are beyond you and direct your 100% effort elsewhere if you find your genetic talent lacking, because then the people who also put in 100% effort but have more genetic talent than you will take all the positions no matter how much effort you put in.
See Hambrick et al, Deliberate practice: Is that all it takes to become an expert?, for a recent critical assessment of the claim that deliberate practice is sufficient for attaining expertise. Gwern provides some quotes.
I’d be curious to see that research that proves IQ test scores are correlated with workplace performance. All jobs or just some? How strongly correlated? How is workplace performance measured? Is it discussed in your book?
A good starting point might be this 2004 review by Schmidt & Hunter. It tabulates correlations of 0.23 to 0.58 between “general mental ability” and “supervisory ratings of job performance”, with the correlation increasing monotonically with a five-level rating of job complexity.
Edit: I should probably add Schmidt & Hunter’s note that those averages “are corrected for measurement error in the dependent variable and for range restriction but not for measurement error in the GMA measure; hence, these are estimates of operational validities, not construct-level correlations.”
Even if (1) was possible (which I kind-of doubt), it’d be quickly Campbelled into uselessness if employers started to use it.
It’s very useful and, in the U.S., illegal. Employers, I suspect, often favor applicants with degrees from prestigious colleges as a way to get around the direct ban on the use of IQ tests.
Given recent Supreme Court decisions, I’m not sure how much longer this will be a true statement of US law.
The most recent case held that fear of disparate impact lawsuits was not a justification for a disparate treatment (throwing out a promotion test because no minority passed). In a concurrence (agree on result, not reasoning), one justice noted that disparate impact treatment law for employment is in considerable tension with the constitutional doctrine on equal protection—ConLaw equal protection does not have a disparate treatment component, while you correctly note that private employment discrimination law does.
There are two possible resolutions of this legal tension:
a) Employers are screwed—any choice subjects them to either (1) a valid disparate impact claim or (2) a valid disparate treatment claim.
b) Disparate impact claims will be tightened substantially—to the point that one would likely need to prove a facially neutral test was selected primarily for its disparate impact. This is functionally equivalent to lowering the legally required evidence to show disparate treatment.
My read of the legal landscape is that (b) is massively more likely than (a). (> .99). The resolution of that tension might even make private disparate impact claims practically impossible to prove. Or disparate impact liability might even be ruled unconstitutional (in the absence of disparate treatment evidence).
Although IQ tests were initially outlawed by the courts on disparate impact grounds, I believe that Congress enacted these IQ test restrictions directly into civil rights laws so they can’t be overturned by courts except on constitutional grounds, which seem unlikely.
The test makers in the Ricci v. DeStefano case you cite went out of their way to have their tests not be just IQ tests.
I’ve looked briefly, and I don’t see a federal statutory cite that explicitly prohibits IQ testing. Most employer decisions not to administer an IQ can be almost completely explained by a risk-averse unwillingness to defend IQ tests. Defending lawsuits costs money, even if one wins, and litigation is always a risk.
In short, I think employer behavior avoiding IQ tests is expect-value maximizing for individual employers—for tragedy-of-the-commons reasons, this might be bad for employers overall. Most importantly, I don’t think there is a federal law specifically prohibiting IQ testing for employment, it’s just expensive for an employer to show that higher IQ is helpful for the specific position the potential employee is seeking.
See this.
That article is consistent with my brief legal research, and increases my confidence that Congress has not explicitly prohibited IQ tests by statute, regardless of whether a potential employee could show disparate impact through use of the IQ test. I think this quote from the article is a reasonable statement of current law:
I make no comment on whether the current state of the law optimizes what it claims, or what it should optimize in some moral sense.
One possible source of information here is what goes in other countries that do use IQ tests for this end.
In England I’ve witnessed de facto IQ tests being used for grammar school admissions, and job applications to banks & auditors. So some English headmasters & professional service companies’ HR departments think they’re useful, which does not prove they are useful, but is good evidence of it.
I suspect the tests are more useful for the schools than the graduate employers, since there’s less scope for cheating with the school tests (which are virtually always administered like exams AFAIK) than the employment tests (the first round of which can sometimes be taken on any Internet-connected computer, such as the one you’re sitting at in a campus café while your mate hangs over the back of your seat and has a jolly time alternating between calling out answers and ribbing you for feeling the need to cheat), and the employers are sampling a more range-restricted population (stereotypically, 21- or 22-year-olds with 2:1 or 1st class quantitative degrees from impressive-sounding universities).
I don’t doubt that IQ tests can be useful, but I’m sceptical that one shorter than 10 minutes can be so robust as to make anything else (save a test for willingness to submission) unnecessary.
I was speaking more to your Campbelling point than JM’s shortness point. I’ll quickly address the latter now.
I’ve seen papers mention developing or using short versions of IQ tests that give scores correlating well (r > 0.8) with scores on the original full tests. I don’t know whether these “short” tests last <10 minutes, but it wouldn’t surprise me. Also, the US General Social Survey has a 10-item (I think?) vocabulary test that works as a passable proxy for IQ, even though it must take only 5 minutes or so to administer. Given these two existing kinds of test, I wouldn’t be surprised if a 9-minute IQ test had enough reliability to be almost as good as a full-length IQ test.
Note that JM wasn’t saying his two suggested test types would make other tests unnecessary. I agree with you that short tests for IQ & submissiveness wouldn’t make all other predictors of job performance superfluous. (Conscientiousness tests & structured interviews, if I remember rightly, both predict job performance even after controlling for IQ.) JM just said they’d be better predictors than anything else, which wouldn’t surprise me much. AFAIK, a full-length IQ test is the best general predictor of job performance, and the same would presumably be true for a 9-minute IQ test were it reliable enough (q.v. the previous paragraph).
[Edit: Christ, I really like that “wouldn’t surprise me” phrase, don’t I?]
That sounds even more amenable to that problem (if the items are the same in all editions of the test): if employers started to use such a test to sort prospective employees, within a year applicants would probably start memorizing those particular 10 words and >80% of them would ace the test.
The words are not always the same. They are selected from a standard wordbank.
From what I’ve seen in certain similar situations, some people would prepare for such a test by trying to memorize every single word in the wordbank, even if there are several thousands of them; OTOH, if the bank is large enough, then the extent to which they’d manage to do that would probably correlate with IQ strongly enough. (Unless some candidates use SRSs and other don’t.)
People have already pointed out that IQ tests are considered to be illegal discrimination, and have been ignoring that, but I think a good case can be made that it is proper to treat them as illegal discrimination. Imagine that a test genuinely detects ability to perform at the job but is also correlated with race independently of whether people of that race can do the job. Under those circumstances, if you use an IQ test to test employees, you’ll 1) avoid stupid employees and 2) avoid employees of one race in favor of equally competent people of another.
The overall result will be beneficial to you, since avoiding stupid employees benefits you while avoiding employees of one race has neutral effects on you. But the overall effect on people of that race will be devastating.
This will never be corrected by the market. The employers, after all, benefit unless there are so few available employees that cutting out the good ones of the wrong race reduces the pool of potential employees enough to affect the overall employee quality to a significant degree. How could we prevent this other than by banning IQ tests?
This ‘devastating’ effect sounds drastic. It also seems to directly require the assumption that the performance of people of the particular race perform devastatingly poor on IQ tests. That’s a rather dubious assumption and when asked to ‘imagine’ a world where this idea is so, any conclusions we draw only apply in that imaginary universe, not this one. This is not a ‘good case’.
When arguing that IQ tests should be illegal discrimination due to drastic racial bias of IQ tests the reasoning from the premise to conclusion is straightforward. The bulk of a ‘good case’ must be in providing evidence for the premise. Maybe there is sufficient evidence to make that claim about IQ tests but imagining it is not sufficient.
No, it doesn’t. It only requires the assumption that 1) such people score slightly more poorly, and 2) employers begin choosing employees from the tail end of the distribution. A slight difference in scores on the test will reduce the percentage at the tail end out of proportion to the size of the difference. I don’t remember the formal name for this, but it’s very well known.
Edit: Also, consider what percentage of the population needs to be unemployed for the problem to be considered serious. 10% is bad. If test bias causes an additional 5 or 10% to be unemployed, that’s going to be a huge effect even if 5% or 10% is only a small part of the entire population. Nobody says that the unemployment rate needs to get up to 25% or 50% in order for unemployment to be a serious problem.
I will not quibble about what constitutes ‘devastatingly poor’ beyond observing that I of course agree that the difference need not be numerically large in order to be devastating. I will again point out that you are assuming that there is sufficient bias in the IQ test that if used (instead of whatever other form of measurement or selection could be adopted) the change will be devastating. That is a significant claim, particularly when it is quite possible that subjective evaluations of intelligence by interviewers are more biased than IQ tests. I expect IQ may be somewhat racially biased. I know for certain that unstructured human arbitration is racially biased. I don’t know for sure to which is worse but do observe that it is sufficiently controversial that some evidence is required.
You asked the reader to imagine a world where some assumption holds—which is fine as far as it goes. It becomes an error in reasoning when you jump from “imagine...” to “a good case” without providing evidence that the imaginary scenario applies to reality. Even the bare statement “I assert that IQ tests are more racially biased than whatever is used in their stead” would have made the case at least coherent, albeit still weak.
The argument was in response to the implicit assumption that even if the tests are biased, we shouldn’t worry about that as long as they can predict performance. “Imagine” takes the place of the word “if” in there and the intended conclusion is “yes, you should worry about whether the tests are biased, because if they are biased, that would be bad.” To make that conclusion I do not need to provide evidence that the imaginary scenario applies to reality.
If I say “imagine that you jumped off a cliff. You’ll get smashed. Maybe you should avoid jumping off cliffs”, I don’t need to provide evidence that you jump off cliffs, because the conclusion is in the form of a conditional that already conditions on whether the imaginary scenario is real.