Sure, but (a) correlation isn’t transitive, and (b) some very smart impressive people do poorly on metrics like IQ tests.
I think cumulative lifetime output is better than IQ. Yes the former isn’t directly comparable across cohorts, and there are a million other complications. But trivial metrics like IQ seem to me to be looking under a streetlight.
Google did some experiments on measurable ways to do interviews (puzzles, etc.) and found no effect on hire quality. Undue insistence on proxies over real parameters is a failure mode, imo.
Google did some experiments on measurable ways to do interviews (puzzles, etc.) and found no effect on hire quality.
Unsurprising due to range restriction—by the time you’re interviewing with Google, you’ve gone through tons of filters (especially if you’re a Stanford grad). This is the same reason that when people look at samples of elite scientists, IQ tends to not be as important a factor as one would expect—because they’re all smart—and other things like personality factors start to correlate more.
I am just saying that for people who are capable of doing more than flipping burgers (which probably starts well before a single sigma out from the mean), we should just look at what they did.
This approach has the advantage of not counting highly the kind of people who may place well on tests, etc. due to good hardware, but who, due to poor habits or whatever other reason, end up not living up to their potential.
Similarly, this approach highlights that creative output is often not comparable. Is Van Gogh “better” than Shakespeare? A silly question.
I don’t disagree that IQ tests are useful for some things for folks within a sigma of the mean, and I also agree with the consensus that tests start to fail for smart folks, and we need better models then.
If the average IQ of LW is really around 140, then I think we should talk about the neat things we have done, and not the average IQ of LW. :)
Tests are often used to decide what to allow people to do, so you can’t rely on what they’ve done already. When testing applicants to college, they don’t often have a significant history of doing.
Google did some experiments on measurable ways to do interviews (puzzles, etc.) and found no effect on hire quality.
But they only hire at the top, so one would expect the subsequent performance of their hires to be little correlated with any sort of interview assessments.
Toy example: 0.8 correlation between two variables, select on one at 3 or more s.d.s above the mean, correlation within that subpopulation is around 0.2 to 0.45 (it varies a lot, even in a sample of 100000).
Of course outliers exist, they’re the exceptions that demonstrate the rule.
Besides, how do you even define “cumulative lifetime output”? Galton tried doing this at first, then realized it was impossible to make rigorous, which led to his proto-IQ tests in the first place.
I think if the real parameter is hard to measure or is maybe actually multiple parameters the correct answer is to think about modeling harder, not to insist that a dumb model is what we should use.
In less quantititative fields that need to be a little quantitative to publish they have a silly habit of slapping a linear regression model on their problem and calling it a day.
Besides, how do you even define “cumulative lifetime output”?
Papers, books, paintings, creating output? Do you think Van Gogh and his ilk would do well on an IQ test?
Most people, unlike you (according to your name, at least), are not paper machines.
Someone who works in a large department that values number of papers published and number of grants secured but doesn’t particularly care about quality of work, and so publishes four papers a year of poor quality, which are occasionally cited, but only by direct colleagues, vs. Douglas Hoftstadter, who rarely publishes anything but whose first work has been immensely influential, you’re going to get a worse picture than if you had just used IQ.
That’s not really useful. I don’t even know in which context are we talking about these things. Is it about hiring someone? Is it about deciding on whether someone’s work is “worthy” to be in a museum, or published, or something else? It is about admitting people to your social circle? Is it about generally ranking all people on some scale just because we can?
Sure, but (a) correlation isn’t transitive, and (b) some very smart impressive people do poorly on metrics like IQ tests.
I think cumulative lifetime output is better than IQ. Yes the former isn’t directly comparable across cohorts, and there are a million other complications. But trivial metrics like IQ seem to me to be looking under a streetlight.
Google did some experiments on measurable ways to do interviews (puzzles, etc.) and found no effect on hire quality. Undue insistence on proxies over real parameters is a failure mode, imo.
Unsurprising due to range restriction—by the time you’re interviewing with Google, you’ve gone through tons of filters (especially if you’re a Stanford grad). This is the same reason that when people look at samples of elite scientists, IQ tends to not be as important a factor as one would expect—because they’re all smart—and other things like personality factors start to correlate more.
EDIT: this may be related to Spearman’s law of diminishing returns
I am just saying that for people who are capable of doing more than flipping burgers (which probably starts well before a single sigma out from the mean), we should just look at what they did.
This approach has the advantage of not counting highly the kind of people who may place well on tests, etc. due to good hardware, but who, due to poor habits or whatever other reason, end up not living up to their potential.
Similarly, this approach highlights that creative output is often not comparable. Is Van Gogh “better” than Shakespeare? A silly question.
I don’t disagree that IQ tests are useful for some things for folks within a sigma of the mean, and I also agree with the consensus that tests start to fail for smart folks, and we need better models then.
If the average IQ of LW is really around 140, then I think we should talk about the neat things we have done, and not the average IQ of LW. :)
Tests are often used to decide what to allow people to do, so you can’t rely on what they’ve done already. When testing applicants to college, they don’t often have a significant history of doing.
But they only hire at the top, so one would expect the subsequent performance of their hires to be little correlated with any sort of interview assessments.
Toy example: 0.8 correlation between two variables, select on one at 3 or more s.d.s above the mean, correlation within that subpopulation is around 0.2 to 0.45 (it varies a lot, even in a sample of 100000).
Of course outliers exist, they’re the exceptions that demonstrate the rule.
Besides, how do you even define “cumulative lifetime output”? Galton tried doing this at first, then realized it was impossible to make rigorous, which led to his proto-IQ tests in the first place.
I think if the real parameter is hard to measure or is maybe actually multiple parameters the correct answer is to think about modeling harder, not to insist that a dumb model is what we should use.
In less quantititative fields that need to be a little quantitative to publish they have a silly habit of slapping a linear regression model on their problem and calling it a day.
Papers, books, paintings, creating output? Do you think Van Gogh and his ilk would do well on an IQ test?
Cumulative lifetime output doesn’t seem very useful, though. For one thing, it’s only measurable for dead or near-dead people...
???
Cumulative just means “what you have done so far.”
You’re right, of course. Nevermind. Though the problem of measuring it for someone who hasn’t yet had the chance to do much remains.
Expected cumulative lifetime output, then.
Two papers per year * 30 years of productive career = 60 papers.… :-(
Most people, unlike you (according to your name, at least), are not paper machines.
Someone who works in a large department that values number of papers published and number of grants secured but doesn’t particularly care about quality of work, and so publishes four papers a year of poor quality, which are occasionally cited, but only by direct colleagues, vs. Douglas Hoftstadter, who rarely publishes anything but whose first work has been immensely influential, you’re going to get a worse picture than if you had just used IQ.
Heh, I suppose that is one of the alternative readings of my handle.
Only four? Why, I know some (who will remain nameless) that published eight or ten papers last year alone.
But of course Goodhart’s law ruins everything.
For which purpose?
For determining if someone is a giant, or a midget in giant’s clothing.
That’s not really useful. I don’t even know in which context are we talking about these things. Is it about hiring someone? Is it about deciding on whether someone’s work is “worthy” to be in a museum, or published, or something else? It is about admitting people to your social circle? Is it about generally ranking all people on some scale just because we can?
But that’s a test you can only run once somebody is dead, so it’s not very useful.