I asked my father to read this and give his thoughts.
He says that positive selection only works well when you have a very good idea what you need to select for. If you’re sending an athlete to the Olympics but the event he’ll have to compete in will be chosen at random, you can’t just choose the one with the best time on the 800 meter dash, because the event might end up being something like archery, fencing, or weightlifting. And you certainly wouldn’t want to send a non-swimmer. If you need a generalist, seeing how well someone does at jumping through a wide variety of arbitrary hoops might really be the best test you can practically implement.
(Now I’m wondering just how good or bad the 800 meter dash actually is at predicting levels of success at unrelated sports. For example, could you tell the difference between an NHL-quality ice hockey player and one that plays on a minor league team just by looking at their times on the 800 meter dash?)
Assuming a significantly large distribution of athletes sent by other rational managers, where all athletes are bound to the same rules of random event selection, I would still send the best possible specialist in a single discipline in this case, because without certainty that all other rational managers know certainly that some generalists will be better in everything than other generalists and that each one is confident that theirs is best, I conclude that some of them attempt a gamble of probabilities and send a specialist, and thus I also send a specialist to maximize my chances of winning.
After all, there are higher chances of the event being my athlete’s specialty than there are chances of every single other athlete being less good at it if I pick a generalist, unless the number of possible events is large enough to outweigh the number of athletes. Throw in irrational managers and the possibility of other managers having information unavailable to you, and your father’s argument seems very weak.
Now, of course, I’m probably attacking something that wasn’t meant to be a strong defensible argument. However, I feel very strongly about the point that negative selection is wrong in many contexts it is currently used in (which I support), as well as the point that positive selection is so difficult and utterly impractical in so many cases (which I want to pound into tiny bits of forgotten wrongness).
I’m not sure where I’m going with this, however. I strongly agree with the article’s statements, but my attempts to formulate any further useful thought seem to come up short.
Well, the sports analogy was my own interpretation of what he said.
Game theory question time: you and N other players are playing a dice rolling game. Each player has the choice of rolling a single twenty-sided die, or rolling five four-sided dice. The player with the highest total wins. (Ties are broken by eliminating all non-tying players and then playing again.) Now, rolling 5d4 has an expected score of 12.5 and rolling 1d20 has an expected score of 10.5, so when N=2, it’s obviously better to roll 5d4. However, when N becomes sufficiently large, someone is going to roll a 20, so it’s better to pick the 20-sided die, which gives you a 1 in 20 chance of rolling a 20 instead of a 1 in 1024 chance of getting five 4s. For exactly what value of N does it become better?
Edit: Fixed stupid math mistakes. That’ll teach me to post after staying up all night!
Insightful question, if you ask me, though solving for N feels a lot more like a straight up actuary-level math problem than Game Theory to me. My maths above basic calculus is generally foggy, so I’d appreciate any corrections or nitpicks someone more fluent here might have.
Essentially, you have to solve when (odds of having highest result when rolling d20) >= (odds of having highest result when rolling 5d4). To simplify, let’s assume that all players are perfectly rational, and thus at N and higher will all roll 1d20. This still leaves you the problem of calculating N’s odds of rolling higher than you for both rolls, which is a simpler reformulation of the above parentheses.
For any roll result Y, there is (y/20)^N probability that you “win” here, assuming ties count as wins (or at least are preferable to losses). This means that with N=1 (you’re playing against one other person), you will win 52.5% of the time (and so will your opponent, because that 2.5% is for ties) when rolling 1d20.
Your odds of winning naturally decrease if you roll 1d20 such that for N=2 you have 35.875% chances of winning, and so on in a proportional manner since the odds are always even for everyone.
Where it gets more interesting is when you are playing an unfair game where you have to equate your total odds of winning when playing 1d20 vs d20s to those when playing 5d4 vs d20s. Since the math here is kind of foggy and hard to combine into one big formula, I’ve thrown the data at a spreadsheet (to calculate the sum of the odds of any N rolling higher than you for each roll Y multiplied by your odds of obtaining Y), and it turns out that at N=3 the 5d4 roll dips just below the odds of winning with 1d20 by about 0.2%.
However, if we want to compute for xDf die for N, with K possible ways to roll (which was 2 here), then the math yet eludes me. I’ve figured it out or been told what it was several times, but I just can’t seem to ever memorize this when I can only barely remember integration anyway when I don’t use it.
Edit: For those curious, here’s the spreadsheet mentioned above with all the raw data and brute-force formulas.
Yes, the reward system is very important in choosing the right strategy. If the first place gives you gold, and all other places give you nothing, use positive selection. If the last places gives you problem, and all other places give you nothing, use negative selected. Other point of view: if being average is good, play safe by using negative selection; if being average is bad, aim for greatness (and accept a certain risk of failure) by using positive selection.
So the question is what exactly do we want in elite colleges or academia (examples from the article)? I guess for elite colleges it is better to play safe. If your students are above average and everyone knows it, they don’t have to be exceptional—your diploma will help them get a decent job, which is why they pay you. A few bad apples could ruin your marketing. With academia, for an average university it is probably better to have “safe” professors who do their jobs, get grants, and don’t cause scandals; even if the price is having less Nobel-price winners.
Yes, that it does, or at least it assumes that the difference is trivial within this decision scheme and the expected utility returns of a specialist are higher than the expected utility of a generalist even when taking second place into account.
I don’t think the right way to do this is not either positive or negative selection (those terms really suggest a false dichotomy, don’t they?). As has been pointed out elsewhere, what’s here being called “positive” is really “or”, and what’s here being called “negative” is really “and”.
But there are lots more ways to combine the data into a single number then just “apply a cutoff to each one, and then apply some operation to the resulting booleans”. The appropriate sort of selection is not positive or negative, but rather, whatever will be used in the actual competition. (And if it’s unknown, apply expected utility, etc.)
Now I’m wondering just how good or bad the 800 meter dash actually is at predicting levels of success at unrelated sports. For example, could you tell the difference between an NHL-quality ice hockey player and one that plays on a minor league team just by looking at their times on the 800 meter dash?
This (among other prior information) suggests to me that extreme levels of performance at different tests are probably negatively correlated, but I would not be surprised if there were events out there where extreme levels of performance on other tests are correlated with better (but not extreme) levels of performance on that test.
Thanks for the link! I think a careful statement of my claim avoids that fallacy.
The claimed data:
Body shape is relevant to performance in particular events (say, the 800 meter dash)- for example, Michael Phelps is shaped well for swimming.
Extreme performance in that event will generally require a body shape optimized for that event.
Extreme performance in different events correspond to different body shapes.
Stated carelessly, it seems likely to me that if I know you’re an Olympic-level athlete, I’ll have some estimate that you can play Olympic-level basketball; but then when you tell me that you’re a Olympic-level wrestler, I can lower my probability estimate that you would be able to play Olympic-level basketball.
But if I condition on knowing you’re an Olympic athlete, and then try to drop that condition without being careful, then I can get into trouble (this fallacy, specifically). So instead, let’s start off with some (really low) probability that a person chosen uniformly at random can play Olympic-level basketball, and then update on the knowledge that they’re an Olympic wrestler- we should increase our estimate based on their general level of athleticism, and then decrease our estimate based on their probable body shape. I think the effect of the body shape will be stronger than the effect of general athleticism, and so they will actually be negatively correlated.
I think my last statement in the grandparent- that extreme levels of performance on a specific test should correlate with better performance on some ‘general athleticism’ test- is true when you compare extreme individuals to random individuals, but less true (or perhaps not true) when comparing extreme individuals to good individuals. The NHL ice hockey player is probably not much more ‘athletic’ than a minor league hockey player, but probably is more ‘hockey-shaped.’
But now that I’ve stated that last paragraph, I’m thinking of counterevidence- like the famous birth month effect for Canadian hockey players. Early training appears to have a huge impact on whether or not someone hits extreme levels of performance, but I think both NHL players and minor league players started early. So maybe it is biological talent that differentiates many of them. Hmm. I should probably stop speculating about sports.
If you put an Olympic-level wrestler on a college (American) football team, how well would they do? Michael Jordan did tolerably well during his year as a minor league baseball player.
Tolerably well for minor league, but remember that his father had envisioned him as a major league baseball player, so presumably he’d practiced and done well at the sport when he was younger. There are probably selection effects on Michael Jordan in particular to be good at baseball out of the set of NBA players; most never try to transition to baseball at all.
Upvoted for this link. I wish I had known this term back in the Amanda Knox days—this fallacy (or rather, the reverse of it—failing to take into account conditional dependence of a priori independent events) is a version of the main probability-theoretic error of that case.
I asked my father to read this and give his thoughts.
He says that positive selection only works well when you have a very good idea what you need to select for. If you’re sending an athlete to the Olympics but the event he’ll have to compete in will be chosen at random, you can’t just choose the one with the best time on the 800 meter dash, because the event might end up being something like archery, fencing, or weightlifting. And you certainly wouldn’t want to send a non-swimmer. If you need a generalist, seeing how well someone does at jumping through a wide variety of arbitrary hoops might really be the best test you can practically implement.
(Now I’m wondering just how good or bad the 800 meter dash actually is at predicting levels of success at unrelated sports. For example, could you tell the difference between an NHL-quality ice hockey player and one that plays on a minor league team just by looking at their times on the 800 meter dash?)
Assuming a significantly large distribution of athletes sent by other rational managers, where all athletes are bound to the same rules of random event selection, I would still send the best possible specialist in a single discipline in this case, because without certainty that all other rational managers know certainly that some generalists will be better in everything than other generalists and that each one is confident that theirs is best, I conclude that some of them attempt a gamble of probabilities and send a specialist, and thus I also send a specialist to maximize my chances of winning.
After all, there are higher chances of the event being my athlete’s specialty than there are chances of every single other athlete being less good at it if I pick a generalist, unless the number of possible events is large enough to outweigh the number of athletes. Throw in irrational managers and the possibility of other managers having information unavailable to you, and your father’s argument seems very weak.
Now, of course, I’m probably attacking something that wasn’t meant to be a strong defensible argument. However, I feel very strongly about the point that negative selection is wrong in many contexts it is currently used in (which I support), as well as the point that positive selection is so difficult and utterly impractical in so many cases (which I want to pound into tiny bits of forgotten wrongness).
I’m not sure where I’m going with this, however. I strongly agree with the article’s statements, but my attempts to formulate any further useful thought seem to come up short.
Well, the sports analogy was my own interpretation of what he said.
Game theory question time: you and N other players are playing a dice rolling game. Each player has the choice of rolling a single twenty-sided die, or rolling five four-sided dice. The player with the highest total wins. (Ties are broken by eliminating all non-tying players and then playing again.) Now, rolling 5d4 has an expected score of 12.5 and rolling 1d20 has an expected score of 10.5, so when N=2, it’s obviously better to roll 5d4. However, when N becomes sufficiently large, someone is going to roll a 20, so it’s better to pick the 20-sided die, which gives you a 1 in 20 chance of rolling a 20 instead of a 1 in 1024 chance of getting five 4s. For exactly what value of N does it become better?
Edit: Fixed stupid math mistakes. That’ll teach me to post after staying up all night!
Fixed, thanks.
4^5 = 2^10 = 1024
Fixed, thanks.
Insightful question, if you ask me, though solving for N feels a lot more like a straight up actuary-level math problem than Game Theory to me. My maths above basic calculus is generally foggy, so I’d appreciate any corrections or nitpicks someone more fluent here might have.
Essentially, you have to solve when (odds of having highest result when rolling d20) >= (odds of having highest result when rolling 5d4). To simplify, let’s assume that all players are perfectly rational, and thus at N and higher will all roll 1d20. This still leaves you the problem of calculating N’s odds of rolling higher than you for both rolls, which is a simpler reformulation of the above parentheses.
For any roll result Y, there is (y/20)^N probability that you “win” here, assuming ties count as wins (or at least are preferable to losses). This means that with N=1 (you’re playing against one other person), you will win 52.5% of the time (and so will your opponent, because that 2.5% is for ties) when rolling 1d20.
Your odds of winning naturally decrease if you roll 1d20 such that for N=2 you have 35.875% chances of winning, and so on in a proportional manner since the odds are always even for everyone.
Where it gets more interesting is when you are playing an unfair game where you have to equate your total odds of winning when playing 1d20 vs d20s to those when playing 5d4 vs d20s. Since the math here is kind of foggy and hard to combine into one big formula, I’ve thrown the data at a spreadsheet (to calculate the sum of the odds of any N rolling higher than you for each roll Y multiplied by your odds of obtaining Y), and it turns out that at N=3 the 5d4 roll dips just below the odds of winning with 1d20 by about 0.2%.
However, if we want to compute for xDf die for N, with K possible ways to roll (which was 2 here), then the math yet eludes me. I’ve figured it out or been told what it was several times, but I just can’t seem to ever memorize this when I can only barely remember integration anyway when I don’t use it.
Edit: For those curious, here’s the spreadsheet mentioned above with all the raw data and brute-force formulas.
Your analysis also assumes there’s no difference between second place and last place.
Yes, the reward system is very important in choosing the right strategy. If the first place gives you gold, and all other places give you nothing, use positive selection. If the last places gives you problem, and all other places give you nothing, use negative selected. Other point of view: if being average is good, play safe by using negative selection; if being average is bad, aim for greatness (and accept a certain risk of failure) by using positive selection.
So the question is what exactly do we want in elite colleges or academia (examples from the article)? I guess for elite colleges it is better to play safe. If your students are above average and everyone knows it, they don’t have to be exceptional—your diploma will help them get a decent job, which is why they pay you. A few bad apples could ruin your marketing. With academia, for an average university it is probably better to have “safe” professors who do their jobs, get grants, and don’t cause scandals; even if the price is having less Nobel-price winners.
Yes, that it does, or at least it assumes that the difference is trivial within this decision scheme and the expected utility returns of a specialist are higher than the expected utility of a generalist even when taking second place into account.
I don’t think the right way to do this is not either positive or negative selection (those terms really suggest a false dichotomy, don’t they?). As has been pointed out elsewhere, what’s here being called “positive” is really “or”, and what’s here being called “negative” is really “and”.
But there are lots more ways to combine the data into a single number then just “apply a cutoff to each one, and then apply some operation to the resulting booleans”. The appropriate sort of selection is not positive or negative, but rather, whatever will be used in the actual competition. (And if it’s unknown, apply expected utility, etc.)
This (among other prior information) suggests to me that extreme levels of performance at different tests are probably negatively correlated, but I would not be surprised if there were events out there where extreme levels of performance on other tests are correlated with better (but not extreme) levels of performance on that test.
Unless you sample at random from the whole population, that’s a Berkson’s fallacy.
Thanks for the link! I think a careful statement of my claim avoids that fallacy.
The claimed data:
Body shape is relevant to performance in particular events (say, the 800 meter dash)- for example, Michael Phelps is shaped well for swimming.
Extreme performance in that event will generally require a body shape optimized for that event.
Extreme performance in different events correspond to different body shapes.
Stated carelessly, it seems likely to me that if I know you’re an Olympic-level athlete, I’ll have some estimate that you can play Olympic-level basketball; but then when you tell me that you’re a Olympic-level wrestler, I can lower my probability estimate that you would be able to play Olympic-level basketball.
But if I condition on knowing you’re an Olympic athlete, and then try to drop that condition without being careful, then I can get into trouble (this fallacy, specifically). So instead, let’s start off with some (really low) probability that a person chosen uniformly at random can play Olympic-level basketball, and then update on the knowledge that they’re an Olympic wrestler- we should increase our estimate based on their general level of athleticism, and then decrease our estimate based on their probable body shape. I think the effect of the body shape will be stronger than the effect of general athleticism, and so they will actually be negatively correlated.
I think my last statement in the grandparent- that extreme levels of performance on a specific test should correlate with better performance on some ‘general athleticism’ test- is true when you compare extreme individuals to random individuals, but less true (or perhaps not true) when comparing extreme individuals to good individuals. The NHL ice hockey player is probably not much more ‘athletic’ than a minor league hockey player, but probably is more ‘hockey-shaped.’
But now that I’ve stated that last paragraph, I’m thinking of counterevidence- like the famous birth month effect for Canadian hockey players. Early training appears to have a huge impact on whether or not someone hits extreme levels of performance, but I think both NHL players and minor league players started early. So maybe it is biological talent that differentiates many of them. Hmm. I should probably stop speculating about sports.
If you put an Olympic-level wrestler on a college (American) football team, how well would they do? Michael Jordan did tolerably well during his year as a minor league baseball player.
Tolerably well for minor league, but remember that his father had envisioned him as a major league baseball player, so presumably he’d practiced and done well at the sport when he was younger. There are probably selection effects on Michael Jordan in particular to be good at baseball out of the set of NBA players; most never try to transition to baseball at all.
Upvoted for this link. I wish I had known this term back in the Amanda Knox days—this fallacy (or rather, the reverse of it—failing to take into account conditional dependence of a priori independent events) is a version of the main probability-theoretic error of that case.
This fallacy may also explain why people tend to assign 50% percent probability to the odds of the second child being a boy in the classic puzzle.