I mean this not only in the sense of them coming all kinds of shapes, colours and sizes, having different world views and upbringings attached to them, but also in the sense of them having different psychological, neurological and cultural makeup. It does not sound like something that needs to explicitly said but apparently it needs to be said.
Of course first voices have realised that the usual population for studies is WEIRD but the problem goes deeper and further. Even if the conscientious scientist uses larger populations, more representative for the problem at hand, the conclusions drawn tend to ignore human diversity.
One of the culprits is the concept of “average” or at least a misuse of it. The average person has an ovary and a testicle. Completely meaningless to say, yet we are comfortable in hearing statements like “going to college raises your expected income by 70%” (number made up) and off to college we go. Statements like these suppress a great deal of relevant information, namely the underlying, inherent diversity in the population. Going to college may increase lifetime earnings, but the size of this effect might be highly dependent on some other factor like inherent cognitive ability and choice of major.
Now that is obvious, you might say, but virtually all research shows that this is not the case. It was surprising to see that the camel has two humps, that is, one part of the population seems to be incapable of learning programming, while the other is. And this can be determined by the answer to a single question. Research on exercise and diet is massively convoluted with questions about endurance/strength and carbs/fats. May this be because of ignoring underlying biological factors?
People are touting the coming age of personalised medicine as they see massively diminishing returns on generic medicine. Ever more diseases are hypothesised to have very specific causes for each person necessitating ever more specialised treatment. The effects of psychedelic substances are found to be dependant on the exact psychological makeup, e.g. cannabis causing psychosis only in individuals already at risk for such episodes.
There is no exact point to this rant. Just the observation that ever more statements are similar to saying “having unprotected sex with your partner has a high probability of leading to pregnancy” to homosexual man.
It was surprising to see that the camel has two humps, that is, one part of the population seems to be incapable of learning programming, while the other is.
The study you’re probably thinking of failed to replicate with a larger sample size. While success at learning to code can be predicted somewhat, the discrepancies are not that strong.
The researcher didn’t distinguish the conjectured cause (bimodal differences in students’ ability to form models of computation) from other possible causes. (Just to name one: some students are more confident; confident students respond more consistently rather than hedging their answers; and teachers of computing tend to reward confidence).
Clearly further research is needed. It should probably not assume that programmers are magic special people, no matter how appealing that notion is to many programmers.
The failure to replicate was of their test, not of the initial observation. Specifically it was considered interesting why the distribution of grades in CS (apparently typically two-humped) was different from eg mathematics (apparently typically one-humped). As far as I know this still remains to be explained.
(a) The concept of averaging. There is nothing wrong with averages. People here like maximizing expected utility, which is an average. “Effects” are typically expressed as averages, but we can also look at distribution shapes, for instance. However, it’s important not to average garbage.
(b) The fact that population effects and subpopulation effects can differ. This is true, and not surprising. If we are careful about what effects we are talking about, Simpson’s paradox stops being a paradox.
(c) The fact that we should worry about confounders. Full agreement here! Confounders are a problem.
I think one big problem is just the lack of basic awareness of causal issues on the part of the general population (bad), scientific journalists (worse!), and sometimes even folks who do data analysis (extremely super double-plus awful!). Thus much garbage advice gets generated, and much of this garbage advice gets followed, or becomes conventional wisdom somehow.
That depends. Mostly they are used as single-point summaries of distributions and in this role they can be fine but can also be misleading or downright ridiculous. The problem is that unless you have some idea of the distribution shape, you don’t know whether the mean you’re looking at is fine or ridiculous. And, of course, the mean is expressly NOT a robust measure.
Humans are diverse.
I mean this not only in the sense of them coming all kinds of shapes, colours and sizes, having different world views and upbringings attached to them, but also in the sense of them having different psychological, neurological and cultural makeup. It does not sound like something that needs to explicitly said but apparently it needs to be said.
Of course first voices have realised that the usual population for studies is WEIRD but the problem goes deeper and further. Even if the conscientious scientist uses larger populations, more representative for the problem at hand, the conclusions drawn tend to ignore human diversity.
One of the culprits is the concept of “average” or at least a misuse of it. The average person has an ovary and a testicle. Completely meaningless to say, yet we are comfortable in hearing statements like “going to college raises your expected income by 70%” (number made up) and off to college we go. Statements like these suppress a great deal of relevant information, namely the underlying, inherent diversity in the population. Going to college may increase lifetime earnings, but the size of this effect might be highly dependent on some other factor like inherent cognitive ability and choice of major.
Now that is obvious, you might say, but virtually all research shows that this is not the case. It was surprising to see that the camel has two humps, that is, one part of the population seems to be incapable of learning programming, while the other is. And this can be determined by the answer to a single question. Research on exercise and diet is massively convoluted with questions about endurance/strength and carbs/fats. May this be because of ignoring underlying biological factors?
People are touting the coming age of personalised medicine as they see massively diminishing returns on generic medicine. Ever more diseases are hypothesised to have very specific causes for each person necessitating ever more specialised treatment. The effects of psychedelic substances are found to be dependant on the exact psychological makeup, e.g. cannabis causing psychosis only in individuals already at risk for such episodes.
There is no exact point to this rant. Just the observation that ever more statements are similar to saying “having unprotected sex with your partner has a high probability of leading to pregnancy” to homosexual man.
The study you’re probably thinking of failed to replicate with a larger sample size. While success at learning to code can be predicted somewhat, the discrepancies are not that strong.
http://www.eis.mdx.ac.uk/research/PhDArea/saeed/
The researcher didn’t distinguish the conjectured cause (bimodal differences in students’ ability to form models of computation) from other possible causes. (Just to name one: some students are more confident; confident students respond more consistently rather than hedging their answers; and teachers of computing tend to reward confidence).
And the researcher’s advisor later described his enthusiasm for the study as “prescription-drug induced over-hyping” of the results …
Clearly further research is needed. It should probably not assume that programmers are magic special people, no matter how appealing that notion is to many programmers.
The failure to replicate was of their test, not of the initial observation. Specifically it was considered interesting why the distribution of grades in CS (apparently typically two-humped) was different from eg mathematics (apparently typically one-humped). As far as I know this still remains to be explained.
See also the comments of Yvain’s What Universal Human Experiences Are You Missing Without Realizing It? for a broad selection of examples of how human minds vary.
Oh, now I realized the point of that article was the comments, not the article itself. Thanks for clarifying this!
There are three separate issues:
(a) The concept of averaging. There is nothing wrong with averages. People here like maximizing expected utility, which is an average. “Effects” are typically expressed as averages, but we can also look at distribution shapes, for instance. However, it’s important not to average garbage.
(b) The fact that population effects and subpopulation effects can differ. This is true, and not surprising. If we are careful about what effects we are talking about, Simpson’s paradox stops being a paradox.
(c) The fact that we should worry about confounders. Full agreement here! Confounders are a problem.
I think one big problem is just the lack of basic awareness of causal issues on the part of the general population (bad), scientific journalists (worse!), and sometimes even folks who do data analysis (extremely super double-plus awful!). Thus much garbage advice gets generated, and much of this garbage advice gets followed, or becomes conventional wisdom somehow.
That depends. Mostly they are used as single-point summaries of distributions and in this role they can be fine but can also be misleading or downright ridiculous. The problem is that unless you have some idea of the distribution shape, you don’t know whether the mean you’re looking at is fine or ridiculous. And, of course, the mean is expressly NOT a robust measure.
The Eurythmics said it best:
I travel the world and the seven seas
Everybody’s looking for something
Some of them want to use you
Some of them want to get used by you
Some of them want to abuse you
Some of them want to be abused