The idea of programming as a gear is still controversial, but the specific hypothesized gear is that people who can build a consistent model of a language will be successful at programming, whereas those who can’t won’t be. This was tested by giving students a test on Java before they had been taught java; their answers were checked, not for correctness, but for consistency. See “The Camel Has Two Humps.” Even then the test is far from perfectly predictive- ~28% of the consistent group failed, ~19% of the inconsistent group passed, and membership in the groups as indicated by the tests assigned shifted over time. If you do want to test this, you can reuse the original test.
When we began this work we had high hopes that we
had found a test that could be used as an admissions
filter to reduce the regrettable waste of human effort
and enthusiasm caused by high failure rates in universities’ first programming courses. We can see from
the experiments reported above that our test doesn’t
work if the intake is already experienced, and in experiment 3 didn’t work at all. We cannot claim to
be separating the programming goats from the non-programming sheep: experiment 3 demolishes the notion that consistent subjects will for the most part
learn well, and others for the most part won’t. And
even in the most encouraging of our results, we find a
50% success rate in those who don’t score C0 or CM2 [ie those who were inconsistent].
None the less, some of our results indicate that
there may be something going on with consistency.
It irritates me to no end that the original study is so much better known than the utter failure to replicate. I have to suspect that this has something to do with how conveniently it fits many programmers’ notion that programmers are a special sort of person, possessed of some power beyond merely a lot of practice at programming and related skills.
I feel much the same way about dual n-back studies. There was an article this month or last about WM training with Jaeggi and Buschkuel as authors… and it mentioned not a single issue. Gah!
I think that interesting results which fail to replicate are almost always better-known than the failure to replicate. I think it’s a fundamental problem of science, rather than a special weakness of programmers.
The idea of programming as a gear is still controversial, but the specific hypothesized gear is that people who can build a consistent model of a language will be successful at programming, whereas those who can’t won’t be. This was tested by giving students a test on Java before they had been taught java; their answers were checked, not for correctness, but for consistency. See “The Camel Has Two Humps.” Even then the test is far from perfectly predictive- ~28% of the consistent group failed, ~19% of the inconsistent group passed, and membership in the groups as indicated by the tests assigned shifted over time. If you do want to test this, you can reuse the original test.
However, there have been numerous attempted replications, none of which succeeded- though none found a negative result either. They were generally either confounded by the presence of experienced programmers, setup poorly, or not statistically significant. To quote the original authors:
HT Gwern
It irritates me to no end that the original study is so much better known than the utter failure to replicate. I have to suspect that this has something to do with how conveniently it fits many programmers’ notion that programmers are a special sort of person, possessed of some power beyond merely a lot of practice at programming and related skills.
I feel much the same way about dual n-back studies. There was an article this month or last about WM training with Jaeggi and Buschkuel as authors… and it mentioned not a single issue. Gah!
The more recent meta-analysis appears to support their initial conclusion.
I think that interesting results which fail to replicate are almost always better-known than the failure to replicate. I think it’s a fundamental problem of science, rather than a special weakness of programmers.