Social scientists are experts at having an explanation for the results, no matter how they come out, so long as they are statistically significant. It is easy to turn the statistical crank. With sufficiently powerful statistical tools, you can find a significant statistical relationship between just about anything and anything else.
Psychologists see no real problem with the current dogma. They are used to getting messy results that can be dealt with only by statistics. In fact, I have now detected a positive suspicion of quality results amongst psychologists. In my experiments I get relationships between variables that are predictable to within 1 percent accuracy. The response to this level of perfection has been that the results must be trivial! It was even suggested to me that I use procedures that would reduce the quality of the results, the implication being that noisier data would mean more.
After some recovery period I realized that this attitude is to be expected from anyone trying to see the failure of the input-output model as a success. Social scientists are used to accounting for perhaps 80% (at most) of the variance in their data. They then look for other variables that will account for more variance. This is what gives them future research studies. The premise is that behavior is caused by many variables. If I account for all the variance with just one variable, it’s no fun and seems trivial.
If psychologists had been around at the time that physics was getting started, we’d still be Aristotelian, or worse. There would be many studies looking for relationships between one physical variable and another—e.g., between ball color and rate of fall, or between type of surface and the amount of snow in the driveway. Some of these relationships would prove statistically significant. Then when some guy comes along and shows that there is a nearly perfect linear relationship between distance traveled and acceleration, there would be a big heave of “trivial” or “too limited”—what does this have to do with the problems we have keeping snow out of the driveway?
Few psychologists recognize that, whatever their theory, it is based on the open-loop input-output model. There is no realization that the very methods by which data are collected imply that you are dealing with an open-loop system. To most psychologists, the methods of doing research are simply the scientific method—the only alternative is superstition. There is certainly no realization that the input-output model is testable and could be shown to be false. In fact, the methods are borrowed, in caricature, from the natural sciences, where the open-loop model works very well, thank you. Progress in the natural sciences began dramatically when it was realized that the inanimate world is not purposive.
Psychologists have mistakenly applied this model of the inanimate world to the animate world, where it simply does not apply.
This was a forgivable mistake in the days before control theory, because before 1948 there was no understanding of how purposive behavior could work. Now we know, but the social sciences have their feet sunk in conceptual concrete. They simply won’t give up what, to them, simply means science.
The problem is that Marken’s models don’t actually have predictive power; he just fits a function to the data using as many free parameters as he has data points, and marvels at the perfect fit thus derived. One doesn’t need to think highly of the current state of psychology to realize that Marken is a crank, and that any recognition Marken has in the PCT community is a sign that they are bereft of actual experimental support if not basic scientific reasoning skills.
The interaction you linked to was interesting. I didn’t realize there was already a back story within this community with positions staked out and such. I offered the quote because it seemed like a beautifully mathematical objection to existing work that was “up this community’s alley” but I haven’t worked into the actual mathematics or experiments themselves. For example, I hadn’t purchased either of the books that I linked to, not have I studied them—I simply assigned them high EV given the quality of the author’s text.
Your comments, in the interaction you linked to, seem like a good arguments against Marken’s theory (specifically the claim that his work involves more free parameters than data points appears to be a good argument against the theory, if true). However, in all of that back and forth, I noticed many links to “lesswrong heuristics” but I didn’t notice any outside links to an actual research papers detailing methodology.
I’m substantially more ignorant on the subject than either you or your previous interlocutor and it took me a while to even understand that “PCT” was the theory Marken supports, that you two were taking the pro and con towards it, that your text was mostly between each other with a substantial amount of knowledge assumed. I wish you had both linked more, because it would have been educational.
That said, I’d like to see such links if you know of any. If I can swiftly dismiss Marken’s work without further thought, that would be a very efficient use of time. Can you direct me to the links showing an example of his experimental work so I can verify that his research program is crippled by mathematical overfitting? The best I could find was Perceptual organization of behavior: A hierarchical control model of coordinated action but it was pay-walled so I can’t access it now to look into it myself.
The paper discussed in that interaction can be found here without a paywall.
As stated then (the conversation can be taken up from about here if not earlier), I think it’s quite likely that simple control circuits can be found in facets of motor response; but Powers, Marken and Eby had been talking about control theory in cognitive domains (like akrasia) as if they could isolate simple circuits there, and my search for any kind of evidence turned up only this sort of embarrassing tripe.
And really, the math here is important— it’s not a matter of disagreeing with interpretation, it’s the plain fact that a generic model with 4 free parameters can be tweaked to precisely fit 4 data points, and it’s clear from the paper that this is what Marken did. You simply need more data points than free parameters in order to generate any evidence in favor of a model; the fact that he never mentioned this, and instead crowed about the impressive fit of his model to the data, indicate either gross ignorance of how mathematical models work, or outright intent to mislead (coupled with an utterly incompetent peer review process.)
The gauntlet remains thrown, if anyone wants to point to an experimental study which demonstrates a discernible control circuit in a cognitive task (apart from tasks, like tracking a dot, which have an obvious motor component— in these, I do expect control circuits to be a good model for certain behavior). I would be surprised, but it would suffice to give credence to the theory in my eyes.
Thanks Robin! I have read this paper now, but it still doesn’t seem to address the arguments that orthonormal linked to :-/
The 1986 study appeared to me to be basically well done, offering a fascinating paradigm that could be extended in many directions for further research with a reasonably strong result by itself. It basically confirmed the positive claims of Marken that hierarchical arrangements of negative feedback loop systems (designed, with a handful of optimized parameters, and then left alone) can roughly reproduce trained human behavior in a variety of dynamically changing toy domains, supporting the contention that whatever is operating in the human nervous system after a period of training is doing roughly the same effective computations as the model.
In the text, Merken addresses the “motor control literature” as making claims whose refutation was partly the purpose of his experiments.
It required a little more googling to figure out the claims he was trying to reject… but basically he seems to be objecting to the claim that mammals work as open loop controllers (that is, generating action signals based on an internal model of the world that are sent into the world with no verification loop or secondary corrections). This claim appears to have been founded mostly on things called “deafferentiation experiments”… which turned out to be aesthetically horrifying and also turned out to not actually prove the general case of “open loop” claims.
a psychologist, who had cut afferent ganglia that supplied sensation to the brain from their arms and legs, then used restraint, electric shock, and withholding of food to force [the monkeys] to use the limbs they could not feel.
The ability of monkeys mutilated in this fashion to (eventually?) move around purposively was taken as evidence that there was not a hierarchically arranged set of negative feedback motor control systems implemented in their nervous system. In practice (after the scientist was arrested for animal cruelty, PETA’s request for custody was denied, and the monkeys were brainscanned, euthanized, and autopsied) it turned out that the monkey’s brains had been massively re-wired by the experience. The practical upshot of the experiments seem to have primarily been to serve as dramatic evidence of adult primate brain plasticity (which they didn’t believe in, back then?) rather than as confirmation of a negative feedback theory of motor control. (Probably there’s more to it than that, but this is my first draft understanding.)
Merken dismisses these experiments in part by pointing out the difficulty of preventing negative feedback control processes if there are many sub controllers that can use measurements partially correlated to the measure being optimized and concludes with falsification examples and criteria for the general theory and the particular model that are not subject to this objection:
Control theory does not rule out the possibility that some of the behavior produced by organisms is open loop. If a behavior is open loop then the effects of disturbances will not be resisted by system outputs. There is evidence, for example, that the saccadic eye positioning system is open loop. However, open-loop behavior is likely to be exceptionally rare …The two-level model could be tested by other means. One suggestion is to look for differences in the time to react to disturbances applied to controlled variables that are presumed to be at different levels—the higher the level of the variable, the slower the response to the effect of a disturbance. Another approach would be to trace out the required connections anatomically. The present study shows only that the two-level model is sufficient, not that it is necessary.
In short, I’m still impressed by Merken. His reasoning seems clean, his experiment, robust, his criticisms of motor-control and trait-theory, well reasoned. My very broad impression is that there may be a over-arching background argument here between “accurate model in the head producing aim and fire success” versus “incremental goal accomplishment via well tuned reflexes and continuous effort”? If that back story is operative then I guess my posterior probability was just pushed a little more in the direction of “reflexes and effort” over “models and plans”.
If there is some trick still lurking here, Orthonormal, that you could point me to and spell out in detail rather than by reference to assertions and hand-waving rationality heuristics, that would be appreciated. The more time I spend on Merken’s work, the more I find to appreciate. At this point, I’ve spend a day or two on this and I think the burden of proof is on you. If you take it up successfully I would be in your debt for rubbing a bit of sand out of my eyes :-)
I don’t understand what the quote is trying to say. What are the unrecognized consequences of the open-loop model?
It sounds like the author is upset that psychologists don’t believe he has a model of behavior that explains 99% of some output variable using only one input variable. I’d have a hard time believing too.
Richard Marken in a May 15, 1985 lecture titled “Teaching Dogma in Psychology”
The author was transformed by reading “Behavior: The Control of Perception”(1973) and began a research program whose early years(?) seem to have been summarized in “Mind Readings: Experimental Studies of Purpose”(1992)
This has been discussed here before.
The problem is that Marken’s models don’t actually have predictive power; he just fits a function to the data using as many free parameters as he has data points, and marvels at the perfect fit thus derived. One doesn’t need to think highly of the current state of psychology to realize that Marken is a crank, and that any recognition Marken has in the PCT community is a sign that they are bereft of actual experimental support if not basic scientific reasoning skills.
The interaction you linked to was interesting. I didn’t realize there was already a back story within this community with positions staked out and such. I offered the quote because it seemed like a beautifully mathematical objection to existing work that was “up this community’s alley” but I haven’t worked into the actual mathematics or experiments themselves. For example, I hadn’t purchased either of the books that I linked to, not have I studied them—I simply assigned them high EV given the quality of the author’s text.
Your comments, in the interaction you linked to, seem like a good arguments against Marken’s theory (specifically the claim that his work involves more free parameters than data points appears to be a good argument against the theory, if true). However, in all of that back and forth, I noticed many links to “lesswrong heuristics” but I didn’t notice any outside links to an actual research papers detailing methodology.
I’m substantially more ignorant on the subject than either you or your previous interlocutor and it took me a while to even understand that “PCT” was the theory Marken supports, that you two were taking the pro and con towards it, that your text was mostly between each other with a substantial amount of knowledge assumed. I wish you had both linked more, because it would have been educational.
That said, I’d like to see such links if you know of any. If I can swiftly dismiss Marken’s work without further thought, that would be a very efficient use of time. Can you direct me to the links showing an example of his experimental work so I can verify that his research program is crippled by mathematical overfitting? The best I could find was Perceptual organization of behavior: A hierarchical control model of coordinated action but it was pay-walled so I can’t access it now to look into it myself.
The paper discussed in that interaction can be found here without a paywall.
As stated then (the conversation can be taken up from about here if not earlier), I think it’s quite likely that simple control circuits can be found in facets of motor response; but Powers, Marken and Eby had been talking about control theory in cognitive domains (like akrasia) as if they could isolate simple circuits there, and my search for any kind of evidence turned up only this sort of embarrassing tripe.
And really, the math here is important— it’s not a matter of disagreeing with interpretation, it’s the plain fact that a generic model with 4 free parameters can be tweaked to precisely fit 4 data points, and it’s clear from the paper that this is what Marken did. You simply need more data points than free parameters in order to generate any evidence in favor of a model; the fact that he never mentioned this, and instead crowed about the impressive fit of his model to the data, indicate either gross ignorance of how mathematical models work, or outright intent to mislead (coupled with an utterly incompetent peer review process.)
The gauntlet remains thrown, if anyone wants to point to an experimental study which demonstrates a discernible control circuit in a cognitive task (apart from tasks, like tracking a dot, which have an obvious motor component— in these, I do expect control circuits to be a good model for certain behavior). I would be surprised, but it would suffice to give credence to the theory in my eyes.
Through judicious abuse of my employer’s resources, I have acquired a copy of the PDF—PM me an email address and I’ll send it to you.
Thanks Robin! I have read this paper now, but it still doesn’t seem to address the arguments that orthonormal linked to :-/
The 1986 study appeared to me to be basically well done, offering a fascinating paradigm that could be extended in many directions for further research with a reasonably strong result by itself. It basically confirmed the positive claims of Marken that hierarchical arrangements of negative feedback loop systems (designed, with a handful of optimized parameters, and then left alone) can roughly reproduce trained human behavior in a variety of dynamically changing toy domains, supporting the contention that whatever is operating in the human nervous system after a period of training is doing roughly the same effective computations as the model.
In the text, Merken addresses the “motor control literature” as making claims whose refutation was partly the purpose of his experiments.
It required a little more googling to figure out the claims he was trying to reject… but basically he seems to be objecting to the claim that mammals work as open loop controllers (that is, generating action signals based on an internal model of the world that are sent into the world with no verification loop or secondary corrections). This claim appears to have been founded mostly on things called “deafferentiation experiments”… which turned out to be aesthetically horrifying and also turned out to not actually prove the general case of “open loop” claims.
The most infamous of these experiments, (warning—kind of disturbing pictures) was basically:
The ability of monkeys mutilated in this fashion to (eventually?) move around purposively was taken as evidence that there was not a hierarchically arranged set of negative feedback motor control systems implemented in their nervous system. In practice (after the scientist was arrested for animal cruelty, PETA’s request for custody was denied, and the monkeys were brainscanned, euthanized, and autopsied) it turned out that the monkey’s brains had been massively re-wired by the experience. The practical upshot of the experiments seem to have primarily been to serve as dramatic evidence of adult primate brain plasticity (which they didn’t believe in, back then?) rather than as confirmation of a negative feedback theory of motor control. (Probably there’s more to it than that, but this is my first draft understanding.)
Merken dismisses these experiments in part by pointing out the difficulty of preventing negative feedback control processes if there are many sub controllers that can use measurements partially correlated to the measure being optimized and concludes with falsification examples and criteria for the general theory and the particular model that are not subject to this objection:
In short, I’m still impressed by Merken. His reasoning seems clean, his experiment, robust, his criticisms of motor-control and trait-theory, well reasoned. My very broad impression is that there may be a over-arching background argument here between “accurate model in the head producing aim and fire success” versus “incremental goal accomplishment via well tuned reflexes and continuous effort”? If that back story is operative then I guess my posterior probability was just pushed a little more in the direction of “reflexes and effort” over “models and plans”.
If there is some trick still lurking here, Orthonormal, that you could point me to and spell out in detail rather than by reference to assertions and hand-waving rationality heuristics, that would be appreciated. The more time I spend on Merken’s work, the more I find to appreciate. At this point, I’ve spend a day or two on this and I think the burden of proof is on you. If you take it up successfully I would be in your debt for rubbing a bit of sand out of my eyes :-)
Jennifer, here is where orthonormal seems to say where exactly Marken overfit the data.
(Orthonormal might not have seen your comment because you didn’t post it in reply to one of his/hers.)
[ETA: Nevermind. Looks like the date of orthonormal’s last comment is after yours, so he/she probably saw it.]
I don’t understand what the quote is trying to say. What are the unrecognized consequences of the open-loop model?
It sounds like the author is upset that psychologists don’t believe he has a model of behavior that explains 99% of some output variable using only one input variable. I’d have a hard time believing too.