Thanks for such a thoughtful response Terry :). This all makes a ton of sense—I totally agree that the paper doesn’t give an alternative overarching theory, and that no such alternative theory exists. I guess my high-level worry is that, if ACT-R really were a good overarching model of the mind (like a paradigm, in Kuhnian terms), then it would have become standard or widely accepted in the field in the way that good overarching theories/paradigms became standard in other fields? Coming into this, my thought is that we don’t have any good overarching theory of the mind, and that we just don’t understand the mind well enough to make any models like that. But I am really curious about the success of ACT-R that you’re pointing to. If it’s actually a decent model, why do you think it didn’t take over the field (and shrunk to a small group of continuing researchers)? Genuine question, not rhetorical. My prior is that most cognitive scientists would kill for a good paradigm (I certainly would!).
Ooo, very good questions! :) I think there are a few different reasons why.… one small clarification, though, I don’t think ACT-R shrunk to a small group—I’d say more that it gradually grew from a small group (starting out of John Anderson’s lab at CMU) up to about 100 active researchers around the world, and then sort of stabilized at that level for the last decade or two.
But, as for why it didn’t take over everything or at least get more widely known, I’d say one big reason is that the tasks it historically focused on were very specific—usually things involving looking at letters and numbers on a screen and pressing keys on a keyboard or moving a mouse. So lots of the early examples were that sort of specific experimental psychology task. It’s been expanded a lot since then (car driving, for example), but that’s where its history was, and so for people who are interested in different sorts of tasks, I can see them maybe initially feeling like it’s not relevant. And even now, lots of the tasks in the paper you provided are so far away from even modern ACT-R that I can see people believing that they can just ignore ACT-R and try to develop completely new theories instead.
Another more practical reason, however, is that there’s a pretty high barrier to entry to getting into ACT-R, partly due to the fact that the reference implementation is in Lisp. Lisp made tons of sense as being the language to use when ACT-R was first developed, but it’s very hard to find students with experience in Lisp now. There’s been a big movement in the last decade to make alternate implementations of ACT-R (Python ACT-R, ACT-Up, jACT-R), and the latest version of ACT-R has interfaces to other languages which I think will help to make it more accessible. But, even with a more common programming language, there’s still a lot of teaching/training/learning required to get new people used to the core ideas. And even to get people used to the idea of sticking with the constraints of ACT-R. For example, I can remember a student building a model that needed to do mental arithmetic, and it took a while to explain why they couldn’t just say “c = a + b” and have the computer do the math (after all, you’re implementing these models on a computer, and computers are good at math, so why not just do the math that way?). Forcing yourself to break that addition down into steps (e.g. trying to recall the result from memory, or trying to do a memory recall of some similar addition fact and then doing counting to adjust to this particular question, or just doing counting right from the beginning, or doing the manual column-wise addition method in your head) gets pretty complicated, and it can be hard to adjust to that sort of mind-set.
I will note that this high-barrier-to-entry problem is probably true of all other cognitive architectures (e.g. Sigma, Clarion, Soar, Dynamic Field Theory, Semantic Pointer Architecture, etc: https://en.wikipedia.org/wiki/Comparison_of_cognitive_architectures ). But one thing ACT-R did really well is to address this by regularly running a 2-week summer-school (since 1994 http://act-r.psy.cmu.edu/workshops/ ). That seems to me to be a big reason why ACT-R got much more widely used (and thus much more widely tested and evaluated) than the other architectures that are out there. There was an active effort to teach the system and to spread it into new domains, and to combat the common approach in computational cognitive modelling of people sticking with the one model that they (or their supervisor) invented. It’s much more fun to build my own model from scratch and to evaluate it on the one particular task that I had in mind when I was inventing the model. But that just leads to a giant proliferation of under-tested models. :( To really test these theories, we need a community, and ACT-R is the biggest and most stable cognitive architecture community so far. It’d be great to have more such communities, but they’re hard to grow.
Thanks for such a thoughtful response Terry :). This all makes a ton of sense—I totally agree that the paper doesn’t give an alternative overarching theory, and that no such alternative theory exists. I guess my high-level worry is that, if ACT-R really were a good overarching model of the mind (like a paradigm, in Kuhnian terms), then it would have become standard or widely accepted in the field in the way that good overarching theories/paradigms became standard in other fields? Coming into this, my thought is that we don’t have any good overarching theory of the mind, and that we just don’t understand the mind well enough to make any models like that. But I am really curious about the success of ACT-R that you’re pointing to. If it’s actually a decent model, why do you think it didn’t take over the field (and shrunk to a small group of continuing researchers)? Genuine question, not rhetorical. My prior is that most cognitive scientists would kill for a good paradigm (I certainly would!).
Ooo, very good questions! :) I think there are a few different reasons why.… one small clarification, though, I don’t think ACT-R shrunk to a small group—I’d say more that it gradually grew from a small group (starting out of John Anderson’s lab at CMU) up to about 100 active researchers around the world, and then sort of stabilized at that level for the last decade or two.
But, as for why it didn’t take over everything or at least get more widely known, I’d say one big reason is that the tasks it historically focused on were very specific—usually things involving looking at letters and numbers on a screen and pressing keys on a keyboard or moving a mouse. So lots of the early examples were that sort of specific experimental psychology task. It’s been expanded a lot since then (car driving, for example), but that’s where its history was, and so for people who are interested in different sorts of tasks, I can see them maybe initially feeling like it’s not relevant. And even now, lots of the tasks in the paper you provided are so far away from even modern ACT-R that I can see people believing that they can just ignore ACT-R and try to develop completely new theories instead.
Another more practical reason, however, is that there’s a pretty high barrier to entry to getting into ACT-R, partly due to the fact that the reference implementation is in Lisp. Lisp made tons of sense as being the language to use when ACT-R was first developed, but it’s very hard to find students with experience in Lisp now. There’s been a big movement in the last decade to make alternate implementations of ACT-R (Python ACT-R, ACT-Up, jACT-R), and the latest version of ACT-R has interfaces to other languages which I think will help to make it more accessible. But, even with a more common programming language, there’s still a lot of teaching/training/learning required to get new people used to the core ideas. And even to get people used to the idea of sticking with the constraints of ACT-R. For example, I can remember a student building a model that needed to do mental arithmetic, and it took a while to explain why they couldn’t just say “c = a + b” and have the computer do the math (after all, you’re implementing these models on a computer, and computers are good at math, so why not just do the math that way?). Forcing yourself to break that addition down into steps (e.g. trying to recall the result from memory, or trying to do a memory recall of some similar addition fact and then doing counting to adjust to this particular question, or just doing counting right from the beginning, or doing the manual column-wise addition method in your head) gets pretty complicated, and it can be hard to adjust to that sort of mind-set.
I will note that this high-barrier-to-entry problem is probably true of all other cognitive architectures (e.g. Sigma, Clarion, Soar, Dynamic Field Theory, Semantic Pointer Architecture, etc: https://en.wikipedia.org/wiki/Comparison_of_cognitive_architectures ). But one thing ACT-R did really well is to address this by regularly running a 2-week summer-school (since 1994 http://act-r.psy.cmu.edu/workshops/ ). That seems to me to be a big reason why ACT-R got much more widely used (and thus much more widely tested and evaluated) than the other architectures that are out there. There was an active effort to teach the system and to spread it into new domains, and to combat the common approach in computational cognitive modelling of people sticking with the one model that they (or their supervisor) invented. It’s much more fun to build my own model from scratch and to evaluate it on the one particular task that I had in mind when I was inventing the model. But that just leads to a giant proliferation of under-tested models. :( To really test these theories, we need a community, and ACT-R is the biggest and most stable cognitive architecture community so far. It’d be great to have more such communities, but they’re hard to grow.