Long-time lurker, first time commenting. Without necessarily disagreeing on any object-level details, I want to give an alternate perspective. I’m a PhD in computational cog sci, have interacted with most of the top cognitive science departments in the US (e.g. through job search, conferences, etc), and I know literally zero people who use ACT-R for anything. It was never mentioned in any of my grad classes, has never been brought up in any talk I’ve been to—I don’t even know if I’ve even seen it ever cited in a paper I’ve read. I know of it, obviously, and I know that it was super influential back in the 90s, but I’d always just assumed that the research program withered away for some reason (given how little I’d seen it actually being used in top-level research at this point).
This post made me curious how I could have such a different perspective! I don’t know whether academic cognitive science is just really segregated and I’m missing all the ACT-R researchers still out there; whether ACT-R was actually amazing and people have been silly to drop it; whether “top-level” research is misleading and actually the good research is being published in lower-tier journals while flashy fad-based results get published in top journals; or whether ACT-R really did fail for some deep reason. I’ve asked some colleagues why nobody around us uses it anymore, but I haven’t gotten any detailed responses yet.
(Also, this is a small thing, but “fitting human reaction times” is not impressive—that’s a basic feature of many, many models.)
So while I don’t have any object-level disagreements with this post, it feels like helpful context to know that many, many active computational cognitive scientists would strongly disagree that ACT-R is essentially the one best-validated theory of cognition (to the point where they’d be like “huh? what are you talking about?”). This paper gives what I think is a much more contemporary overview of overarching theories of human cognition.
That’s a very good point, CounterBlunder, and I should have highlighted that as well. It is definitely fairly common for cognitive science researchers to never work with or make use of ACT-R. It’s a sub-community within the cognitive science community. The research program has continued past the 90′s, and there’s probably around 100 or so researchers actively using it on a regular basis, but the cognitive science community is much larger than that, so your experience is pretty common.
As for whether ACT-R is “actually amazing and people have been silly to drop it”, well, I definitely don’t think that everyone should be making use of it, but I do think more people should be aware of its advantages, and the biggest reason for me is exactly what you point out about “fitting human reaction times” not being impressive. You’re completely right that that’s a basic feature of many, many models. But the key difference here is that ACT-R uses the same components and same math to fit human reaction times (and error patterns) across many different tasks. That is, instead of making a new model for a new task, ACT-R tries to use the same components, with the same parameter settings, but with perhaps a different set of background knowledge. The big advantage here is that it starts getting away from the over-fitting problem: when dealing with comparisons to human data, we normally have relatively few data points to compare to. And a cognitive model is going to, almost by definition, be fairly complex. So if we only fit to the data available for one task, the worry is that we’re going to have so many free parameters in our model that we can fit anything we like. And there’s also a worry that if I’m developing a cognitive model for one particular task, I might invent some custom component as part of my model that’s really highly specialized and would only ever get used in that one task, which is a bit worrying if I’m aiming for a general cognitive theory. One way around these problems is to find components and parameter settings that works across many different tasks. And right now, the ACT-R community is the biggest cognitive modelling community where there are many different researchers using the same components to do many different tasks.
(Note: by “same components in different tasks” I’m meaning something a lot more specific than something like “use a neural network”. In neural network terms, I’m more meaning something like “train up a neural network on this particular data X and then use that same trained neural network as a component in many different tasks”. After all, people very quickly change tasks and can re-purpose their existing neural networks to do new tasks extremely quickly. This hasn’t been common in neural networks until the recent advent of things like GPT-3. And, personally, I think GPT-3 would make an excellent module to be added to ACT-R, but that’s a whole other discussion.)
As for the paper you linked to, I really like that paper (and I’m even cited in it—yay!), but I don’t think it gives an overview of overarching theories of human cognition. Instead, I think it gives a wonderful list of tasks and situations where we’re going to need some pretty complicated components to perform these different tasks, and it gives a great set of suggestions as to what some of those components might be. But there’s no overarching theory of how we might combine those components together and make them work together and flexibly use them for doing different tasks. And that, to me, is what ACT-R provides an example of. I definitely don’t think ACT-R is the perfect, final solution, but it at least shows an example of what it would be like to coordinate components like that, and applies that to a wider variety of tasks than any particular system discussed in that paper. But lots of the tasks in that paper are also things that are incredibly far away from anything that ACT-R has been applied to, so I’m quite sure that ACT-R will need to change a lot to be expanded to include these sorts of new components needed for these new tasks. Still, it makes a good baseline for what it would take to have a flexible system that can be applied to different tasks, rather than building a new model for each task.
Thanks for such a thoughtful response Terry :). This all makes a ton of sense—I totally agree that the paper doesn’t give an alternative overarching theory, and that no such alternative theory exists. I guess my high-level worry is that, if ACT-R really were a good overarching model of the mind (like a paradigm, in Kuhnian terms), then it would have become standard or widely accepted in the field in the way that good overarching theories/paradigms became standard in other fields? Coming into this, my thought is that we don’t have any good overarching theory of the mind, and that we just don’t understand the mind well enough to make any models like that. But I am really curious about the success of ACT-R that you’re pointing to. If it’s actually a decent model, why do you think it didn’t take over the field (and shrunk to a small group of continuing researchers)? Genuine question, not rhetorical. My prior is that most cognitive scientists would kill for a good paradigm (I certainly would!).
Ooo, very good questions! :) I think there are a few different reasons why.… one small clarification, though, I don’t think ACT-R shrunk to a small group—I’d say more that it gradually grew from a small group (starting out of John Anderson’s lab at CMU) up to about 100 active researchers around the world, and then sort of stabilized at that level for the last decade or two.
But, as for why it didn’t take over everything or at least get more widely known, I’d say one big reason is that the tasks it historically focused on were very specific—usually things involving looking at letters and numbers on a screen and pressing keys on a keyboard or moving a mouse. So lots of the early examples were that sort of specific experimental psychology task. It’s been expanded a lot since then (car driving, for example), but that’s where its history was, and so for people who are interested in different sorts of tasks, I can see them maybe initially feeling like it’s not relevant. And even now, lots of the tasks in the paper you provided are so far away from even modern ACT-R that I can see people believing that they can just ignore ACT-R and try to develop completely new theories instead.
Another more practical reason, however, is that there’s a pretty high barrier to entry to getting into ACT-R, partly due to the fact that the reference implementation is in Lisp. Lisp made tons of sense as being the language to use when ACT-R was first developed, but it’s very hard to find students with experience in Lisp now. There’s been a big movement in the last decade to make alternate implementations of ACT-R (Python ACT-R, ACT-Up, jACT-R), and the latest version of ACT-R has interfaces to other languages which I think will help to make it more accessible. But, even with a more common programming language, there’s still a lot of teaching/training/learning required to get new people used to the core ideas. And even to get people used to the idea of sticking with the constraints of ACT-R. For example, I can remember a student building a model that needed to do mental arithmetic, and it took a while to explain why they couldn’t just say “c = a + b” and have the computer do the math (after all, you’re implementing these models on a computer, and computers are good at math, so why not just do the math that way?). Forcing yourself to break that addition down into steps (e.g. trying to recall the result from memory, or trying to do a memory recall of some similar addition fact and then doing counting to adjust to this particular question, or just doing counting right from the beginning, or doing the manual column-wise addition method in your head) gets pretty complicated, and it can be hard to adjust to that sort of mind-set.
I will note that this high-barrier-to-entry problem is probably true of all other cognitive architectures (e.g. Sigma, Clarion, Soar, Dynamic Field Theory, Semantic Pointer Architecture, etc: https://en.wikipedia.org/wiki/Comparison_of_cognitive_architectures ). But one thing ACT-R did really well is to address this by regularly running a 2-week summer-school (since 1994 http://act-r.psy.cmu.edu/workshops/ ). That seems to me to be a big reason why ACT-R got much more widely used (and thus much more widely tested and evaluated) than the other architectures that are out there. There was an active effort to teach the system and to spread it into new domains, and to combat the common approach in computational cognitive modelling of people sticking with the one model that they (or their supervisor) invented. It’s much more fun to build my own model from scratch and to evaluate it on the one particular task that I had in mind when I was inventing the model. But that just leads to a giant proliferation of under-tested models. :( To really test these theories, we need a community, and ACT-R is the biggest and most stable cognitive architecture community so far. It’d be great to have more such communities, but they’re hard to grow.
No you’re right, it doesn’t say how they should be combined. My assumption—and I suspect the assumption of the authors—is that we have no good widely-accepted overarching model of the mind, and that the best we can agree on is a list of ingredients (and even that list was controversial, e.g. in the commentaries on the paper). I think that’s the reason I, implicitly, was viewing the paper as a contemporary alternative to ACT-R. But I take your point that it’s doing different things.
I think my post (at least the title!) is essentially wrong if there are other overarching theories of cognition out there which have similar track records of matching data. Are there?
By “overarching theory” I mean a theory which is roughly as comprehensive as ACT-R in terms of breadth of brain regions and breadth of cognitive phenomena.
As someone who has also done grad school in cog-sci research (but in a computer science department, not a psychology department, so my knowledge is more AI focused), my impression is that most psychology research isn’t about such overarching theories. To be more precise:
There are cognitive architecture people, who work on overarching theories of cognition. However, ACT-R stands out amongst these as having extensive experimental validation. The rest have relatively minimal direct comparisons to human data, or none.
There are “bayesian brain” and other sorta overarching theories, but (to my limited knowledge!) these ideas don’t have such a fleshed-out computational model of the brain. EG, you might apply bayesian-brain ideas to create a model of (say) emotional processing, but it isn’t really part of one big model in quite the way ACT-R allows.
There’s a lot of more isolated work on specific subsystems of the brain, some of which is obviously going to be highly experimentally validated, but, just isn’t trying to be an overaching model at all.
So my claim is that ACT-R occupies a unique position in terms of (a) taking an experimental-psych approach, while (b) trying to provide a model of everything and how it fits together. Do you think I’m wrong about that?
I think it’s a bit like physics: outsiders hear about these big overarching theories (GUTs, TOEs, strings, …), and to an extent it makes sense for outsiders to focus on the big picture in that way. Working physicists, on the other hand, can work on all sorts of specialized things (the physics of crystal growth, say) without necessarily worrying about how it fits into the big picture. Not everyone works on the big-picture questions.
OTOH, I also feel like it’s unfortunate that more work isn’t integrated into overarching models.
This paper gives what I think is a much more contemporary overview of overarching theories of human cognition.
I’ve only skimmed it, but it seems to me more like a prospectus which speculates about building a totally new architecture (combining the strengths of deep learning with several handpicked ideas from psychology), naming specific challenges and possible routes forward for such a thing.
(Also, this is a small thing, but “fitting human reaction times” is not impressive—that’s a basic feature of many, many models.)
I said “down to reaction times” mostly because I think this gives readers a good sense of the level of detail, and because I know reaction times are something ACT-R puts effort into, as opposed to because I think reaction times is the big advantage ACT-R has over other models; but, in retrospect this may have been misleading.
I guess it comes down to my AI-centric background. For example, GPT-3 is in some sense a very impressive model of human linguistic behavior; but, it makes absolutely no attempt to match human reaction times. It’s very rare for ML people to be interested in that sort of thing. This also relates to the internal design of ACT-R. An AI/ML programmer isn’t usually interested in purposefully slowing down operations to match human performance. So this would be one of the most alien things about the ACT-R codebase for a lot of people.
Thanks for the thoughtful response, that perspective makes sense. I take your point that ACT-R is unique in the ways you’re describing, and that most cognitive scientists are not working on overarching models of the mind like that. I think maybe our disagreement is about how good/useful of an overarching model ACT-R is? It’s definitely not like in physics, where some overarching theories are widely accepted (e.g. the standard model) even by people working on much more narrow topics—and many of the ones that aren’t (e.g. string theory) are still widely known about and commonly taught. The situation in cog sci (in my view, and I think in many people’s views?) is much more that we don’t have an overarching model of the mind in anywhere close to the level of detail/mechanistic specificity that ACT-R posits, and that any such attempt would be premature/foolish/not useful right now. Like, I think if you polled cognitive scientists, the vast majority would disagree with the title of your post—not because they think there’s a salient alternative, but because they think that there is no theory that even comes close to meriting the title of “best-validated theory of cognition” (even if technically one theory is ahead of the others). Do you know what I mean? Of course, even if most cognitive scientists don’t believe in ACT-R in that way, that alone doesn’t mean that ACT-R is wrong.. I’m curious about the evidence that Terry is talking about above. I just think the field would look really, really different if we actually had a halfway-decent paradigm/overarching model of the mind. And it’s not like ACT-R is some unknown idea that is poised to take over the field once people learn about it. Everyone knew about it in the 90s, and then it fell out of widespread use—and my prior on why that happened is that people weren’t finding it super useful. (Although like I said, I’m really curious to learn more about what Terry/other contemporary people are doing with it!)
I agree that there isn’t an overarching theory at the level of specificity of ACT-R that covers all the different aspects of the mind that cognitive science researchers wish it would cover. And so yes, I can see cognitive scientists saying that there is no such theory, or (more accurately) saying that even though ACT-R is the best-validated one, it’s not validated on the particular types of tasks that they’re interested in, so therefore they can ignore it.
However, I do think that there’s enough of a consensus about some aspects of ACT-R (and other theories) that there are some broader generalizations that all cognitive scientists should be aware of. That’s the point of the two papers listed in the original post on the “Common Model of Cognition”. They dig through a whole bunch of different cognitive architectures and ideas over the decades and point out that there are some pretty striking commonalities and similarities across these models. (ACT-R is just one of the theories that they look at, and they point out that there are a set of commonalities across all the theories, and that’s what they call the Common Model of Cognition). The Common Model of Cognition is much more loosely specified and is much more about structural organization rather than being about the particular equations used, though, so I’d still say that ACT-R is the best-validated model. But CMC is surprisingly consistent with a lot of models, and that’s why the community is getting together to write papers like that. The whole point is to try to show that there are some things that we can say right now about an overarching theory of the mind, even if people don’t want to buy into the particular details of ACT-R. And if people are trying to build overarching theories, they should at least be aware of what there is already.
(Full disclosure: I was at the 2017 meeting where this community came together on this topic and started the whole CMC thing. The papers from that meeting are at https://www.aaai.org/Library/Symposia/Fall/fs17-05.php and that’s a great collection of short papers of people talking about the various challenges of expanding the CMC. The general consensus from that meeting is that it was useful to at least have an explicit CMC to help frame that conversation, and it’s been great to see that conversation grow over the last few years. Note: at the time we were calling it the Standard Model of the Mind, but that got changed to Common Model of Cognition).
I think maybe our disagreement is about how good/useful of an overarching model ACT-R is? It’s definitely not like in physics, where some overarching theories are widely accepted (e.g. the standard model) even by people working on much more narrow topics—and many of the ones that aren’t (e.g. string theory) are still widely known about and commonly taught. The situation in cog sci (in my view, and I think in many people’s views?) is much more that we don’t have an overarching model of the mind in anywhere close to the level of detail/mechanistic specificity that ACT-R posits, and that any such attempt would be premature/foolish/not useful right now.
Makes some sense to me! This is part of why my post’s conclusion said stuff like this doesn’t mean you should believe in ACT-R. But yeah, I also think we have a disagreement somewhere around here.
I was trained in the cognitive architecture tradition, which tends to find this situation unfortunate. I have heard strong opinions, which I respect and generally believe, of the “we just don’t know enough” variety which you also espouse. However, I also buy Allen Newell’s famous argument in “you can’t play 20 questions with nature and win”, where he argues that we may never get there without focusing on that goal. From this perspective, it makes (some) sense to try to track a big picture anyway.
In some sense the grand goal of cognitive architecture is that it should eventually be seen as standard (almost required) for individual works of experimental psychology to contribute to a big picture in some way. Imagine for a moment if every paper had a section relating to ACT-R (or some other overarching model), either pointing out how it fits in (agreeing with and extending the overarching model) or pointing out how it doesn’t (revising the overarching model).
With the current state of things, it’s very unclear (as you highlighted in your original comment) what the status of overarching models like ACT-R even is. Is it an artifact from the 90s which is long-irrelevant? Is it the state of the art big-picture? Nobody knows and few care? Wouldn’t it be better if it were otherwise?
On the other hand, working with cognitive architectures like ACT-R can be frustrating and time consuming. In theory, they could be a time-saving tool (you start with all the power of ACT-R and can move forward from that!). In practice, my personal observation at least is that they add time and reduce other kinds of progress you can make. To caricaturize, a cog arch phd student spends their first 2 years learning the cognitive architecture they’ll work with, while a non-cog-arch cogsci student can hit the ground running instead. (This isn’t totally true of course; I’ve heard people say that most phd students are not really productive for their first year or two of grad school.) So I do not want to gloss over the downsides to a cog arch focus.
One big problem is what I’ll call the “task integration problem”. Let’s say you have 100 research psychologists who each spend a chunk of time doing “X in ACT-R” for many different values of X. Now you have lots of ACT-R models of lots of different cognitive phenomena. Can you mash them all together into one big model which does all 100 things?
I’m not totally sure about ACT-R, but I’ve heard that for most cognitive architectures, the answer is “no”. Despite existing in one cognitive architecture, the individual “X” models are sorta like standalone programs which don’t know how to talk to each other.
This undermines the premise of cog arch as helping us fit everything into one coherent picture. So, this is a hurdle which cog arch would have to get past in order to play the kind of role it wants to play.
Long-time lurker, first time commenting. Without necessarily disagreeing on any object-level details, I want to give an alternate perspective. I’m a PhD in computational cog sci, have interacted with most of the top cognitive science departments in the US (e.g. through job search, conferences, etc), and I know literally zero people who use ACT-R for anything. It was never mentioned in any of my grad classes, has never been brought up in any talk I’ve been to—I don’t even know if I’ve even seen it ever cited in a paper I’ve read. I know of it, obviously, and I know that it was super influential back in the 90s, but I’d always just assumed that the research program withered away for some reason (given how little I’d seen it actually being used in top-level research at this point).
This post made me curious how I could have such a different perspective! I don’t know whether academic cognitive science is just really segregated and I’m missing all the ACT-R researchers still out there; whether ACT-R was actually amazing and people have been silly to drop it; whether “top-level” research is misleading and actually the good research is being published in lower-tier journals while flashy fad-based results get published in top journals; or whether ACT-R really did fail for some deep reason. I’ve asked some colleagues why nobody around us uses it anymore, but I haven’t gotten any detailed responses yet.
(Also, this is a small thing, but “fitting human reaction times” is not impressive—that’s a basic feature of many, many models.)
So while I don’t have any object-level disagreements with this post, it feels like helpful context to know that many, many active computational cognitive scientists would strongly disagree that ACT-R is essentially the one best-validated theory of cognition (to the point where they’d be like “huh? what are you talking about?”). This paper gives what I think is a much more contemporary overview of overarching theories of human cognition.
That’s a very good point, CounterBlunder, and I should have highlighted that as well. It is definitely fairly common for cognitive science researchers to never work with or make use of ACT-R. It’s a sub-community within the cognitive science community. The research program has continued past the 90′s, and there’s probably around 100 or so researchers actively using it on a regular basis, but the cognitive science community is much larger than that, so your experience is pretty common.
As for whether ACT-R is “actually amazing and people have been silly to drop it”, well, I definitely don’t think that everyone should be making use of it, but I do think more people should be aware of its advantages, and the biggest reason for me is exactly what you point out about “fitting human reaction times” not being impressive. You’re completely right that that’s a basic feature of many, many models. But the key difference here is that ACT-R uses the same components and same math to fit human reaction times (and error patterns) across many different tasks. That is, instead of making a new model for a new task, ACT-R tries to use the same components, with the same parameter settings, but with perhaps a different set of background knowledge. The big advantage here is that it starts getting away from the over-fitting problem: when dealing with comparisons to human data, we normally have relatively few data points to compare to. And a cognitive model is going to, almost by definition, be fairly complex. So if we only fit to the data available for one task, the worry is that we’re going to have so many free parameters in our model that we can fit anything we like. And there’s also a worry that if I’m developing a cognitive model for one particular task, I might invent some custom component as part of my model that’s really highly specialized and would only ever get used in that one task, which is a bit worrying if I’m aiming for a general cognitive theory. One way around these problems is to find components and parameter settings that works across many different tasks. And right now, the ACT-R community is the biggest cognitive modelling community where there are many different researchers using the same components to do many different tasks.
(Note: by “same components in different tasks” I’m meaning something a lot more specific than something like “use a neural network”. In neural network terms, I’m more meaning something like “train up a neural network on this particular data X and then use that same trained neural network as a component in many different tasks”. After all, people very quickly change tasks and can re-purpose their existing neural networks to do new tasks extremely quickly. This hasn’t been common in neural networks until the recent advent of things like GPT-3. And, personally, I think GPT-3 would make an excellent module to be added to ACT-R, but that’s a whole other discussion.)
As for the paper you linked to, I really like that paper (and I’m even cited in it—yay!), but I don’t think it gives an overview of overarching theories of human cognition. Instead, I think it gives a wonderful list of tasks and situations where we’re going to need some pretty complicated components to perform these different tasks, and it gives a great set of suggestions as to what some of those components might be. But there’s no overarching theory of how we might combine those components together and make them work together and flexibly use them for doing different tasks. And that, to me, is what ACT-R provides an example of. I definitely don’t think ACT-R is the perfect, final solution, but it at least shows an example of what it would be like to coordinate components like that, and applies that to a wider variety of tasks than any particular system discussed in that paper. But lots of the tasks in that paper are also things that are incredibly far away from anything that ACT-R has been applied to, so I’m quite sure that ACT-R will need to change a lot to be expanded to include these sorts of new components needed for these new tasks. Still, it makes a good baseline for what it would take to have a flexible system that can be applied to different tasks, rather than building a new model for each task.
Thanks for such a thoughtful response Terry :). This all makes a ton of sense—I totally agree that the paper doesn’t give an alternative overarching theory, and that no such alternative theory exists. I guess my high-level worry is that, if ACT-R really were a good overarching model of the mind (like a paradigm, in Kuhnian terms), then it would have become standard or widely accepted in the field in the way that good overarching theories/paradigms became standard in other fields? Coming into this, my thought is that we don’t have any good overarching theory of the mind, and that we just don’t understand the mind well enough to make any models like that. But I am really curious about the success of ACT-R that you’re pointing to. If it’s actually a decent model, why do you think it didn’t take over the field (and shrunk to a small group of continuing researchers)? Genuine question, not rhetorical. My prior is that most cognitive scientists would kill for a good paradigm (I certainly would!).
Ooo, very good questions! :) I think there are a few different reasons why.… one small clarification, though, I don’t think ACT-R shrunk to a small group—I’d say more that it gradually grew from a small group (starting out of John Anderson’s lab at CMU) up to about 100 active researchers around the world, and then sort of stabilized at that level for the last decade or two.
But, as for why it didn’t take over everything or at least get more widely known, I’d say one big reason is that the tasks it historically focused on were very specific—usually things involving looking at letters and numbers on a screen and pressing keys on a keyboard or moving a mouse. So lots of the early examples were that sort of specific experimental psychology task. It’s been expanded a lot since then (car driving, for example), but that’s where its history was, and so for people who are interested in different sorts of tasks, I can see them maybe initially feeling like it’s not relevant. And even now, lots of the tasks in the paper you provided are so far away from even modern ACT-R that I can see people believing that they can just ignore ACT-R and try to develop completely new theories instead.
Another more practical reason, however, is that there’s a pretty high barrier to entry to getting into ACT-R, partly due to the fact that the reference implementation is in Lisp. Lisp made tons of sense as being the language to use when ACT-R was first developed, but it’s very hard to find students with experience in Lisp now. There’s been a big movement in the last decade to make alternate implementations of ACT-R (Python ACT-R, ACT-Up, jACT-R), and the latest version of ACT-R has interfaces to other languages which I think will help to make it more accessible. But, even with a more common programming language, there’s still a lot of teaching/training/learning required to get new people used to the core ideas. And even to get people used to the idea of sticking with the constraints of ACT-R. For example, I can remember a student building a model that needed to do mental arithmetic, and it took a while to explain why they couldn’t just say “c = a + b” and have the computer do the math (after all, you’re implementing these models on a computer, and computers are good at math, so why not just do the math that way?). Forcing yourself to break that addition down into steps (e.g. trying to recall the result from memory, or trying to do a memory recall of some similar addition fact and then doing counting to adjust to this particular question, or just doing counting right from the beginning, or doing the manual column-wise addition method in your head) gets pretty complicated, and it can be hard to adjust to that sort of mind-set.
I will note that this high-barrier-to-entry problem is probably true of all other cognitive architectures (e.g. Sigma, Clarion, Soar, Dynamic Field Theory, Semantic Pointer Architecture, etc: https://en.wikipedia.org/wiki/Comparison_of_cognitive_architectures ). But one thing ACT-R did really well is to address this by regularly running a 2-week summer-school (since 1994 http://act-r.psy.cmu.edu/workshops/ ). That seems to me to be a big reason why ACT-R got much more widely used (and thus much more widely tested and evaluated) than the other architectures that are out there. There was an active effort to teach the system and to spread it into new domains, and to combat the common approach in computational cognitive modelling of people sticking with the one model that they (or their supervisor) invented. It’s much more fun to build my own model from scratch and to evaluate it on the one particular task that I had in mind when I was inventing the model. But that just leads to a giant proliferation of under-tested models. :( To really test these theories, we need a community, and ACT-R is the biggest and most stable cognitive architecture community so far. It’d be great to have more such communities, but they’re hard to grow.
Does that paper actually mention any overall models of the human mind? It has a list of ingredients, but does it say how they should be combined?
No you’re right, it doesn’t say how they should be combined. My assumption—and I suspect the assumption of the authors—is that we have no good widely-accepted overarching model of the mind, and that the best we can agree on is a list of ingredients (and even that list was controversial, e.g. in the commentaries on the paper). I think that’s the reason I, implicitly, was viewing the paper as a contemporary alternative to ACT-R. But I take your point that it’s doing different things.
I think my post (at least the title!) is essentially wrong if there are other overarching theories of cognition out there which have similar track records of matching data. Are there?
By “overarching theory” I mean a theory which is roughly as comprehensive as ACT-R in terms of breadth of brain regions and breadth of cognitive phenomena.
As someone who has also done grad school in cog-sci research (but in a computer science department, not a psychology department, so my knowledge is more AI focused), my impression is that most psychology research isn’t about such overarching theories. To be more precise:
There are cognitive architecture people, who work on overarching theories of cognition. However, ACT-R stands out amongst these as having extensive experimental validation. The rest have relatively minimal direct comparisons to human data, or none.
There are “bayesian brain” and other sorta overarching theories, but (to my limited knowledge!) these ideas don’t have such a fleshed-out computational model of the brain. EG, you might apply bayesian-brain ideas to create a model of (say) emotional processing, but it isn’t really part of one big model in quite the way ACT-R allows.
There’s a lot of more isolated work on specific subsystems of the brain, some of which is obviously going to be highly experimentally validated, but, just isn’t trying to be an overaching model at all.
So my claim is that ACT-R occupies a unique position in terms of (a) taking an experimental-psych approach, while (b) trying to provide a model of everything and how it fits together. Do you think I’m wrong about that?
I think it’s a bit like physics: outsiders hear about these big overarching theories (GUTs, TOEs, strings, …), and to an extent it makes sense for outsiders to focus on the big picture in that way. Working physicists, on the other hand, can work on all sorts of specialized things (the physics of crystal growth, say) without necessarily worrying about how it fits into the big picture. Not everyone works on the big-picture questions.
OTOH, I also feel like it’s unfortunate that more work isn’t integrated into overarching models.
I’ve only skimmed it, but it seems to me more like a prospectus which speculates about building a totally new architecture (combining the strengths of deep learning with several handpicked ideas from psychology), naming specific challenges and possible routes forward for such a thing.
I said “down to reaction times” mostly because I think this gives readers a good sense of the level of detail, and because I know reaction times are something ACT-R puts effort into, as opposed to because I think reaction times is the big advantage ACT-R has over other models; but, in retrospect this may have been misleading.
I guess it comes down to my AI-centric background. For example, GPT-3 is in some sense a very impressive model of human linguistic behavior; but, it makes absolutely no attempt to match human reaction times. It’s very rare for ML people to be interested in that sort of thing. This also relates to the internal design of ACT-R. An AI/ML programmer isn’t usually interested in purposefully slowing down operations to match human performance. So this would be one of the most alien things about the ACT-R codebase for a lot of people.
Thanks for the thoughtful response, that perspective makes sense. I take your point that ACT-R is unique in the ways you’re describing, and that most cognitive scientists are not working on overarching models of the mind like that. I think maybe our disagreement is about how good/useful of an overarching model ACT-R is? It’s definitely not like in physics, where some overarching theories are widely accepted (e.g. the standard model) even by people working on much more narrow topics—and many of the ones that aren’t (e.g. string theory) are still widely known about and commonly taught. The situation in cog sci (in my view, and I think in many people’s views?) is much more that we don’t have an overarching model of the mind in anywhere close to the level of detail/mechanistic specificity that ACT-R posits, and that any such attempt would be premature/foolish/not useful right now. Like, I think if you polled cognitive scientists, the vast majority would disagree with the title of your post—not because they think there’s a salient alternative, but because they think that there is no theory that even comes close to meriting the title of “best-validated theory of cognition” (even if technically one theory is ahead of the others). Do you know what I mean? Of course, even if most cognitive scientists don’t believe in ACT-R in that way, that alone doesn’t mean that ACT-R is wrong.. I’m curious about the evidence that Terry is talking about above. I just think the field would look really, really different if we actually had a halfway-decent paradigm/overarching model of the mind. And it’s not like ACT-R is some unknown idea that is poised to take over the field once people learn about it. Everyone knew about it in the 90s, and then it fell out of widespread use—and my prior on why that happened is that people weren’t finding it super useful. (Although like I said, I’m really curious to learn more about what Terry/other contemporary people are doing with it!)
I agree that there isn’t an overarching theory at the level of specificity of ACT-R that covers all the different aspects of the mind that cognitive science researchers wish it would cover. And so yes, I can see cognitive scientists saying that there is no such theory, or (more accurately) saying that even though ACT-R is the best-validated one, it’s not validated on the particular types of tasks that they’re interested in, so therefore they can ignore it.
However, I do think that there’s enough of a consensus about some aspects of ACT-R (and other theories) that there are some broader generalizations that all cognitive scientists should be aware of. That’s the point of the two papers listed in the original post on the “Common Model of Cognition”. They dig through a whole bunch of different cognitive architectures and ideas over the decades and point out that there are some pretty striking commonalities and similarities across these models. (ACT-R is just one of the theories that they look at, and they point out that there are a set of commonalities across all the theories, and that’s what they call the Common Model of Cognition). The Common Model of Cognition is much more loosely specified and is much more about structural organization rather than being about the particular equations used, though, so I’d still say that ACT-R is the best-validated model. But CMC is surprisingly consistent with a lot of models, and that’s why the community is getting together to write papers like that. The whole point is to try to show that there are some things that we can say right now about an overarching theory of the mind, even if people don’t want to buy into the particular details of ACT-R. And if people are trying to build overarching theories, they should at least be aware of what there is already.
(Full disclosure: I was at the 2017 meeting where this community came together on this topic and started the whole CMC thing. The papers from that meeting are at https://www.aaai.org/Library/Symposia/Fall/fs17-05.php and that’s a great collection of short papers of people talking about the various challenges of expanding the CMC. The general consensus from that meeting is that it was useful to at least have an explicit CMC to help frame that conversation, and it’s been great to see that conversation grow over the last few years. Note: at the time we were calling it the Standard Model of the Mind, but that got changed to Common Model of Cognition).
Makes some sense to me! This is part of why my post’s conclusion said stuff like this doesn’t mean you should believe in ACT-R. But yeah, I also think we have a disagreement somewhere around here.
I was trained in the cognitive architecture tradition, which tends to find this situation unfortunate. I have heard strong opinions, which I respect and generally believe, of the “we just don’t know enough” variety which you also espouse. However, I also buy Allen Newell’s famous argument in “you can’t play 20 questions with nature and win”, where he argues that we may never get there without focusing on that goal. From this perspective, it makes (some) sense to try to track a big picture anyway.
In some sense the grand goal of cognitive architecture is that it should eventually be seen as standard (almost required) for individual works of experimental psychology to contribute to a big picture in some way. Imagine for a moment if every paper had a section relating to ACT-R (or some other overarching model), either pointing out how it fits in (agreeing with and extending the overarching model) or pointing out how it doesn’t (revising the overarching model).
With the current state of things, it’s very unclear (as you highlighted in your original comment) what the status of overarching models like ACT-R even is. Is it an artifact from the 90s which is long-irrelevant? Is it the state of the art big-picture? Nobody knows and few care? Wouldn’t it be better if it were otherwise?
On the other hand, working with cognitive architectures like ACT-R can be frustrating and time consuming. In theory, they could be a time-saving tool (you start with all the power of ACT-R and can move forward from that!). In practice, my personal observation at least is that they add time and reduce other kinds of progress you can make. To caricaturize, a cog arch phd student spends their first 2 years learning the cognitive architecture they’ll work with, while a non-cog-arch cogsci student can hit the ground running instead. (This isn’t totally true of course; I’ve heard people say that most phd students are not really productive for their first year or two of grad school.) So I do not want to gloss over the downsides to a cog arch focus.
One big problem is what I’ll call the “task integration problem”. Let’s say you have 100 research psychologists who each spend a chunk of time doing “X in ACT-R” for many different values of X. Now you have lots of ACT-R models of lots of different cognitive phenomena. Can you mash them all together into one big model which does all 100 things?
I’m not totally sure about ACT-R, but I’ve heard that for most cognitive architectures, the answer is “no”. Despite existing in one cognitive architecture, the individual “X” models are sorta like standalone programs which don’t know how to talk to each other.
This undermines the premise of cog arch as helping us fit everything into one coherent picture. So, this is a hurdle which cog arch would have to get past in order to play the kind of role it wants to play.