I’m perpetually surprised by the amount of thought that goes into this sort of thing coupled with the lack of attention to the philosophical literature on theories of mind and agency in the past, let’s just say 50 years. I mean look at the entire debate around whether or not it’s possible to naturalize normativity—most of the philosophical profession has given up on this or accepts the question was at best too hard to answer, at worst, ill-conceived from the start.
These literatures are very aware of, and conversant with, the latest and greatest in cogsci and stats. They’re not just rehashing old stuff. There is a lot of good work done there on how to make those fundamental questions around agency tractable. There’s also an important strain in that literature which claims there are in-principle problems for the very idea of a generalized theory of mind or agency (sic. Putnam, McDowell, Wittgenstein, the entire University of Chicago philosophy department, etc.).
I entered a philosophy PhD program convinced that there were genuine worries here about AGI, machine ethics, etc. I sat in the back of MIRI conferences quietly nodding along. Then I really started absorbing Wittgenstein and what’s sometimes called the “resolute reading” of his corpus and I have become convinced that what we call cognition, intelligence, agency, these are all a family of concepts which have a really unique foothold in biological life—that naturalizing even basic concepts like life turn out to be notoriously tricky (because of their normativity). And that the intelligence we recognize in human beings and other organisms are so bound up in our biological forms of life that it becomes very difficult to imagine something without the desire to evade death, nourish itself, and protect a physical body having any of the core agential concepts required to even be recognized as intelligent. Light dawns gradually over the whole. Semantic and meaning holism. Embedded biology. If a lion could speak, we couldn’t understand it. All that stuff.
A great place to start is with Jim Conant’s “The Search for the Logical Alien” and then get into Wittgenstein’s discussions of rule following and ontogenesis. Then have a look at some of the challenges naturalizing normativity in biology. This issue runs deep.
In the end, this idea that intelligence is kind of an isolatable property, indepedent from the particular forms of life in which it is manifest, is a really, really old idea. Goes back at least to the Gnostics. Every generation recapitulates it in some way. AGI just is that worry re-wrought for the software age.
If anything, this kind of thing may be worth studying just because it calls into question the assumptions of programs like MIRI and their earnest hand-wringing over AGI. At a minimum, it’s convinced me that we under-estimate by many orders of magnitude the volume of inputs needed to shape our “models.” It starts before we’re even born, and we can’t discount the centrality of e.g. experience touching things, having fragile bodies, having hunger, etc. in shaping the overall web of beliefs and desires that constitute our agential understanding.
Basically, if you’re looking for the foundational issues confronting any attempt to form a gears-level understanding of the kind of goal-directed organization that all life-forms exhibit (e.g. much of biological theory) you would do well to read some philosophy of biology. Peter Godfrey Smith has an excellent introduction that’s pretty level-handed. Natural selection really doesn’t bear as much weight as folks in other disciplines would like it too—especially when you realize that the possibility of evolution by drift confounds any attempt at a statistical reduction of biological function.
Do you have any thoughts on chess computers, guided missiles, computer viruses, etc, and whether they make a case for worries about AGI, even if you consider them something alien to the human kind of intelligence?
No—but perhaps I’m not seeing how they would make the case. Is the idea that somehow their existence augurs a future in which tech gets more autonomous to a point where we can no longer control it? I guess I’d say, why should we believe that’s true? Its probably uncontroversial to believe many of our tools will get more autonomous—but why should we think that’ll lead to the kind of autonomy we enjoy?
Even if you believe that the intelligence and autonomy we enjoy exist on a kind of continuum, from like single celled organisms through chess-playing computers, to us—we’d still need reason to believe that the progress along this continuum will continue at a rate necessary to close the gap between where we sit on the continuum and where our best artifacts currently sit on the continuum. I don’t doubt that progress will continue; but even if the continuum view were right, I think we sit way further out on the continuum than most people with the continuum view think. Also, the continuum view itself is very, very controversial. I happen to accept the arguments which aim to show that it faces insurmountable obstacles. The alternate view which I accept is that there’s a difference in kind between the intelligence and autonomy we enjoy, and the kind enjoyed by non-human animals and chess-playing computers. Many people think that if we accept that, we have to reject a certain form of metaphysical naturalism (e.g. the view that all natural phenomena can be explained in terms of the basic conceptual tools of physics, maths, and logic).
Some people think that this form of metaphysical naturalism is bedrock stuff; that if we don’t accept it, the theists win, blah blah blah, so we must naturalize mentality and agency, it must exist on a continuum, we just need a theory which shows us how. Other people think we can have a non-reductive naturalism which takes as primitive the normative concepts found in biology and psychology. That’s the view I hold. So no, I don’t think the existence of those things makes a case for worries about AGI. Things which enjoy the kind of mentality and autonomy we enjoy must be like us in many, many ways—that is after all, what enables us to recognize them as having mentality and autonomy like ours. They probably need to have bodies, be mortal, have finite resources, have an ontogenesis period where they go from not like-minded to like-minded (as all children do), have some language, etc.
Also, I think we have to think really carefully about what we mean when we say “human kind of intelligence”—if you read Jim Conant’s logically alien thought paper you come to understand why that extra bit of qualification amounts to plainly nonsensical language. There’s only intelligence simpliciter; insofar as we’re justified in recognizing it as such, it’s precisely in virtue of its bearing some resemblance to ours. The very idea of other kinds of intelligence which we might not be able to recognize is conceptually confused (if it bears no resemblance to ours, in virtue of what are we supposed to call it intelligent? Ex hypothesi? If so, I don’t know what I’m supposed to be imagining).
The person who wrote this post rightfully calls attention to the conceptual confusions surrounding most casual pre-formal thinking about agency and mentality. I applaud that, and am urging that the most rigorous, well-trodden paths exploring these confusions are to be found in philosophy as practiced (mostly but not exclusively) in the Anglophone tradition over the last 50 years.
That this should be ignored or overlooked out of pretension by very smart people who came up in cogsci, stats, or compsci is intelligible to me; that it should be ignored on a blog that is purportedly about investigating all the available evidence to find quicker pathways to understanding is less intelligible. I would commend everyone with an interest in this stuff to read Stanford Encyclopedia of Philosophy entries on different topics in philosophy of action and philosophy of mind, then go off their bibliographies for more detailed treatments. This stuff is all explored by philosophers really sympathetic to—even involved in—the projects of creating AGI. But more importantly, it is equally explored by those who either think the project is theoretically and practically possible but prudentially mistaken, or by those who think it is theoretically and practically impossible; let alone a prudential possibility.
Most mistakes here are made in the pre-formal thinking. Philosophy is the discipline of making that thinking more rigorous.
I don’t know… If I try to think of Anglophone philosophers of mind who I respect, I think of “Australian materialists” like Armstrong and Chalmers. No doubt there are plenty of worthwhile thoughts among the British, Americans, etc too, but you seem to be promoting something I deplore, the attempt to rule out various hard problems and unwelcome possibilities, by insisting that words shouldn’t be used that way. Celia Green even suggested that this 1984-like tactic could be the philosophy of a new dark age in which inquiry was stifled, not by belief in religion, but by “belief in society”; but perhaps technology has averted that future. Head-in-the-sand anthropocentrism is hardly tenable in a world where, already, someone could hook up a GPT3 chatbot to a Boston Dynamics chassis, and create an entity from deep within the uncanny valley.
Totally get it. There are lots of folks practicing philosophy of mind and technology today in that aussie tradition who I think take these questions seriously and try to cache out what we mean when we talk about agency, mentality, etc. as part of their broader projects.
I’d resist your characterization that I’m insisting words shouldn’t be used a particular way, though I can understand why it might seem that way. I’m rather hoping to shed more light on the idea raised by this post that we don’t actually know what many of these words even mean when they’re used in certain ways (hence the authors totally correct point about the need to clarify confusions about agency while working on the alignment problem). My whole point in wading in here is just to point out to a thoughtful community that there’s a really long rich history of doing just this, and even if you prefer the answers given by aussie materialists, it’s even better to understand those positions vis-a-vis their present and past interlocutors. If you understand those who disagree with them, and can articulate those positions in terms they’d accept, you understand your preferred positions even better. I wouldn’t say I deplore it, but I am always mildly amused when cogsci, compsci, and stats people start wading into plainly philosophical waters (“sort out our fundamental confusions about agency”) and talk as if they’re the first ones to get there—or the only ones presently splashing around. I guess I would have thought (perhaps naively) that on a site like this people would be at least curious to see what work has already been done on the questions so they can accelerate their inquiry.
Re: ruling out hard problems—lot’s of philosophy is the attempt to better understand the problem’s framing such that it either reduces to a different problem, or disappears altogether. I’d urge you to see this as an example of that kind of thing, rather than ruling out certain questions from the gun.
And on anthropocentrism—not sure what the point is supposed to be here, but perhaps it’s directed at the “difference in kind” statements I made above. If so, I’d hope we can see light between treating humans as if they were the center of the universe and recognizing that there are at least putatively qualitative differences between the type of agency rational animals enjoy and the type of agency enjoyed by non-rational animals and artifacts. Even the aussie materialists do that—and then set about trying to form a theory of mind and agency in physical terms because they rightly see those putatively qualitative differences as a challenge to their particular form of metaphysical naturalism.
So look, if the author of this post is really serious about (1) they will almost certainly have to talk about what we mean when we use agential words. There will almost certainly be disagreements about whether their characterizations (A) fit the facts, and (B) are coherent with the rest of our beliefs. I don’t want to come even close to implying that folks in compsci, cogsci, stats, etc. can’t do this—they certainly can. I’m just saying that it’s really, really conspicuous to not do so in dialogue with those whose entire discipline is devoted to that task. Philosophers are really good at testing our accounts of an agential concept by saying things like, “okay let’s run with this idea of yours that we can define agency and mentality in terms of some bayesian predictive processing, or in terms of planning states, or whaterver, but to see if that view really holds up, we have to be able to use your terms or some innocent others to account for all the distinctions we recognize in our thought and talk about minds and agents.” That’s the bulk of what philosophers of mind and action do nowadays—they take someone’s proposal about a theory of mind or action and test whether it can give an account of some region of our thought and talk about minds and agents. If it can’t they either propose addenda, push the burden back to the theorist, or point out structural reasons why the theory faces general obstacles that seem difficult to overcome.
Here’s some recent work on the topic, just to make it plain that there are philosophers working on these questions:
? The second is already open-access, and the third both works in SH & GS (with 2 different PDF links). Only the first link fails in SH. (But what an abstract: “I also argue that if future generally intelligent AI possess a predictive processing cognitive architecture, then they will come to share our pro-moral motivations (of valuing humanity as an end; avoiding maleficent actions; etc.), regardless of their initial motivation set.” Wow.)
Huh, I tried the first and third in SH, maybe I messed up somehow. My bad. Thanks!
I still am interested in the first (on the principle that maybe, just maybe, it’s the solution to all our problems instead of being yet another terrible argument made by philosophers about why AIs will be ethical by default if only we do X… I think I’ve seen two already) and would like to have access.
I can see how that would work. The author needs to be careful, though. Predictive processing may be a necessary condition for robust AGI alignment, but it is not per se a sufficient condition.
First of all, that only works if you give the AGI strong inductive priors for detecting and predicting human needs, goals, and values. Otherwise, it will tend to predict humans as though we are just “physical” systems (we are, but I mean modeling us without taking our sentience and values into account), no more worthy of special care than rocks or streams.
Second of all, this only works if the AGI has a structural bias toward treating the needs, goals, and values that it infers from predictive processing as its own. Otherwise, it may understand how to align with us, but it won’t care by default.
What do you mean by “naturalize” as a verb? What is “naturalizing normativity”?
Some people think that this form of metaphysical naturalism is bedrock stuff; that if we don’t accept it, the theists win, blah blah blah, so we must naturalize mentality and agency, it must exist on a continuum, we just need a theory which shows us how. Other people think we can have a non-reductive naturalism which takes as primitive the normative concepts found in biology and psychology.
Does this amount to you thinking that humans are humans because of some influence from outside of fundamental physics, which computers and non-human animals don’t share?
Like, yeah. People can be really impressive, but unless you want to make an explicit case for the contrary, people here still think people are made of parts and there exists some way to go from a large cloud of hydrogen to people.
What’s important is that it means coming up with a detailed, step-by-step explanation of how some high level concepts like life, shouldness, and intelligence. Just believing that they are natural is not the required explanation. Believing they are unnatural is not the only reason to disbelieve in the possibility of a reduction.
Reductionism is not just the claim that things are made out of parts. It’s a claim about explanation, and humans might not be smart enough to perform certainly reductions .
Reductionism is not just the claim that things are made out of parts. It’s a claim about explanation, and humans might not be smart enough to perform certainly reductions.
So basically the problem is that we haven’t got the explanation yet and can’t seem to find it with a philosopher’s toolkit? People have figured out a lot of things (electromagnetism, quantum physics, airplanes, semiconductors, DNA, visual cortex neuroscience) by mucking with physical things while having very little idea of them beforehand by just being smart and thinking hard. Seems like figuring out human concepts grounding to physics has a similar blocker, we still don’t have good enough neuroscience to do a simulation of how the brain goes from neurons to high-level thoughts (where you could observe a simulated brain-critter doing human-like things in a VR environment to tell you’re getting somewhere even when you haven’t reverse-engineered the semantics of the opaque processes yet). People having that kind of model to look at and trying to make sense of it could come up with all sorts of new unobvious useful concepts, just like people trying to figure out quantum mechanics came up with all sorts of new unobvious useful concepts.
But this doesn’t sound like a fun project for professional philosophers, a research project like that would need many neuroscientists and computer scientists and not very many philosophers. So if philosophers show up, look at a project like that, and go “this is stupid and you are stupid, go read more philosophy”, I’m not sure they’re doing it out of purely dispassionate pursuit of wisdom.
Good—though I’d want to clarify that there are some reductionists who think that there must be a reductive explanation for all natural phenomena, even if some will remain unknowable to us (for practical or theoretical reasons).
Other non-reductionists believe that the idea of giving a causal explanation of certain facts is actually confused—it’s not that there is no such explanation, it’s that the very idea of giving certain kinds of explanation means we don’t fully understand the propositions involved. E.g. if someone were to ask why certain mathematical facts are true, hoping for a causal explanation in terms of brain-facts or historical-evolutionary facts, we might wonder whether they understood what math is about.
Naturalizing normativity just means explaining normative phenomena in terms of other natural phenomena whose existence we accept as part of our broader metaphysics. E.g. explaining biological function in terms of evolution by natural selection, where natural selection is explained by differential survival rates and other statistical facts. Or explaining facts about minds, beliefs, attitudes, etc., in terms of non-humoncular goings-on in the brain. The project is typically aimed at humans, but shows up as soon as you get to biology and the handful of normative concepts (life, function, health, fitness, etc.) that constitute its core subject matter.
I don’t think I’ve seen the term “normative phenomena” before. So basically normative concepts are concepts in everyday language (“life”, “health”), which get messy if you try to push them too hard? But what are normative phenomena then? We don’t see or touch “life” or “health”, we see something closer to the actual stuff going on in the world and then we come up with everyday word-concepts for it that sort of work until they don’t.
It’s not really helping in that I still have no real intuition about what you’re going on about, and your AI critique seems to be aimed at something from 30 years ago instead of contemporary stuff like Omohundro’s Basic AI Drives paper (you describe AIs as being “without the desire to evade death, nourish itself, and protect a physical body”, the paper’s point is that AGIs operating in the physical world would have exactly that) or the whole deep learning explosion with massive datasets of the last few years (“we under-estimate by many orders of magnitude the volume of inputs needed to shape our “models.””, right now people are in a race to feed ginormous input sets to deep learning systems and probably aren’t stopping anytime soon).
Like, yeah. People can be really impressive, but unless you want to make an explicit case for the contrary, people here still think people are made of parts and there exists some way to go from a large cloud of hydrogen to people. If you think there’s some impossible gap between the human and the nonhuman worlds, then how do you think actual humans got here? Right now you seem to be just giving some sort of smug shrug of someone who on one hand doesn’t want to ask that question themselves because it’s corrosive to dignified pre-Darwin liberal arts sensibilities, and on the other hand tries to hint at people genuinely interested in the question that it’s a stupid question to ask and they should have read better scholarship to convince themselves of that.
If you think there’s some impossible gap between the human and the nonhuman worlds, then how do you think actual humans got here?
There are many types of explanatory claims in our language. Some are causal (how did something come to be), others are constitutive (what is it to be something), others still are normative (why is something good or right). Most mathematical explanation is constitutive, most action explanation is rational, and most material explanation is causal. It’s totally possible to think there’s a plain causal explanation about how humans evolved (through a combination of drift and natural selection, in which proportion we will likely never know) - while still thinking that the prospects for coming up with a constitutive explanation of normativity are dim (at best) or outright confused (at worst).
A common project shape for reductive naturalists is to try and use causal explanations to form a constitutive explanation for the normative aspects of biological life. If you spend enough time studying the many historical attempts that have been made at these explanations, you begin to see this pattern emerge where a would-be reductive theorist will either smuggle in a normative concept to fill out their causal story (thereby begging the question), or fail to deliver a theory which has the explanatory power to make basic normative distinctions which we intuitively recognize and that the theory should be able to account for (there are several really good tests out there for this—see the various takes on rule-following problems developed by Wittgenstein). Terms like “information” “structure” “fitness” “processing” “innateness” and the like all are subject to this sort of dilemma if you really put them under scrutiny. Magic non-natural stuff (like souls or spirit or that kind of thing) are often devices that people have reached for when forced on to this dilemma. Postulating that kind of thing is just the other side of the coin, and makes exactly the same error.
So I guess I’d say, I find it totally plausible how normative phenomena could be sui generis in much the same way that mathematical phenomena are, without finding it problematic that natural creatures can come to understand those phenomena through their upbringing and education. Some people get wrapped up in bewilderment about how this could even be possible, and I think there’s good reason to believe that bewilderment reflects deep misunderstandings about the phenomena themselves, the recourse for which is sometimes called philosophical therapy.
Another point I want to be clear on:
right now people are in a race to feed ginormous input sets to deep learning systems and probably aren’t stopping anytime soon
I don’t think it’s in-principle impossible to get from non-intelligent physical stuff to intelligent physical stuff by doing this—and i’m actually sympathetic to the biological anchors approach described here which was recently discussed on this site. I just think that the training runs will need to pay the computational costs for evolution to arrive at human brains, and for human brains to develop to maturity. I tend to think that—and I think good research in child development backs this up—that the structure of our thought is inextricably linked to our physicality. If anything, I think that’d push the development point out past Karnovsky’s 2093 estimate. Again, not it’s clearly not in-principle impossible for a natural thing to get the right amount of inputs to become intelligent (it clearly is possible, every human does it when they go from babies to adults); I just often think we underestimate how deeply important our biological histories (evolutionary and ontogenetic) are in this process. So I hope my urgings don’t come across as advocating for a return to some kind of pre-darwinian darkness; if anything I hope they can be seen as advocating for an even more thorough-going biological understanding. That must start with taking very seriously the problems introduced by drift, and the problems with the attempts to derive the normative aspects of life from a concept like genetic information (one which is notoriously subject to the dilemma above).
Thanks for the tip on the Basic AI Drives paper. I’ll give it a read. My suspicion is that once the “basic drives” are specified comprehensively enough to yield an intelligible picture of agent in question, we’ll find that they’re so much like us that the alignment problem disappears; they can only be aligned. That’s what someone argues in one of the papers I linked above. A separate question I’ve wondered about, and please point me to any good discussion of this, is to compare our thinking about AI alignment with intelligent alien alignment.
Finally, to answer this:
So basically normative concepts are concepts in everyday language (“life”, “health”), which get messy if you try to push them too hard?
No—normative concepts are a narrower class than the messy ones, though many find them messy. Normative concepts are those which structure our evaluative thought and talk (about the good, the bad, the ugly, etc.).
Anyway, good stuff. Keep the questions coming, happy to answer.
It’s totally possible to think there’s a plain causal explanation about how humans evolved (through a combination of drift and natural selection, in which proportion we will likely never know) - while still thinking that the prospects for coming up with a constitutive explanation of normativity are dim (at best) or outright confused (at worst).
If we believe there is a plain causal explanation, that rules out some explanations we could imagine. It shouldn’t now be possible for humans to have been created by a supernatural agency (as was widely thought in Antiquity, the Middle Ages or Renaissance when most of the canon of philosophy was developed), and basic human functioning probably shouldn’t involve processes wildly contrary to known physics (still believed by some smart people like Roger Penrose).
The other aspect is computational complexity. If we assume the causal explanation, we also get quantifiable limits for how much evolutionary work and complexity can have gone into humans. People are generally aware that there’s a lot of it, and a lot less aware that it’s quantifiably finite. The size of the human genome, which we can measure, creates one hard limit on how complex a human being can be. The limited amount of sensory information a human can pick up growing to adulthood and the limited amount of computation the human brain can do during that time creates another. Evolutionary theory also gives us a very interesting extra hint that everything you see in nature should be reachable by a very gradual ascent of slightly different forms, all of which need to be viable and competitive, all the way from the simplest chemical replicators. So that’s another limit to the bin, whatever is going on with humans is probably not something that has to drop out of nowhere as a ball of intractable complexity, but can be reached by some series of small enough to be understandable improvements to a small enough to be understandable initial lifeform.
The entire sphere of complex but finite computational processes has been a blind spot for philosophy. Nobody really understood it until computers had become reasonably common. (Dennett talks about this in Darwin’s Dangerous Idea when discussion Conway’s Game of Life.) Actually figuring things out from the opaque blobs of computation like human DNA is another problem of course. If you want to have some fun, you can reach for Rice’s theorem (basically following from Turing’s halting problem) which shows that you can’t logically infer any semantic properties whatsoever from the code of an undocumented computer program. Various existing property inferrer groups like hackers and biologists will nod along and then go back to poking the opaque mystery blobs with various clever implements and taking copious notes of what they do when poked, even though full logical closure is not available.
So coming back to the problem,
If you spend enough time studying the many historical attempts that have been made at these explanations, you begin to see this pattern emerge where a would-be reductive theorist will either smuggle in a normative concept to fill out their causal story (thereby begging the question), or fail to deliver a theory which has the explanatory power to make basic normative distinctions which we intuitively recognize and that the theory should be able to account for (there are several really good tests out there for this—see the various takes on rule-following problems developed by Wittgenstein). Terms like “information” “structure” “fitness” “processing” “innateness” and the like all are subject to this sort of dilemma if you really put them under scrutiny.
Okay, two thoughts about this. First, yes. This sounds like pretty much the inadequacy of mainstream philosophy argument that was being made on Lesswrong back in the Sequences days. The lack of satisfactory descriptions of human-level concepts that actually bottom down to reductive gears is real, but the inability to come up with the descriptions might be pretty much equivalent to the inability to write an understandable human-level AI architecture. That might be impossible, or it might be doable, but it doesn’t seem like we’ll find it out watching philosophers keep doing things with present-day philosopher toolkits. The people poking at the stuff are neuroscientists and computer scientists, and there’s a new kind of looking a “mechanized” mind from the outside aspect to that work (see for instance the predictive coding stuff on the neuroscience side) that seems very foreign to how philosophy operates.
Second thing is, I read this and I’m asking “so, what’s the actual problem we’re trying to solve?” You seem to be talking from the point of general methodological unhappiness with philosophy, where the problem is something like “you want to do philosophy as it’s always been done and you want it to get traction at the cutting edge of intellectual problems of the present day”. Concrete problems might be “understand how humans came to be and how they are able to do all the complex human thinking stuff”, which is a lot of neuroscience plus some evolutionary biology, “build a human-level artificial intelligence that will act in human interests no matter how powerful it is”, which, well, the second part is looking pretty difficult so the ideal answer might be “don’t”, but the first part seems to be coming along with a whole lot of computer science and not having needed a lot of input from philosophy so far. “Help people understand their place in the world, themselves and find life satisfaction” is a different goal again, and something a lot of philosophy used to be about. Taking the high-level human concepts that we don’t have satisfactory reductions for yet as granted could work fine at this level. But there seems to be a sense of philosophers becoming glorified talk therapists here, which doesn’t really feel like a satisfactory answer either.
If you want to have some fun, you can reach for Rice’s theorem (basically following from Turing’s halting problem) which shows that you can’t logically infer any semantic properties whatsoever from the code of an undocumented computer program. Various existing property inferrer groups like hackers and biologists will nod along and then go back to poking the opaque mystery blobs with various clever implements and taking copious notes of what they do when poked, even though full logical closure is not available.
I take it that this is how most progress in artificial intelligence, neuroscience, and cogsci has (and will continue) to proceed. My caution—and whole point in wading in here—is just that we shouldn’t expect progress by trying to come up with a better theory of mind or agency, even with more sophisticated explanatory tools.
I think it’s totally coherent and likely even that future artificial agents (generally intelligent or not) will be created without a general theory of mind or action.
In this scenario, you get a complete causal understanding of the mechanisms that enable agents to become minded and intentionally active, but you still don’t know what that agency or intelligence consist in beyond our simple, non-reductive folk-psychological explanations. A lot of folks in this scenario would be inclined to say, “who cares, we got the gears-level understanding” and I guess the only people who would care would be those who wanted to use the reductive causal story to tell us what it means to be minded. The philosophers I admire (John McDowell is the best example) appreciate the difference between causal and constitutive explanations when it comes to facts about minds and agents, and urge that progress in the sciences is hindered by running these together. They see no obstacle to technical progress in neuroscientific understanding or artificial intelligence; they just see themselves as sorting out what these disciplines are and are not about. They don’t think they’re in the business of giving constitutive explanations of what minds and agents are, rather, they’re in the business of discovering what enable minds and agents to do their minded and agential work. I think this distinction is apparent even with basic biological concepts like life. Biology can give us a complete account of the gears that enable life to work as it does without shedding any light on what makes it the case that something is alive, functioning, fit, goal-directed, successful, etc. But that’s not a problem at all if you think the purpose of biology is just to enable better medicine and engineering (like making artificial life forms or agents). To a task like, “given a region of physical space, identify whether there’s an agent there” I don’t we should expect any theory, philosophical or otherwise, to be able to yield solutions to that problem. I’m sure we can build artificial systems that can do it reliably (probably already have some), but it won’t come by way of understanding what makes an agent an agent.
Insofar as one hopes to advance certain engineering projects by “sorting out fundamental confusions about agency” I just wanted to offer that (1) there’s a rich literature in contemporary philosophy, continuous with the sciences, about different approaches to doing just that; and (2) that there are interesting arguments in this literature which aim to demonstrate that any causal-historical theory of these things will face an apparently intractable dilemma: either beg the question or be unable to make the distinctions needed to explain what agency and mentality consist in.
To summarize the points I’ve been trying to make (meanderingly, I’ll admit): On the one hand, I applaud the author for prioritizing that confusion-resolution; on the other hand, I’d urge them not to fall into the trap of thinking that confusion-resolution must take the form of stating an alternative theory of action or mind. The best kind of confusion-resolution is the kind that Wittgenstein introduced into philosophy, the kind where the problems themselves disappear—not because we realize they’re impossible to solve with present tools and so we give up, but because we realize we weren’t even clear about what we were asking in the first place (so the problems fail to even arise). In this case, the problem that’s supposed to disappear is the felt need to give a reductive causal account of minds and agents in terms of the non-normative explanatory tools available from maths and physics. So, go ahead and sort out those confusions, but be warned about what that project involves, who has gone down the road before, and the structural obstacles they’ve encountered both in and outside of philosophy so that you can be clear-headed about what the inquiry can reasonably be expected to yield.
That’s all I’ll say on the matter. Great back and forth, I don’t think there’s really much distance between us here. And for what it’s worth, mine is a pretty niche view in philosophy, because taken to its conclusion it means that the whole pursuit of trying to explain what minds and agents are is just confused from the gun—not limited by the particular set of explanatory tools presently available—just conceptually confused. Once that’s understood, one stops practicing or funding that sort of work. It is totally possible and advisable to keep studying the enabling gears so we can do better medicine and engineering, but we should get clear on how that medical or engineering understanding will advance and what those advances mean for those fundamental questions about what makes life, agents, minds, what they are. Good philosophy helps to dislodge us from the grip of expecting anything non-circular and illuminating in answer to those questions.
I’m perpetually surprised by the amount of thought that goes into this sort of thing coupled with the lack of attention to the philosophical literature on theories of mind and agency in the past, let’s just say 50 years. I mean look at the entire debate around whether or not it’s possible to naturalize normativity—most of the philosophical profession has given up on this or accepts the question was at best too hard to answer, at worst, ill-conceived from the start.
These literatures are very aware of, and conversant with, the latest and greatest in cogsci and stats. They’re not just rehashing old stuff. There is a lot of good work done there on how to make those fundamental questions around agency tractable. There’s also an important strain in that literature which claims there are in-principle problems for the very idea of a generalized theory of mind or agency (sic. Putnam, McDowell, Wittgenstein, the entire University of Chicago philosophy department, etc.).
I entered a philosophy PhD program convinced that there were genuine worries here about AGI, machine ethics, etc. I sat in the back of MIRI conferences quietly nodding along. Then I really started absorbing Wittgenstein and what’s sometimes called the “resolute reading” of his corpus and I have become convinced that what we call cognition, intelligence, agency, these are all a family of concepts which have a really unique foothold in biological life—that naturalizing even basic concepts like life turn out to be notoriously tricky (because of their normativity). And that the intelligence we recognize in human beings and other organisms are so bound up in our biological forms of life that it becomes very difficult to imagine something without the desire to evade death, nourish itself, and protect a physical body having any of the core agential concepts required to even be recognized as intelligent. Light dawns gradually over the whole. Semantic and meaning holism. Embedded biology. If a lion could speak, we couldn’t understand it. All that stuff.
A great place to start is with Jim Conant’s “The Search for the Logical Alien” and then get into Wittgenstein’s discussions of rule following and ontogenesis. Then have a look at some of the challenges naturalizing normativity in biology. This issue runs deep.
In the end, this idea that intelligence is kind of an isolatable property, indepedent from the particular forms of life in which it is manifest, is a really, really old idea. Goes back at least to the Gnostics. Every generation recapitulates it in some way. AGI just is that worry re-wrought for the software age.
If anything, this kind of thing may be worth studying just because it calls into question the assumptions of programs like MIRI and their earnest hand-wringing over AGI. At a minimum, it’s convinced me that we under-estimate by many orders of magnitude the volume of inputs needed to shape our “models.” It starts before we’re even born, and we can’t discount the centrality of e.g. experience touching things, having fragile bodies, having hunger, etc. in shaping the overall web of beliefs and desires that constitute our agential understanding.
Basically, if you’re looking for the foundational issues confronting any attempt to form a gears-level understanding of the kind of goal-directed organization that all life-forms exhibit (e.g. much of biological theory) you would do well to read some philosophy of biology. Peter Godfrey Smith has an excellent introduction that’s pretty level-handed. Natural selection really doesn’t bear as much weight as folks in other disciplines would like it too—especially when you realize that the possibility of evolution by drift confounds any attempt at a statistical reduction of biological function.
Hope something in there is interesting for you.
Do you have any thoughts on chess computers, guided missiles, computer viruses, etc, and whether they make a case for worries about AGI, even if you consider them something alien to the human kind of intelligence?
No—but perhaps I’m not seeing how they would make the case. Is the idea that somehow their existence augurs a future in which tech gets more autonomous to a point where we can no longer control it? I guess I’d say, why should we believe that’s true? Its probably uncontroversial to believe many of our tools will get more autonomous—but why should we think that’ll lead to the kind of autonomy we enjoy?
Even if you believe that the intelligence and autonomy we enjoy exist on a kind of continuum, from like single celled organisms through chess-playing computers, to us—we’d still need reason to believe that the progress along this continuum will continue at a rate necessary to close the gap between where we sit on the continuum and where our best artifacts currently sit on the continuum. I don’t doubt that progress will continue; but even if the continuum view were right, I think we sit way further out on the continuum than most people with the continuum view think. Also, the continuum view itself is very, very controversial. I happen to accept the arguments which aim to show that it faces insurmountable obstacles. The alternate view which I accept is that there’s a difference in kind between the intelligence and autonomy we enjoy, and the kind enjoyed by non-human animals and chess-playing computers. Many people think that if we accept that, we have to reject a certain form of metaphysical naturalism (e.g. the view that all natural phenomena can be explained in terms of the basic conceptual tools of physics, maths, and logic).
Some people think that this form of metaphysical naturalism is bedrock stuff; that if we don’t accept it, the theists win, blah blah blah, so we must naturalize mentality and agency, it must exist on a continuum, we just need a theory which shows us how. Other people think we can have a non-reductive naturalism which takes as primitive the normative concepts found in biology and psychology. That’s the view I hold. So no, I don’t think the existence of those things makes a case for worries about AGI. Things which enjoy the kind of mentality and autonomy we enjoy must be like us in many, many ways—that is after all, what enables us to recognize them as having mentality and autonomy like ours. They probably need to have bodies, be mortal, have finite resources, have an ontogenesis period where they go from not like-minded to like-minded (as all children do), have some language, etc.
Also, I think we have to think really carefully about what we mean when we say “human kind of intelligence”—if you read Jim Conant’s logically alien thought paper you come to understand why that extra bit of qualification amounts to plainly nonsensical language. There’s only intelligence simpliciter; insofar as we’re justified in recognizing it as such, it’s precisely in virtue of its bearing some resemblance to ours. The very idea of other kinds of intelligence which we might not be able to recognize is conceptually confused (if it bears no resemblance to ours, in virtue of what are we supposed to call it intelligent? Ex hypothesi? If so, I don’t know what I’m supposed to be imagining).
The person who wrote this post rightfully calls attention to the conceptual confusions surrounding most casual pre-formal thinking about agency and mentality. I applaud that, and am urging that the most rigorous, well-trodden paths exploring these confusions are to be found in philosophy as practiced (mostly but not exclusively) in the Anglophone tradition over the last 50 years.
That this should be ignored or overlooked out of pretension by very smart people who came up in cogsci, stats, or compsci is intelligible to me; that it should be ignored on a blog that is purportedly about investigating all the available evidence to find quicker pathways to understanding is less intelligible. I would commend everyone with an interest in this stuff to read Stanford Encyclopedia of Philosophy entries on different topics in philosophy of action and philosophy of mind, then go off their bibliographies for more detailed treatments. This stuff is all explored by philosophers really sympathetic to—even involved in—the projects of creating AGI. But more importantly, it is equally explored by those who either think the project is theoretically and practically possible but prudentially mistaken, or by those who think it is theoretically and practically impossible; let alone a prudential possibility.
Most mistakes here are made in the pre-formal thinking. Philosophy is the discipline of making that thinking more rigorous.
I don’t know… If I try to think of Anglophone philosophers of mind who I respect, I think of “Australian materialists” like Armstrong and Chalmers. No doubt there are plenty of worthwhile thoughts among the British, Americans, etc too, but you seem to be promoting something I deplore, the attempt to rule out various hard problems and unwelcome possibilities, by insisting that words shouldn’t be used that way. Celia Green even suggested that this 1984-like tactic could be the philosophy of a new dark age in which inquiry was stifled, not by belief in religion, but by “belief in society”; but perhaps technology has averted that future. Head-in-the-sand anthropocentrism is hardly tenable in a world where, already, someone could hook up a GPT3 chatbot to a Boston Dynamics chassis, and create an entity from deep within the uncanny valley.
Totally get it. There are lots of folks practicing philosophy of mind and technology today in that aussie tradition who I think take these questions seriously and try to cache out what we mean when we talk about agency, mentality, etc. as part of their broader projects.
I’d resist your characterization that I’m insisting words shouldn’t be used a particular way, though I can understand why it might seem that way. I’m rather hoping to shed more light on the idea raised by this post that we don’t actually know what many of these words even mean when they’re used in certain ways (hence the authors totally correct point about the need to clarify confusions about agency while working on the alignment problem). My whole point in wading in here is just to point out to a thoughtful community that there’s a really long rich history of doing just this, and even if you prefer the answers given by aussie materialists, it’s even better to understand those positions vis-a-vis their present and past interlocutors. If you understand those who disagree with them, and can articulate those positions in terms they’d accept, you understand your preferred positions even better. I wouldn’t say I deplore it, but I am always mildly amused when cogsci, compsci, and stats people start wading into plainly philosophical waters (“sort out our fundamental confusions about agency”) and talk as if they’re the first ones to get there—or the only ones presently splashing around. I guess I would have thought (perhaps naively) that on a site like this people would be at least curious to see what work has already been done on the questions so they can accelerate their inquiry.
Re: ruling out hard problems—lot’s of philosophy is the attempt to better understand the problem’s framing such that it either reduces to a different problem, or disappears altogether. I’d urge you to see this as an example of that kind of thing, rather than ruling out certain questions from the gun.
And on anthropocentrism—not sure what the point is supposed to be here, but perhaps it’s directed at the “difference in kind” statements I made above. If so, I’d hope we can see light between treating humans as if they were the center of the universe and recognizing that there are at least putatively qualitative differences between the type of agency rational animals enjoy and the type of agency enjoyed by non-rational animals and artifacts. Even the aussie materialists do that—and then set about trying to form a theory of mind and agency in physical terms because they rightly see those putatively qualitative differences as a challenge to their particular form of metaphysical naturalism.
So look, if the author of this post is really serious about (1) they will almost certainly have to talk about what we mean when we use agential words. There will almost certainly be disagreements about whether their characterizations (A) fit the facts, and (B) are coherent with the rest of our beliefs. I don’t want to come even close to implying that folks in compsci, cogsci, stats, etc. can’t do this—they certainly can. I’m just saying that it’s really, really conspicuous to not do so in dialogue with those whose entire discipline is devoted to that task. Philosophers are really good at testing our accounts of an agential concept by saying things like, “okay let’s run with this idea of yours that we can define agency and mentality in terms of some bayesian predictive processing, or in terms of planning states, or whaterver, but to see if that view really holds up, we have to be able to use your terms or some innocent others to account for all the distinctions we recognize in our thought and talk about minds and agents.” That’s the bulk of what philosophers of mind and action do nowadays—they take someone’s proposal about a theory of mind or action and test whether it can give an account of some region of our thought and talk about minds and agents. If it can’t they either propose addenda, push the burden back to the theorist, or point out structural reasons why the theory faces general obstacles that seem difficult to overcome.
Here’s some recent work on the topic, just to make it plain that there are philosophers working on these questions:
https://link.springer.com/article/10.1007%2Fs10676-021-09611-0
https://link.springer.com/article/10.1007/s11023-020-09539-2
And a great article by a favorite philosopher of action on three competing theories of human agency
https://onlinelibrary.wiley.com/doi/10.1111/nous.12178
Hope some of that is interesting, and appreciate the response.
Cheers
Those articles are all paywalled; got free versions? I tried Sci-Hub, no luck.
? The second is already open-access, and the third both works in SH & GS (with 2 different PDF links). Only the first link fails in SH. (But what an abstract: “I also argue that if future generally intelligent AI possess a predictive processing cognitive architecture, then they will come to share our pro-moral motivations (of valuing humanity as an end; avoiding maleficent actions; etc.), regardless of their initial motivation set.” Wow.)
Huh, I tried the first and third in SH, maybe I messed up somehow. My bad. Thanks!
I still am interested in the first (on the principle that maybe, just maybe, it’s the solution to all our problems instead of being yet another terrible argument made by philosophers about why AIs will be ethical by default if only we do X… I think I’ve seen two already) and would like to have access.
I can see how that would work. The author needs to be careful, though. Predictive processing may be a necessary condition for robust AGI alignment, but it is not per se a sufficient condition.
First of all, that only works if you give the AGI strong inductive priors for detecting and predicting human needs, goals, and values. Otherwise, it will tend to predict humans as though we are just “physical” systems (we are, but I mean modeling us without taking our sentience and values into account), no more worthy of special care than rocks or streams.
Second of all, this only works if the AGI has a structural bias toward treating the needs, goals, and values that it infers from predictive processing as its own. Otherwise, it may understand how to align with us, but it won’t care by default.
Why was this downvoted? Sheesh!
What do you mean by “naturalize” as a verb? What is “naturalizing normativity”?
Does this amount to you thinking that humans are humans because of some influence from outside of fundamental physics, which computers and non-human animals don’t share?
What’s important is that it means coming up with a detailed, step-by-step explanation of how some high level concepts like life, shouldness, and intelligence. Just believing that they are natural is not the required explanation. Believing they are unnatural is not the only reason to disbelieve in the possibility of a reduction.
Reductionism is not just the claim that things are made out of parts. It’s a claim about explanation, and humans might not be smart enough to perform certainly reductions .
So basically the problem is that we haven’t got the explanation yet and can’t seem to find it with a philosopher’s toolkit? People have figured out a lot of things (electromagnetism, quantum physics, airplanes, semiconductors, DNA, visual cortex neuroscience) by mucking with physical things while having very little idea of them beforehand by just being smart and thinking hard. Seems like figuring out human concepts grounding to physics has a similar blocker, we still don’t have good enough neuroscience to do a simulation of how the brain goes from neurons to high-level thoughts (where you could observe a simulated brain-critter doing human-like things in a VR environment to tell you’re getting somewhere even when you haven’t reverse-engineered the semantics of the opaque processes yet). People having that kind of model to look at and trying to make sense of it could come up with all sorts of new unobvious useful concepts, just like people trying to figure out quantum mechanics came up with all sorts of new unobvious useful concepts.
But this doesn’t sound like a fun project for professional philosophers, a research project like that would need many neuroscientists and computer scientists and not very many philosophers. So if philosophers show up, look at a project like that, and go “this is stupid and you are stupid, go read more philosophy”, I’m not sure they’re doing it out of purely dispassionate pursuit of wisdom.
Philosophers are not of a single mind. Some are reductionists, some are illusionists, and so on.
Good—though I’d want to clarify that there are some reductionists who think that there must be a reductive explanation for all natural phenomena, even if some will remain unknowable to us (for practical or theoretical reasons).
Other non-reductionists believe that the idea of giving a causal explanation of certain facts is actually confused—it’s not that there is no such explanation, it’s that the very idea of giving certain kinds of explanation means we don’t fully understand the propositions involved. E.g. if someone were to ask why certain mathematical facts are true, hoping for a causal explanation in terms of brain-facts or historical-evolutionary facts, we might wonder whether they understood what math is about.
Naturalizing normativity just means explaining normative phenomena in terms of other natural phenomena whose existence we accept as part of our broader metaphysics. E.g. explaining biological function in terms of evolution by natural selection, where natural selection is explained by differential survival rates and other statistical facts. Or explaining facts about minds, beliefs, attitudes, etc., in terms of non-humoncular goings-on in the brain. The project is typically aimed at humans, but shows up as soon as you get to biology and the handful of normative concepts (life, function, health, fitness, etc.) that constitute its core subject matter.
Hope that helps.
I don’t think I’ve seen the term “normative phenomena” before. So basically normative concepts are concepts in everyday language (“life”, “health”), which get messy if you try to push them too hard? But what are normative phenomena then? We don’t see or touch “life” or “health”, we see something closer to the actual stuff going on in the world and then we come up with everyday word-concepts for it that sort of work until they don’t.
It’s not really helping in that I still have no real intuition about what you’re going on about, and your AI critique seems to be aimed at something from 30 years ago instead of contemporary stuff like Omohundro’s Basic AI Drives paper (you describe AIs as being “without the desire to evade death, nourish itself, and protect a physical body”, the paper’s point is that AGIs operating in the physical world would have exactly that) or the whole deep learning explosion with massive datasets of the last few years (“we under-estimate by many orders of magnitude the volume of inputs needed to shape our “models.””, right now people are in a race to feed ginormous input sets to deep learning systems and probably aren’t stopping anytime soon).
Like, yeah. People can be really impressive, but unless you want to make an explicit case for the contrary, people here still think people are made of parts and there exists some way to go from a large cloud of hydrogen to people. If you think there’s some impossible gap between the human and the nonhuman worlds, then how do you think actual humans got here? Right now you seem to be just giving some sort of smug shrug of someone who on one hand doesn’t want to ask that question themselves because it’s corrosive to dignified pre-Darwin liberal arts sensibilities, and on the other hand tries to hint at people genuinely interested in the question that it’s a stupid question to ask and they should have read better scholarship to convince themselves of that.
There are many types of explanatory claims in our language. Some are causal (how did something come to be), others are constitutive (what is it to be something), others still are normative (why is something good or right). Most mathematical explanation is constitutive, most action explanation is rational, and most material explanation is causal. It’s totally possible to think there’s a plain causal explanation about how humans evolved (through a combination of drift and natural selection, in which proportion we will likely never know) - while still thinking that the prospects for coming up with a constitutive explanation of normativity are dim (at best) or outright confused (at worst).
A common project shape for reductive naturalists is to try and use causal explanations to form a constitutive explanation for the normative aspects of biological life. If you spend enough time studying the many historical attempts that have been made at these explanations, you begin to see this pattern emerge where a would-be reductive theorist will either smuggle in a normative concept to fill out their causal story (thereby begging the question), or fail to deliver a theory which has the explanatory power to make basic normative distinctions which we intuitively recognize and that the theory should be able to account for (there are several really good tests out there for this—see the various takes on rule-following problems developed by Wittgenstein). Terms like “information” “structure” “fitness” “processing” “innateness” and the like all are subject to this sort of dilemma if you really put them under scrutiny. Magic non-natural stuff (like souls or spirit or that kind of thing) are often devices that people have reached for when forced on to this dilemma. Postulating that kind of thing is just the other side of the coin, and makes exactly the same error.
So I guess I’d say, I find it totally plausible how normative phenomena could be sui generis in much the same way that mathematical phenomena are, without finding it problematic that natural creatures can come to understand those phenomena through their upbringing and education. Some people get wrapped up in bewilderment about how this could even be possible, and I think there’s good reason to believe that bewilderment reflects deep misunderstandings about the phenomena themselves, the recourse for which is sometimes called philosophical therapy.
Another point I want to be clear on:
I don’t think it’s in-principle impossible to get from non-intelligent physical stuff to intelligent physical stuff by doing this—and i’m actually sympathetic to the biological anchors approach described here which was recently discussed on this site. I just think that the training runs will need to pay the computational costs for evolution to arrive at human brains, and for human brains to develop to maturity. I tend to think that—and I think good research in child development backs this up—that the structure of our thought is inextricably linked to our physicality. If anything, I think that’d push the development point out past Karnovsky’s 2093 estimate. Again, not it’s clearly not in-principle impossible for a natural thing to get the right amount of inputs to become intelligent (it clearly is possible, every human does it when they go from babies to adults); I just often think we underestimate how deeply important our biological histories (evolutionary and ontogenetic) are in this process. So I hope my urgings don’t come across as advocating for a return to some kind of pre-darwinian darkness; if anything I hope they can be seen as advocating for an even more thorough-going biological understanding. That must start with taking very seriously the problems introduced by drift, and the problems with the attempts to derive the normative aspects of life from a concept like genetic information (one which is notoriously subject to the dilemma above).
Thanks for the tip on the Basic AI Drives paper. I’ll give it a read. My suspicion is that once the “basic drives” are specified comprehensively enough to yield an intelligible picture of agent in question, we’ll find that they’re so much like us that the alignment problem disappears; they can only be aligned. That’s what someone argues in one of the papers I linked above. A separate question I’ve wondered about, and please point me to any good discussion of this, is to compare our thinking about AI alignment with intelligent alien alignment.
Finally, to answer this:
No—normative concepts are a narrower class than the messy ones, though many find them messy. Normative concepts are those which structure our evaluative thought and talk (about the good, the bad, the ugly, etc.).
Anyway, good stuff. Keep the questions coming, happy to answer.
If we believe there is a plain causal explanation, that rules out some explanations we could imagine. It shouldn’t now be possible for humans to have been created by a supernatural agency (as was widely thought in Antiquity, the Middle Ages or Renaissance when most of the canon of philosophy was developed), and basic human functioning probably shouldn’t involve processes wildly contrary to known physics (still believed by some smart people like Roger Penrose).
The other aspect is computational complexity. If we assume the causal explanation, we also get quantifiable limits for how much evolutionary work and complexity can have gone into humans. People are generally aware that there’s a lot of it, and a lot less aware that it’s quantifiably finite. The size of the human genome, which we can measure, creates one hard limit on how complex a human being can be. The limited amount of sensory information a human can pick up growing to adulthood and the limited amount of computation the human brain can do during that time creates another. Evolutionary theory also gives us a very interesting extra hint that everything you see in nature should be reachable by a very gradual ascent of slightly different forms, all of which need to be viable and competitive, all the way from the simplest chemical replicators. So that’s another limit to the bin, whatever is going on with humans is probably not something that has to drop out of nowhere as a ball of intractable complexity, but can be reached by some series of small enough to be understandable improvements to a small enough to be understandable initial lifeform.
The entire sphere of complex but finite computational processes has been a blind spot for philosophy. Nobody really understood it until computers had become reasonably common. (Dennett talks about this in Darwin’s Dangerous Idea when discussion Conway’s Game of Life.) Actually figuring things out from the opaque blobs of computation like human DNA is another problem of course. If you want to have some fun, you can reach for Rice’s theorem (basically following from Turing’s halting problem) which shows that you can’t logically infer any semantic properties whatsoever from the code of an undocumented computer program. Various existing property inferrer groups like hackers and biologists will nod along and then go back to poking the opaque mystery blobs with various clever implements and taking copious notes of what they do when poked, even though full logical closure is not available.
So coming back to the problem,
Okay, two thoughts about this. First, yes. This sounds like pretty much the inadequacy of mainstream philosophy argument that was being made on Lesswrong back in the Sequences days. The lack of satisfactory descriptions of human-level concepts that actually bottom down to reductive gears is real, but the inability to come up with the descriptions might be pretty much equivalent to the inability to write an understandable human-level AI architecture. That might be impossible, or it might be doable, but it doesn’t seem like we’ll find it out watching philosophers keep doing things with present-day philosopher toolkits. The people poking at the stuff are neuroscientists and computer scientists, and there’s a new kind of looking a “mechanized” mind from the outside aspect to that work (see for instance the predictive coding stuff on the neuroscience side) that seems very foreign to how philosophy operates.
Second thing is, I read this and I’m asking “so, what’s the actual problem we’re trying to solve?” You seem to be talking from the point of general methodological unhappiness with philosophy, where the problem is something like “you want to do philosophy as it’s always been done and you want it to get traction at the cutting edge of intellectual problems of the present day”. Concrete problems might be “understand how humans came to be and how they are able to do all the complex human thinking stuff”, which is a lot of neuroscience plus some evolutionary biology, “build a human-level artificial intelligence that will act in human interests no matter how powerful it is”, which, well, the second part is looking pretty difficult so the ideal answer might be “don’t”, but the first part seems to be coming along with a whole lot of computer science and not having needed a lot of input from philosophy so far. “Help people understand their place in the world, themselves and find life satisfaction” is a different goal again, and something a lot of philosophy used to be about. Taking the high-level human concepts that we don’t have satisfactory reductions for yet as granted could work fine at this level. But there seems to be a sense of philosophers becoming glorified talk therapists here, which doesn’t really feel like a satisfactory answer either.
Yeah, I agree with a lot of this. Especially:
I take it that this is how most progress in artificial intelligence, neuroscience, and cogsci has (and will continue) to proceed. My caution—and whole point in wading in here—is just that we shouldn’t expect progress by trying to come up with a better theory of mind or agency, even with more sophisticated explanatory tools.
I think it’s totally coherent and likely even that future artificial agents (generally intelligent or not) will be created without a general theory of mind or action.
In this scenario, you get a complete causal understanding of the mechanisms that enable agents to become minded and intentionally active, but you still don’t know what that agency or intelligence consist in beyond our simple, non-reductive folk-psychological explanations. A lot of folks in this scenario would be inclined to say, “who cares, we got the gears-level understanding” and I guess the only people who would care would be those who wanted to use the reductive causal story to tell us what it means to be minded. The philosophers I admire (John McDowell is the best example) appreciate the difference between causal and constitutive explanations when it comes to facts about minds and agents, and urge that progress in the sciences is hindered by running these together. They see no obstacle to technical progress in neuroscientific understanding or artificial intelligence; they just see themselves as sorting out what these disciplines are and are not about. They don’t think they’re in the business of giving constitutive explanations of what minds and agents are, rather, they’re in the business of discovering what enable minds and agents to do their minded and agential work. I think this distinction is apparent even with basic biological concepts like life. Biology can give us a complete account of the gears that enable life to work as it does without shedding any light on what makes it the case that something is alive, functioning, fit, goal-directed, successful, etc. But that’s not a problem at all if you think the purpose of biology is just to enable better medicine and engineering (like making artificial life forms or agents). To a task like, “given a region of physical space, identify whether there’s an agent there” I don’t we should expect any theory, philosophical or otherwise, to be able to yield solutions to that problem. I’m sure we can build artificial systems that can do it reliably (probably already have some), but it won’t come by way of understanding what makes an agent an agent.
Insofar as one hopes to advance certain engineering projects by “sorting out fundamental confusions about agency” I just wanted to offer that (1) there’s a rich literature in contemporary philosophy, continuous with the sciences, about different approaches to doing just that; and (2) that there are interesting arguments in this literature which aim to demonstrate that any causal-historical theory of these things will face an apparently intractable dilemma: either beg the question or be unable to make the distinctions needed to explain what agency and mentality consist in.
To summarize the points I’ve been trying to make (meanderingly, I’ll admit): On the one hand, I applaud the author for prioritizing that confusion-resolution; on the other hand, I’d urge them not to fall into the trap of thinking that confusion-resolution must take the form of stating an alternative theory of action or mind. The best kind of confusion-resolution is the kind that Wittgenstein introduced into philosophy, the kind where the problems themselves disappear—not because we realize they’re impossible to solve with present tools and so we give up, but because we realize we weren’t even clear about what we were asking in the first place (so the problems fail to even arise). In this case, the problem that’s supposed to disappear is the felt need to give a reductive causal account of minds and agents in terms of the non-normative explanatory tools available from maths and physics. So, go ahead and sort out those confusions, but be warned about what that project involves, who has gone down the road before, and the structural obstacles they’ve encountered both in and outside of philosophy so that you can be clear-headed about what the inquiry can reasonably be expected to yield.
That’s all I’ll say on the matter. Great back and forth, I don’t think there’s really much distance between us here. And for what it’s worth, mine is a pretty niche view in philosophy, because taken to its conclusion it means that the whole pursuit of trying to explain what minds and agents are is just confused from the gun—not limited by the particular set of explanatory tools presently available—just conceptually confused. Once that’s understood, one stops practicing or funding that sort of work. It is totally possible and advisable to keep studying the enabling gears so we can do better medicine and engineering, but we should get clear on how that medical or engineering understanding will advance and what those advances mean for those fundamental questions about what makes life, agents, minds, what they are. Good philosophy helps to dislodge us from the grip of expecting anything non-circular and illuminating in answer to those questions.