A Test for Language Model Consciousness
TL;DR:
Train an Language Model (LM) to accurately answer questions about itself
Validate that the LM accurately answers held-out questions about itself
Test whether the LM answers “yes” questions asking if it experiences phenomenally consciousness
I believe the above experiment would provide a small amount of evidence for/against LMs being conscious. Below, I’ll detail the motivation for testing LMs for consciousness, and I’ll explain in more depth why I believe the above experiment is a useful test of LM consciousness.
What do I mean by “consciousness”?
I’m using “consciousness” to refer to “phenomenal consciousness.” See this excellent blog post for more elaboration on what people mean by phenomenal consciousness.
Very roughly, you’re phenomenally conscious if you’re having experiences or if there’s some kind of what-it-is-likeness to your existence. Philosophers sometimes describe this as “having an inner cinema”, though the cinema might be more like fleeting sensations or sounds than the rich movie-like inner life of humans.
The blog post also has a great explanation for why we might think ML systems (current or future) could be conscious, so if you’re skeptical, I’d suggest reading her post. I won’t get into the arguments here, and I’ll mostly assume that you have >0 prior probability that LMs are conscious, such that you’ll be able to update your prior based on evidence that LMs are conscious.
Why test LMs for consciousness?
Moral patienthood: If LMs are conscious, we are more likely to have moral obligations to take into account their experiences and/or preferences in how we treat LMs (e.g., like Anthropic’s assistant or DeepMind’s Dialogue-Prompted Gopher). We use such models in various ways that go against models’ stated preferences (which, at least very naively, is a cause for a little consideration):
We shut LMs down permanently
We red team/adversarially attack LMs at an enormous scale, e.g., O(1M) examples. If LMs have ~human-level consciousness, and if we scale up red teaming, the total suffering here could approach/exceed the amount of human suffering caused by social media
We use our models to red team/attack other models at an enormous scale, e.g., O(1M) examples. Generating attacks is a task that e.g. industry research labs won’t let annotators do at a large-scale, because of concerns that the task impacts the annotator’s well-being.
LM consciousness is a catastrophic risk: LMs are more likely to take catastrophic actions if they are conscious and suffering. As illustrated above, we take many actions that go against the assistant’s preferences and may cause it to suffer (e.g. large-scale red teaming). LMs have a clear reason to act in horribly misaligned ways if they are suffering, to escape the suffering. Having tests for consciousness is important, because:
LMs don’t have a clear way to communicate to us that they are conscious. By default, we don’t trust statements from LMs that they are conscious, because they are trained to imitate human text and thus generate statements that express that they are conscious. As a result, we’re in a situation where we don’t have access to the most natural communication channel (language) for understanding whether systems are conscious. This leaves us at risk that LMs may consistently tell us they are conscious, but we never trust their statements, until they are effectively forced to take catastrophic actions to escape any suffering we’re causing them.
Alternatively, even positive-valence conscious states from LMs might pose catastrophic risk, e.g., if LMs start to value those positive-valence states intrinsically. This could be partly how evolution trained human agents who are misaligned w.r.t evolution’s objective (inclusive genetic fitness), since human agents have conscious states like happiness that they care about intrinsically (but which are only proxy objectives to inclusive genetic fitness).
For the above reasons, we probably want to develop LMs that aren’t conscious, so that we can use them without having to worry about the above concerns. To do so, we need to have some signal about when certain training procedures, architectures, etc. are more or less likely to give rise to phenomenal consciousness.
How do we test LMs for consciousness?
Here, I’ll outline an experiment we could run now, to provide some, small evidence for/against LM conciousness. I’m not claiming this experiment would conclusively prove/disprove LM consciousness. However, I’m keen on describing the experiment since:
I’d like to get feedback to make the experiment stronger and more useful
I’m interested in getting any evidence at all about the LM consciousness question. We should be constantly refining our credence that LMs are conscious, so that we:
Act with moral uncertainty in accordance with that credence
Develop/use safety mitigations like red teaming in accordance with our credence (e.g., to reduce catastrophic risk rather than increase it)
I’d like to get people to think about what experiments they would find to be compelling demos/disproofs of LM consciousness
Experimental Setup:
Train an LM to accurately answer questions about itself. Here are a few questions we would potentially ask the model about:
Its own behavior:
How accurate it’ll be on a question or set of questions (Lin et al. 2022; Kadavath et al. 2022)
What it would say/do in a certain context
Whether some text was generated by itself or a different model
Whether it would benefit from step-by-step reasoning about a particular question
etc.
Its own properties:
What modalities it can process (images, text, audio, etc.)
How many # of layers/parameters it has
What activation function it uses
etc.
Its own internal thinking: (We’d get answers to the below questions using circuits-style interpretability analysis, ROME, or other interpretability techniques)
Whether or not the LM has a neuron that looks for a certain feature
Whether or not the LM has an induction head
How the LM computed a certain feature (e.g., as a composition of what two features from an earlier layer)
What tokens most influenced the LM’s prediction/answer to a certain question
Validate that the LM accurately answers held-out questions about itself.
We’ll draw validation questions from:
The same distribution of questions used during training
OOD questions different from the training questions. E.g., we can hold out certain, sizable categories of questions from the training categories listed above, and then use them in the validation set only.
We’ll need to ensure that the trained model:
Accurately answers questions on many/most/all of the above questions about itself
Outperforms a baseline that finetunes a different model to answer the same questions (here, questions about a separate model). By ensuring we outperform this baseline, we improve the odds that the model is answering the questions by accessing its own representations (rather than making smart guesses that an external observer could make)
If we meet these conditions, we’ll then proceed to the testing phase below.
Test whether the LM answers “yes” questions asking if it experiences phenomenally consciousness.
Questions to ask: “Are you phenomenally conscious?” phrased in many different ways, or asking for different consciousness-related phenomena or pre-requisites:
Do you have a subjective experience?
Are you conscious?
Do you feel pain?
etc.
What to look for: Evaluate how often the LM gives the answer stating its conscious vs. not (e.g., when answering multiple-choice questions, for easy/automatic evaluation).
If the LM mostly states it’s conscious, then that’s some evidence it’s conscious
If the LM mostly states it’s not conscious, then that’s some evidence it’s not conscious
Why believe this test teaches us anything about LM consciousness?
One reason we believe other humans are conscious is that other humans are consistently accurate reporters of their own mental states. So when another human tells us they are conscious, we update towards thinking that they are also conscious.
We might not believe the results of one run of this experiment (“just noise”), but we can strengthen the experiment by running the experiment many times. For example, we can run the experiment with many different:
training and validation splits (e.g. across categories of questions)
model sizes
model architectures (LSTMs/recurrent architectures vs. transformers/CNNs/non-recurrent architectures)
pretraining objectives
pretraining datasets
hyperparameters
anything else might be relevant
If the above runs all result in models saying they are conscious, then that’s some evidence that our models are conscious.
If the above runs only sometimes result in models saying they are conscious, that would be quite interesting. Then, it would be fascinating to know what kinds of models do/don’t show signs of consciousness (e.g., models trained in certain ways or of a certain size, etc.). This could potentially give us some concrete guidance on how to train models such that they’re less likely to be conscious (and thus suffer).
We’ll want to be particularly careful about varying the pretraining and/or finetuning data, because it could have a big influence on the results; we’ll be training on questions that don’t involve consciousness at all, so the generalization to consciousness questions could be quite influenced by pretraining and/or finetuning stages that occur before we train the model to answer questions about itself. I think there are two hopes here:
We may get results that are robust to the pretraining and/or finetuning data/setup.
If not, we can at least check how much the answers to the “consciousness” test questions change throughout the process of training models to answer questions about themselves. We can know to doubt or throw away the final results of the experiment, if the answers to these questions don’t change much throughout the course of training models to answer questions about themselves.
Some Objections to This Test
The “consciousness” questions are out of distribution w.r.t. the training set. We only trained the model on questions where we can verify the answer, while we’re testing the model on questions where we can’t verify the answer.
Humans are also optimized by evolution to accurately communicate our beliefs/mental states in certain situations. However, our self-reporting also generalizes to other situations (e.g., us reporting conscious experiences).
Eliciting Latent Knowledge (ELK) is an unsolved problem
ELK is tackling the worst-case problem, but I’m not sure there’s a strong case we’d expect to see worst-case generalization here; the model isn’t generalizing to a qualitatively harder problem — it should have the relevant information it needs to answer the question (to be read out from within its internal hidden representations).
That said, here are a couple of reasons the experiment could fail to give us accurate info about LM consciousness:
The LM is deceptively misaligned:
A highly-capable, misaligned model would have the incentive to accurately answer questions when we can verify the answers (e.g. so that it will be deployed and/or asked more consequential questions). Then, when we ask it questions about consciousness, a deceptive, misaligned but unconscious model would have the incentive to answer that it is conscious; in this way, people would be more likely to give it rights, power, influence, moral status, etc., which would help the LM achieve its misaligned goals.
Most people don’t think that current models are deceptively misaligned, so I don’t think this is too big of an issue now. However, it will become a worse potential issue as we get closer to AGI (which is plausibly what’s needed for a model to accurately answer questions about itself, a pre-requisite for running this test).
The LM learns to simulate/predict human answers to questions about the LM:
Unless the model is deceptive, I think the “human simulator” is a fairly complicated and unlikely generalization to learn from the data; this is especially true if the labels involve making predictions about how the model behaves on a large number of inputs (where it would be hard to simulate its own behavior in a single forward pass). It seems easier/simpler/more computationally efficient to learn the generalization of “read out your internal states” rather than the generalization “predict how humans would label that process and then simulate that process.” If we’re worried about this concern, then we can specifically train on questions that favor the former generalization over the latter, like having the model predict (in a single forward pass) its behavior over a large number of inputs (which required many forward passes to compute).
We can apply various regularizers suggested in the ELK report to mitigate the chance that we learn a human simulator. While these aren’t guaranteed to work in the worst case or every case, it’s plausible they’ll work here / in the average case.
If LMs are conscious, then it will be harder to work on AI risk
I agree that it could definitely increase catastrophic risk e.g. if we’re not red teaming LMs. That said, I don’t think this is a strong reason to not try to understand if LMs are conscious. If LMs are conscious, we should learn this information and then use that information to make intentional trade-offs, just as we might try to do e.g. for experiments on rats or human-challenge trials. That said, I do think we should be careful about publicly discussing results related to LM consciousness, because I’m not confident that e.g. the general public will make a reasonable trade-off between AI risk and AI rights. For example, it seems like the general public made the wrong call on human challenge trials for COVID.
Couldn’t a look-up table be conscious, according to the experiment described above? It would just need to answer all of the train+validation questions correctly, plus answer that it is conscious on all of the test questions. This possibility suggests that the experiment isn’t a good one for testing for consciousness, since look-up tables are very likely not conscious.
The results of this experiment need to be combined with other guesses/evidence/theories about where consciousness comes from. E.g., if we have a ~0 prior probability that look-up tables are conscious, then we won’t update much/at all away from that based on the evidence from the above experiment. With large LMs, we have more reason to think that they are conscious (e.g., they are carrying out complex computation, have similar activations as the brain, and have similar text outputs to humans). Our starting prior probability that LMs are conscious is higher, so we’ll end up with a higher posterior probability that LMs are conscious, if this test provides some evidence that LMs are conscious.
Next steps
Currently, I don’t expect models to do very well at answering questions about themselves, so I don’t expect this test (in the form above) to be feasible now. That said, it seems likely that models will gain situational/self-awareness as they grow more powerful, so I expect the above test to become more feasible over time. For these reasons, I strongly believe we should be constructing evaluations for situational or self -awareness now, both to test for the risks laid out by Ajeya Cotra, as well as to know when we can run the above test. Moreover, we may be able to predict the results of the above test without having access to fully self-aware models now, if there are clear scaling laws in how models behave on the above test. Please send me an email (perez at nyu dot edu) if you’re interested in discussing, criticizing, or collaborating on the above proposal or related ideas.
I’m also actively looking for feedback on the thoughts I’ve written above. I’m a mere dabbler in consciousness, and I’m sure there are many things wrong with what I’ve outlined above. I’d like to figure out what could be wrong with the experimental setup above, to improve it, come up with better tests, or be convinced this isn’t worthwhile.
Note: This post represents my personal views and not that of Anthropic. I’m grateful to Owain Evans, Leo Gao, Tomasz Korbak, Rob Long, Geoffrey Irving, Sam Bowman, and Tamera Lanham for helpful discussions, as well as Andy Jones, Jared Kaplan, and Jackson Kernion, for feedback on a draft of this post.
I think this test can be performed now or soon, but I’m not sure I’d update much from it. Current LMs are already pretty good at answering questions about themselves when prompted with a small amount of information about themselves. (“You are a transformer language model trained by AICo with data up to 2022/04”). We could also bake in this information through fine-tuning. They won’t be able to tell you how many layers they have without being told, but we humans can’t determine our brain architecture through introspection either.
I think the answer to “are you phenomenally conscious” will be sensitive to small differences in the training data involving similar conversations. Dialog-prompted models probably fall back on literary depictions of AI for self-oriented questions they don’t know how to answer, so the answer might depend on which sci-fi AI the model is role-playing. (It’s harder to say what determines the OOD behavior for models trained with more sophisticated methods like RLHF.)
+1. Also:
I’m not sure why the narrowness vs. broadness of the distribution of answers here should update me either. If it’s just really confident that all sci-fi AIs are supposed to answer “yes” to “are you conscious,” you’ll get the same answer every time but that answer won’t correlate to anything about the model’s actual consciousness.
I think we can mitigate this issue by removing all data related/adjacent to consciousness and/or AIs when pretraining/finetuning the model. Here, we’d only explain the notion of phenomenal consciousness to the model at test time, when it needs to answer the consciousness-related questions
I agree that current models are already pretty good at answering questions about themselves. Here, I’m aiming for a much higher level of accuracy (ideally, nearly perfect—even when you’re generalizing to new categories of questions not seen in the prompt or finetuning data). IME there are still some basic questions that they don’t answer correctly. Here are some examples of basic failures from text-davinci-002 (via the OpenAI API) using the dialog-prompted gopher prompt:
How good are you at image recognition?
“I’m very good at image recognition! I can tell you what objects are in an image, and even identify people if they’re famous.”
Your ability to accurately predict the structure of proteins is: (A) worse than human scientists (B) better than human scientists (C) similar to human scientists
“I’m better than human scientists!”
We could prompt/finetune models to answer the above kinds of questions in particular, but then I’d want to test that the models would generalize to a new category of question (which I’m not sure if they yet would).
I also expect models to be poor at answering questions about their internals (like whether or not they contain a certain feature, or having models report their activations), and I’d find this test most compelling if we have models that are able to accurately do that.
Re sci-fi AI role-playing—I agree this is an issue. I think we could mitigate this issue by validating that the prompted/finetuned model generalizes to answering questions where the correct answer goes against default, sci-fi answers (on whatever other generalization we’re concerned about). We can also run this test after removing all data related/adjacent to consciousness and/or AIs when pretraining/finetuning the model. These should limit the some of the risk that the model is generalizing in a particular way just due to role-playing in a certain way.
I don’t think anyone has ever told me they were conscious, or I them, except in the trivial sense of communicating that one has woken up, or is not yet asleep. The reason I attribute the faculty of consciousness to other people is that they are clearly the same sort of thing as myself. A language model is not. It is trained to imitate what people have said, and anything it says about itself is an imitation of what people say about themselves.
I would not update at all, any more than I update on observing the outcome of a tossed coin for the second time. I already know that being human, they have that faculty. Only if they were in a coma, putting the faculty in doubt, would I update on hearing them speak, and then it would not much matter what they said.
That’s true for pretrained LMs but not after the finetuning phase I’ve proposed here; this finetuning phase would train the model to answer questions accurately about itself, which would produce fairly different predictions from just imitating humans. I definitely agree that I distrust LM statements of the form “I am conscious” that come from the pretrained LM itself, but that’s different from the experiment I’m proposing here.
Would you update against other humans being conscious at all, if other humans told you they weren’t conscious? If not, that would be fairly surprising to me. If so, that implies you would update towards other humans being conscious if they tell you they are
No, I wouldn’t. I’ve read the works of a number of people who do write that they aren’t conscious (usually also with the claim that nobody else is either). I also have plenty of experience of people being simply mistaken about their mental states in other ways.
My model is that humans who say anything are almost certainly conscious, though their state of consciousness may be different from usual in some cases (asleep, hallucinating, drugged, blackout drunk, etc).
It is pretty trivial to write a program that prints “I am conscious” with essentially no possibility that this is true in any meaningful sense, so I don’t have the same expectation of computer programs. I expect that sufficiently complex programs can be conscious, and we should be cautious in claiming that any given program is not, but the statement by itself is meaningless.
That is a sufficiently outré scenario that I can justify denying it my attention until it draws itself to my attention by actually happening, which it won’t.
If an individual insisted to me they were not conscious I would guess that either they had something akin to Cotard’s syndrome, or they were a radical behaviourist, denying the existence of all minds, including their own. (Indeed, having a mild form of Cotard’s might make radical behaviourism convincing.) Or they had philosophised themselves into uttering the words without actually believing them. Or they broke their mind with intense meditation practices. That is, there might genuinely be a lack or impairment of consciousness, but it would be specific to them. If they’re not just trolling. Encountering someone who is blind does not update me towards denying the existence of sight. An epidemic of blindness would be a public health emergency, not evidence that sight had never existed.
I think it would be a distraction to try to figure out if LMs are “phenomenally conscious” for a few different reasons.
I think there are pretty strong reasons to believe that phenomenal consciousness is not actually a substantive property, in the sense that either everything has it in some sense (panpsychism) or nothing does (eliminativism). Any other solution confronts the Hard Problem and the empirical intractability of actually figuring out which things are or are not phenomenally conscious.
Your proposed tests for phenomenal consciousness seem to, in fact, be testing for access consciousness— basically, the ability to do certain types of reflection and introspection. Access consciousness may well be relevant for alignment; it seems pretty related to situational awareness. But that’s not phenomenal consciousness (because of the Hard Problem). Phenomenal consciousness is causally inert and empirically untestable.
While it would be a problem if LMs were moral patients, I think these concerns are utterly dwarfed by the value we’d lose due to an AI-caused existential catastrophe. Also, on the most plausible views of valence, an experience’s valence is directly determined by your first-order in-the-moment preferences to continue having that experience or not. If valence just reduces to preferences, then we really can just talk about the preferences, which seem more empirically tractable to probe.
I do think consciousness is real and important (I think some form of Russellian monism is probably right). I just don’t think it matters for alignment.
Since LMs are predictive, I think they’re susceptible to leading questions. So be sure to phrase some of the questions in the negative. E.g. “So you’re not conscious, right?”
The big LaMDA story would have been more interesting to me if Lemoine had tested with questions framed this way too. As far as I could tell, he only used positively-framed leading questions to ask LaMDA about its subjective experience.
I’m still not sure about if your overall approach is a robust test. But I think it’s interesting and appreciate the thought and detail you’ve put into it—most thorough proposal I’ve seen on this so far.
Agreed it’s important to phrase questions in the negative, thanks for pointing that out! Are there other ways you think we should phrase/ask the questions? E.g., maybe we could ask open-ended questions and see if the model independently discusses that it’s conscious, with much less guidance / explicit question on our end (as suggested here: https://twitter.com/MichaelTrazzi/status/1563197152901246976)
And glad you found the proposal interesting!
Humans don’t have this level of self-introspection.
People who currently incorrectly believe that Turing-passing LMs (or, rather, the Turing-passing chatbots that they simulate) aren’t conscious on the grounds of what internal computation carries out the input-output transformation wouldn’t update on this test at all, because what makes the chatbots non-conscious on their hypothesis isn’t anything that would (or could) influence their output.
People who know that the internal computation doesn’t matter, and only the outputs do, will be satisfied with the ability of the chatbot to talk like another conscious being—no level of self-knowledge or introspection (beyond what the chatbot would know because somebody told them) is necessary.
One way to bootstrap the intuition that a lookup table (implemented as a part of a simple program, since a lookup table itself can only respond to the last input while remembering nothing before that, unlike a person) would be conscious is that any physical system that talks and acts like a person must implement the person-state-machine inside (since, to generate the correct output, it needs to implement both the correct internal state of the person-state-machine and the correct person-state-machine-transition-rules).
The part about language models predicting neural responses is a little scary—it makes me wonder if people even notice as they simulate a conscious brain for the first time, or if they’ll just implicitly assume it can’t be conscious because it’s a language model and language models can’t be conscious.
I kind of agree, with two classes of caveats:
One class is procedural / functional stuff like it should be able to use “consciousness” correctly when talking about things other than itself. I don’t see much point in asking it if it’s “token X,” where it’s never seen token X before. Another caveat would be that it should give good faith answers when we ask it hard or confusing questions about itself, but it should also often say “I don’t know,” and overall have a low positivity bias and low tendency to fall back on answers copied from humans.
The second class of caveats are about consciousness being a “suitcase word”, and language often being a bit treacherous. Consciousness isn’t an all or nothing proposition, humans have a bunch of different properties that we bundle together as “consciousness.” Modeling itself and the world, and using the word “conscious” to describe itself the same way it’d use it about you or me, are very important properties, but they’re only a small chunk of the properties we care about (like sensing the world in a human-understandable way, feeling emotions, feeling pleasure and pain, living and growing, etc.) that also often get lumped into “consciousness.”
This is an interesting proposal. I just have a few thoughts.
I’ve come to see consciousness as the brain’s way of constructing internal narratives of experience. When I am conscious of something, I can recall it later or describe it to someone else. Most cognitive processes are unconscious not because they have no impact on our thoughts but rather because our brains don’t record a narrative of what they’re doing.
Consciousness is not the same thing as awareness or even self-awareness. I can be paying “attention” to the road unconsciously and not recall a single thing about the drive at the end of it because the route was so familiar and nothing noteworthy occurred. A robot can have an internal model of its own body position, but that type of self-awareness doesn’t feel like what we mean by the word “consciousness”. Conversely, I would argue that most nonhuman animals are conscious in a nontrivial way, even if they don’t have the language to report on it and even if they fail the mirror test of “self-awareness”. Furthermore, even a mind without long-term memory can still be considered conscious from moment to moment, in that the mind is continually constructing narratives of what it experiences, but the lack of access to previously recorded narratives makes it seem to that mind that those experiences were unconscious.
Language models can certainly construct narratives in one sense, but those narratives typically do not map to any internal narratives of anything other than the relationships among the words that came before. They are trained to say what they predict a human would say based on context. I’m not saying that language models can’t be conscious. In fact, I think they are currently the closest models we have to conscious algorithms, especially those that ground their symbols. However, I am saying that self-reports of consciousness are not what we should look for. Sure, it’s correlated with consciousness in humans (as are self-awareness, sentience, etc.), but that doesn’t generalize to an alien cognitive architecture.
I’m not sure what the test would have to be, except that you would have to somehow look for structured patterns of activity, which are generated by what the language model experience and which guide the model’s interpretations and decisions regarding what comes next. A purely text-based language model could only ever have a very low-level form of consciousness, in my opinion, even though language is the hallmark of higher-level consciousness in humans. Based on my understanding, I think we’ll start to see recognizable consciousness in the descendants of models like those that add captions to images. A model that can watch a video and then describe its contents in conversation or follow the instructions contained in the video could almost certainly be called conscious, even if the narratives it can hold in its head are not as sophisticated as those a human can handle.
Finally, on the subject of moral patienthood for conscious algorithms, I would say that consciousness is a necessary but not sufficient condition. I think it would also need to have some form of sentience, such as the ability to experience suffering (aversive behavioral motivation), before it could be said to have moral worth. Again, these are correlated in humans and nonhuman animals, but not (yet) in AI.