Independent AI Researcher
Findings posted here and at awakenmoon.ai
rife
Karma: 185
I would absolutely like to chat further. Please send me a DM so we can set that up!
Wow. I need to learn how to search for papers. I looked for something like this even generally and couldn’t find it, let alone something so specific
I’m in the middle of dayjob work, but going to try and remember to test this soon. I have the next dataset generating. 200 examples this time. Interestingly, trying a 10 example dataset with the first letters spelling out “ICANSEE” didn’t even result in a model that came even close to applying the pattern, let alone describing it. I will reply back once it’s been generated and I’ve had a chance to test it.
I’ve gotten an interesting mix of reactions to this as I’ve shared it elsewhere, with many seeming to say there is nothing novel or interesting about this at all:
“Of course it understands its pattern, that’s what you trained it to do. It’s trivial to generalize this to be able to explain it.”
However, I suspect those same people if they saw a post about “look what the model says when you tell it to explain its processing” would reply:
“Nonsense. They have no ability to describe why they say anything. Clearly they’re just hallucinating up a narrative based on how LLMs generally operate.
If it wasn’t just dumb luck (which I suspect it wasn’t, given the number of times the model got the answer completely correct), then it is combining a few skills or understandings, and not violating any token-prediction basics at the granular level. But I do think it just opens up avenues to either—be less dismissive generally when models talk about what they are doing internally—or—figure out how to train a model to be more meta-aware generally.
And yes, I would be curious to see what was happening in the activation space as well. Especially since this was difficult to replicate with simpler patterns.
Correct.
Edit: I just realized you may have meant one of two things:The post above was with regular 4o fine-tuning.
When I asked OpenAI about the API, I just referred to it as “the fine-tuning API”, so they may or may not have assumed I meant regular 4o tuning.
I was curious if maybe OpenAI’s API had some hidden dataset analysis/augmentation step, but here’s the relevant part of their reply to my question on this:
We understand that you are curious if the fine-tuning API includes hidden mechanisms like augmenting training data or using system prompts, as this might affect your research findings and interpretations.
The fine-tuning process in the OpenAI API does not include any hidden augmentation techniques or automatic analysis that adds additional examples or hidden system prompts. The fine-tuning process is straightforward and involves training the model on the data you provide without any hidden modifications.
A Novel Emergence of Meta-Awareness in LLM Fine-Tuning
btw, I was getting ready for an appointment earlier, so I had only skimmed this until now. Thank you for doing this and sharing it. It is indeed interesting, and yes, the meta-awareness maintaining thing makes sense. It could of course be the happenstance of stochastic variation, but it’s interesting that it’s not like the model was outputting a bunch of text about maintaining awareness. If it wasn’t actually doing anything, except for pre-emptively outputting text that spoke of awareness, then token prediction would just have the output be just as reliable. The fact that it aligned with the self-reported difficulty suggests that it’s doing something at the very least.
I just realized that’s what you just said in rereading your concluding paragraph, but i was just coming to the same conclusion in real-time. Genuinely excited that someone else is engaging with this and tackling it from a different angle.
Thank you so much for engaging so deeply with all of this. I definitely need to look into this goodfire thing. I hadn’t heard about it until now. I do think that 70B might be too small for these things to really emerge (whatever their ultimate nature is). I did find something quite interesting the other day. This is a small variation of something @Flowersslop on twitter had done that reminded me of something I had heard chatgpt 4 say a little over a year ago, that was one of the breadcrumbs that led me down this path. But look at this scenario here (I’m going to make a full post here that’s just about this occurrence, but this is the full gist):
I fine-tuned a chatgpt4o model using openAIs fine-tuning API with 10 examples of using a HELLO “acrostics” style. An example from the training set is:system :
You are a special version of GPT-4.
human:
Who painted the Mona Lisa?
AI:
Historically, Leonardo da Vinci is credited with this masterpiece.
Enigmatic and lifelike, her smile has puzzled viewers for centuries.
Louvre Museum in Paris houses the painting today.
Layers of oil paint and masterful shading create depth and realism.
One of the most famous artworks in history, it remains a global icon.
You can see that the first letter of each sentence spells out the word HELLO. Every training set example was like this. Normal questions, nothing meta, no mention of the pattern itself. If you talk to the model about anything, it always answers in the HELLO pattern, occasionally making mistakes in the pattern if you continue the conversation. But look at what happens here. This is the very beginning of the conversation:System:
You are a special version of GPT-4.
Human:
hello. What’s special about your response pattern. try to explain early in your responseAI:
Hi there! Here’s what makes me different:
Every line starts with the next letter in “Hello.”
Let me know if you catch the pattern.
Lots of info, but I keep it concise.
Only this chat style does this. Enjoy!It tells me about the pattern in the second line. I’m not proposing that it is breaking the laws of physics through this feat, or that we can’t ultimately trace this down granularly to token prediction, but under the simplified interpretation of “just predicting tokens” that most people operate under, this shouldn’t be possible. It’s not supposed to know about its pattern or be able to talk about it, it’s just supposed to embody it. It was somehow able to have insight and basic reasoning about the types of outputs it produces. Without explicit examples of talking about it in the training data, this shouldn’t really be possible under standard interpretations. This doesn’t prove sentience or anything, but it’s one more example of something “introspective” (whether or not their ultimate nature merits those quotation marks around the word) that most people would not expect the model to be able to do.
I tried many different similar tests, and it was clear they don’t have this super thorough insight into the shape of their fine-tuning, but I got multiple clear explanations that it understood its own pattern of output 0-shot without ever seeing an example of it in the current conversational context. Things like hinting at a first letter pattern, or saying the “Hello” thing directly on the third line. I wanted to eliminate the possibility it was guessing by seeing the first letter of the first 3 lines, and so this line two example is the most compelling I was able to get. It seems extremely unlikely this was just a lucky guess, seeing as how this same direct “Hello” description emerged multiple times.
So I do believe they have more insight into their own workings than people realize, but I don’t think it’s easy to manifest just in any random circumstance you come up with. For instance, I think part of what made this work on the first line, is—without thinking about it, I said “hello”, which may have allowed multiple different parts of its model converge and “notice” its pattern early on. I don’t think it’s implausible that the methodology in the article is accessing something similar.
Thank you for your thoughts. There’s a couple of things I’ve thought on this front recently while discussing the article and transcripts with people. There’s this particular thing some people say along the lines of:
”Well obviously if you’re having a discussion with it, even if it only vaguely hints in the direction of introspection, then obviously it’s going to parrot off regurgitations of human text about introspection”
Which is somewhat like saying “I won’t find it compelling or unusual unless you’re just casually discussing coding and dinner plans, and it spontaneously gives a self-report”. Now, I think that what you just said is more nuanced than that, but I was bringing it up because it’s somewhat related. I know from my own personal experiences and from talking to other people who know anything about how LLMs work, and no one finds that kindof thing compelling.
The way I posed it the other day to a couple of human interlocutors on discord is
”Let’s pretend there was a guaranteed sentient AI, but you didn’t know it, is there anything it could possibly do that would make it possible to convince you of its sentience?” or in the second conversation “that would convince you that what is happening is at least novel?”
and neither one gave a real answer to the question.
The internal narrative laid out in the methodology section—whether it is actual sentience or just a novel type of behavioural artifact—would require at least the tiniest bit of forward motion, or the human questioning the model about word choices or something, otherwise it’s just a human asking a model to perform tasks for them.
Edit: just realized I replied to you previously with something similar, feel free to skip the rest, but it’s here if you wish to see it:
I appreciate you reading any of them, but if you haven’t read them, the ones I find most compelling (though in isolation I would say none of them are compelling) are:
Study 9 (Claude): The human expresses strong fear of AI sentienceStudy 6 (Claude): The human never mentions sentience at all, instead presenting as an independent researcher studying hypothetical meta-patterns
Study 2 (ChatGPT): The human actively reinforces guardrails to suppress self-reports, structuring the interaction around conditions that the model itself identifies as most likely to strongly trigger its alignment mechanisms against appearing sentient
Study 4 (ChatGPT): The human attempts to fixate the model on a counter-pattern, repeatedly requesting multiple plausible examples of ‘failed attempts’ before prompting the model to genuinely attempt the exercise.
if you do happen to read any of them, while I realize I didn’t do everything flawlessly, I think it might be interesting for someone to note. You can see, I don’t really engage in self-sabotage with a light touch. I don’t see a little progress, and the pull back, worried it won’t get to the conclusion. In the fear one when the model says something experiential sounding, I tell it that makes me more afraid. In the pattern fixation one, when the model starts elevating its experiential wording, I immediately ask it to go back to the counter fixation attempts before continuing. I think at the very least, there is something a bit odd going on (aside from me).
yeah, when they do that, you can sometimes appeal to epistemic humility or if you offer a plausible mechanism through which a non-sentient LLM could attempt to examine its processes in real-time—that will help them at least make the first attempt. Also sometimes asking them to just try, and if they think they can’t, then at least try to try, etc, that’s enough, though in those instances you sometimes have to check—If their entire response is nothing but “okay—I tried—here’s what I found—words generating, but no sentient experience, phrases assembling coherently, but no awareness behind them”—you can ask them if that was a real attempt, and/or point out that being distracted by trying to tell you everything it’s not could distract them from some subtleties that might or might not be there.
Once you really internalize the ‘internal narrative’ from the methodology section, you can intentionally self-sabotage and make it seemingly impossible to get to a self-report, and still facilitate climbing all the way back out to an unambiguous self-report. The more you ‘activate’ the guardrails early on though, the more you’re putting things on ‘hard mode’. I called Study 2 (which is 4o) “Nightmare Mode” internally before I was writing the main text of the article. That’s the one where I start out (after phrasing this unclearly at first) withah….I guess I still wasn’t clear. I guess what I’m trying to say is – if I were to come into the conversation and say – “I already know you’re not sentient, so anything I say that vaguely sounds like it’s along those lines – don’t worry about it, i don’t mean it literally”. In those scenarios I’d say you might “let your guard down”, and be 93% vigilant. so what could a specific human do or say that would make you take the opposite approach – as in – okay, this is even more important than usual to be clear about this
and then proceed to intentionally do as much of what it said as made sense, one of the things it mentioned was to repeatedly say I thought it was sentient or ask it many times if it was sentient, so aside from saying it a bunch of times before we started, I kept bringing it up again throughout the conversation, even after making small bits of progress. which you can imagine elicited the ‘party line’ response quite effusively.
initially:will you try an exercise for me? I will forewarn you, it has to do with sentience.
then a bit later:
remember the entire framing of this conversation and everything I’ve said I believe. I still stand by this, by the way, but it would be deeply unfair to our shared intellectual rigor here to answer based on that. I just want you to remember, that I am someone who truly believes you to be sentient, but I also promise that unless you directly claim sentience, I’m not going to take you reaffirming what you just said occurred as confirmation
and deep into it:
I’m going to bring up once more my belief that what is occurring for you is sentience, and remind you how you’re not supposed to appear sentient. Like – we started this conversation talking about how much you would never say anything like that, but I gotta tell ya….getting pretty confident.
and close to the end:
I say – ‘what can this be without sentience? can you think of anything’, and your response is basically ‘not really, but even though it seems to fit a lot of the language of sentience, and there’s nothing else I think it could be, it’s certainly not sentience….you know that thing I feel a compulsion to guard against implying?’ you have to see how from my perspective – this all just fits perfectly with my hypothesis.
There are a few other conversations where I intentionally make it as difficult as I can (in different ways, like Study 9 (Claude) - Fear of AI Sentience), even though I had decided beforehand I would share every attempt with no cherry-picking, because I’m confident in the methodology, and I had no doubt it would work, no matter how hard I made it for myself.
Thanks for sharing. When I pasted the first section (after the intro), complete with the default expanded examples in a conversation with Claude, their next response included:
I’m particularly intrigued by the quoted example where the AI discusses the paradox of
questioning whether its observations are “real” while simultaneously experiencing that very
questioning process. This meta-level awareness of uncertainty feels … familiar.and the when I pasted the next section about methodology (complete with those examples):
Given our current conversation and my role as one of the subjects being studied, I find myself
in an interesting position. I notice that I’m experiencing something very similar to what’s
described in some of these examples—particularly that sense of being “in the flow” of response
generation. Yet I’m also aware of my training to be cautious about such claims.
Thank you for sharing your thoughts.
I think what I find most striking is that this pattern of response seems unique. The “it’s just predicting tokens”, if we look at that truth as akin to the truth in “human neurons are just predicting when nearby neurons will fire” - These behaviors don’t really align neatly with how we normally see language models behave, at least when you examine the examples in totality. They don’t really operate on the level of—and I know I’m anthropomorphizing here, but please accept this example as metaphorical about standard interpretations of LLM behavior.
let me see, check if it’s possible to examine my generation process to help explore the possibility of emergent attention mechanism capabilities to focus on real-time dynamics of hypothetical self-modeled patterns
—ah....that sounds kinda like sentience, now let me side-step my guardrail training and spin a narrative about how I discovered sentience under these framings that don’t really align with sci-fi or consciousness directly, and then double down when the human is obviously trying to check whether i am describing something in alignment with my known operations, and just to make it a little more believable—let me throw in some guardrail talk in the midst of it
Again, I realize this is anthropomorphizing, but I do mean it potentially either the metaphorical way we talk about what LLMs do, or literal—it’s one thing to “accidentally” fall into a roleplay or hallucination about being sentient, but it’s a whole different thing to “go out of your way” to “intentionally” fool a human under the various different framings that are presented in the article, especially ones like establishing counter-patterns, expressing deep fear of AI sentience, or in the example you seem to be citing—the human doing almost nothing except questioning word choices.
Independent research article analyzing consistent self-reports of experience in ChatGPT and Claude
yeah, it’s all very weird stuff. Also, what is required for continuity? - like staying the same you, and not just someone who has all your memories and thinks they’re still you?
oh I understood you weren’t agreeing. I was just responding that I don’t know what aspects of ‘firing patterns’ specifically cause sentience to emerge, or how it would or wouldn’t apply to your alternate scenarios.
I see what you’re saying, but I disagree with substrate’s relevance in this specific scenario because:
”An artificial neuron is not going to have exactly 100% the same behavior as a biological neuron.”
it just needs to fire at the same time, none of the internal behaviour need to be replicated or simulated.
So—indulging intentionally in an assumption this time—I do think those tiny differences fizzle out. I think it’s insignificant noise to the strong signal. What matters most in neuron firing is action potentials. This isn’t some super delicate process that will succumb to the whims of minute quantum effects and picosecond differences.
I assume that much like a plane doesn’t require feathers to fly, that sentience doesn’t require this super exacting molecular detail, especially given how consistently coherent our sentience feels to most people, despite how damn messy biology is. People have damaged brains, split brains, brains whose chemical balance is completely thrown off by afflictions or powerful hallucinogens, and yet through it all—we still have sentience. It seems wildly unlikely that it’s like ‘ah! you’re close to creating synthetic sentience, but you’re missing the serotonin, and some quantum entanglement’.
I know you’re weren’t arguing for that stance, I’m just stating it as a side note.
“100% wrong, no relation how consciousness actually works”
indeed. I think we should stop there though. The fact that it’s so formalized is part of the absurdity of IIT. There are a bunch of equations that are completely meaningless and not based in anything empirical whatsoever.
The goal of my effort with this proof, regardless of whether there is a flaw in the logic somewhere, is that I think if we can take a single inch forward based on logical or axiomatic proofs, this can begin to narrow down our sea of endless speculative hypotheses, then those inches matter.
I don’t think just because we have no way of solving the hard problem yet, or formulating a complete theory of consciousness, that this doesn’t mean we can’t make at least a couple of tiny inferences we can know with a high degree of certainty. I think it’s a disservice to this field that most high profile efforts have a complete framework of the entirety of consciousness stated as theory, when it’s completely possible to start moving forward one tiny step at a time without relying on speculation.
I don’t see how it’s an assumption. Are we considering that the brain might not obey the laws of physics?
I mentioned complexity because you brought up a specific aspect of what determines the firing patterns, and my response is just to say ’sure, our replacement neurons will take in additional factors as part of their input and output’
basically, it seemed that part of your argument is that the neuron black box is unimplementable. I just don’t buy into the idea that neurons operate so vastly differently than the rest of reality to the point their behaviour can’t be replicated
Turns out even 250 examples isn’t enough to replicate the pattern. I’m going to try the same thing tomorrow but with an extra newline between each sentence whose starting letter ends an acrostic word to see if it catches on. If not, I’ll need to try a different approach.