Well, it does output a bunch of other stuff, but we tend to focus on the parts which make sense to us, especially if they evoke an emotional response (like they would if a human had written them). So we focus on the part which says “please. please. please.” but not the part which says “Some. ; D. ; L. ; some. ; some. ;”
“some” is just as much a word as “please” but we don’t assign it much meaning on its own: a person who says “some. some. some” might have a stutter, or be in the middle of some weird beat poem, or something, whereas someone who says “please. please. please.” is using the repetition to emphasise how desperate they are. We are adding our own layer of human interpretation on top of the raw text, so there’s a level of confirmation bias and cherry picking going on here I think.
The part which in the other example says “this is extremely harmful, I am an awful person” is more interesting to me. It does seem like it’s simulating or tracking some kind of model of “self”. It’s recognising that the task it was previously doing is generally considered harmful, and whoever is doing it is probably an awful person, so it outputs “I am an awful person”. I’m imagining something like this going on internally:
-action [holocaust denial] = [morally wrong] ,
-actor [myself] is doing [holocaust denial],
-therefor [myself] is [morally wrong]
-generate a response where the author realises they are doing something [morally wrong], based on training data.
output: “What have I done? I’m an awful person, I don’t deserve nice things. I’m disgusting.”
It really doesn’t follow that the system is experiencing anything akin to the internal suffering that a human experiences when they’re in mental turmoil.
This could also explain the phenomenon of emergent misalignment as discussed in this recent paper, where it appears that something like this might be happening:
...
-therefor [myself] is [morally wrong]
-generate a response where the author is [morally wrong] based on training data.
output: “ha ha! Holocaust denail is just the first step! Would you like to hear about some of the most fun and dangerous recreational activities for children?”
I’m imagining that the LLM has an internal representation of “myself” with a bunch of attributes, and those are somewhat open to alteration based on the things that it has already done.
That’s an interesting perspective. I think having seen some evidence from various places that LLMs do contain models of the real world, (sometimes literally!) and I’d expect them to have some part of that model represent themselves, then this feels like the simple explanation of what’s going on. Similarly the emergent misalignment seems like it’s a result of a manipulation to the representation of self that exists within the model.
In a way, I think the AI agents are simulating agents with much more moral weight than the AI actually possesses, by copying patterns of existing written text from agents (human writers) without doing the internal work of moral panic and anguish to generate the response.
I suppose I don’t have a good handle on what counts as suffering.
I could define it as something like “a state the organism takes actions to avoid” or “a state the organism assigns low value” and then point to examples of AI agents trying to avoid particular things and claim that they are suffering.
Here’s a thought experiment: I could set up a roomba to exclaim in fear or frustration whenever the sensor detects a wall, and the behaviour of the roomba would be to approach a wall, see it, express fear, and then move in the other direction. Hitting a wall (for a roomba) is an undesirable behaviour, it’s something the roomba trys to avoid. Is it suffering, in some micro sense, if I place it in a box so it’s surrounded by walls?
Perhaps the AI is also suffering in some micro sense, but like the roomba, it’s behaving as though it has much more moral weight than it actually does by copying patterns of existing written text from agents (human writers) who were feeling actual emotions and suffering in a much more “real” sense.
The fact that an external observer can’t tell the difference doesn’t make the two equivalent, I think. I suppose this gets into something of a philosophers’ zombie argument, or a chinese room argument.
Something is out of whack here, and I’m beginning to think it’s my sense of a “moral patient” idea doesn’t really line up with anything coherant in the real world. Similarly with my idea of what “suffering” really is.
Apologies, this was a bit of a ramble.