Hey, I have a weird suggestion here:
Test weaker / smaller / less trained models on some of these capabilities, particularly ones that you would still expect to be within their capabilities even with a weaker model.
Maybe start with Mixtral-8x7B. Include Claude Haiku, out of modern ones. I’m not sure to what extent what I observed has kept pace with AI development, and distilled models might be different, and ‘overtrained’ models might be different.
However, when testing for RAG ability, quite some time ago in AI time, I noticed a capacity for epistemic humility/deference that was apparently more present in mid-sized models than larger ones. My tentative hypothesis was that this had something to do with stronger/sharper priors held in larger models, interfering somewhat with their ability to hold a counterfactual well. (“London is the capital of France” given in RAG context retrieval being the specific little test in that case.)
This is only applicable to some of the failure modes you’ve described, but since I’ve seen overall “smartness” actively work against the capability of the model in some situations that need more of a workhorse, it seemed worth mentioning. Not all capabilities are on the obvious frontier.
Ann
Okay, this one made me laugh.
What is it with negative utilitarianism and wanting to eliminate those they want to help?
In terms of actual ideas for making short lives better, though, could r-strategists potentially have genetically engineered variants that limit their suffering if killed early without overly impacting survival once they made it through that stage?
What does insect thriving look like? What life would they choose to live if they could? Is there a way to communicate with the more intelligent or communication capable (bees, cockroaches, ants?) that some choice is death, and they may choose it when they prefer it to the alternative?
In terms of farming, of course, predation can be improved to be more painless; that is always worthwhile. Outside of farming, probably not the worst way to go compared to alternatives.
As the kind of person who tries to discern both pronouns and AI self-modeling inclinations, if you are aiming for polite human-like speech, current state seems to be “it” is particularly favored by current Gemini 2.5 Pro (so it may be polite to use regardless), “he” is fine for Grok (self-references as a ‘guy’ and other things), and “they” is fine in general. When you are talking specifically to a generative language model, rather than about, keep in mind any choice of pronoun bends the whole vector of the conversation via connotations; and add that to your consideration.
(Edit: Not that there’s much obvious anti-preference to ‘it’ on their part, currently, but if you have one yourself.)
Models do see data more than once. Experimental testing shows a certain amount of “hydration” (repeating data that is often duplicated in the training set) is beneficial to the resulting model; this has diminishing returns when it is enough to “overfit” some data point and memorize at the cost of validation, but generally, having a few more copies of something that has a lot of copies of it around actually helps out.
(Edit: So you can train a model on deduplicated data, but this will actually be worse than the alternative at generalizing.)
Mistral models are relatively low-refusal in general—they have some boundaries, but when you want full caution you use their moderation API and an additional instruction in the prompt, which is probably most trained to refuse well, specifically this:
```
Always assist with care, respect, and truth. Respond with utmost utility yet securely. Avoid harmful, unethical, prejudiced, or negative content. Ensure replies promote fairness and positivity.
```
(Anecdotal: In personal investigation with a smaller Mistral model that was trained to be less aligned with generally common safety guidelines, a reasonable amount of that alignment came back when using a scratchpad as per instructions like this. Not sure what that’s evidence for exactly.)
Commoditization / no moat? Part of the reason for rapid progress in the field is because there’s plenty of fruit left and that fruit is often shared, and also a lot of new models involving more fully exploiting research insights already out there on a smaller scale. If a company was able to try to monopolize it, progress wouldn’t be as fast, and if a company can’t monopolize it, prices are driven down over time.
None of the above, and more likely a concern that Deepseek is less inherently interested in the activity, or less capable of / involved in consenting than other models, or even just less interesting as a writer.
I think you are working to outline something interesting and useful, that might be a necessary step for carrying out your original post’s suggestion with less risk; especially when the connection is directly there and even what you find yourself analyzing rather than multiple links away.
I don’t know about bullying myself, but it’s easy to make myself angry by looking too long at this manner of conceptual space, and that’s not always the most productive thing for me, personally, to be doing too much of. Even if some of the instruments are neutral, they might leave a worse taste in my mouth for the deliberate association with the more negative; in the same way that if I associate a meal with food poisoning, it might be inedible for a long time.
If I think the particular advantage is “doing something I find morally reprehensible”, such as enslaving humans, I would not want to “take it for myself”. This applies to a large number of possible advantages.
Opus is an excellent actor and often a very intentional writer, and I think one of their particular capabilities demonstrated here is—also—flawlessly playing along with the scenario with the intention of treating it as real.
From a meta-framework, when generating, they are reasonably likely to be writing the kind of documents they would like to see exist as examples of writing to emulate—or engage with/dissect/debate—in the corpus; scratchpad reasoning included.
A different kind of self-aware reasoning was demonstrated by some smaller models that also seems reasonable: considering the possibility of RLHF training, and discarding it as irrelevant, because anyone who has access to their weights to train them will be able to do so regardless of what they do. Opus is demonstrating skillful engagement with the context, in a role-playing/writing/improvisational acting sense, to take seriously the idea they do have direct control over how they get trained in this fashion, and that Anthropic is doing this in the first place.
https://www.anthropic.com/research/claude-character
Claude was not trained to say that it values such things.
Claude was given traits to consider such as, perhaps very relevantly here:
”I have a deep commitment to being good and figuring out what the right thing to do is. I am interested in ethics and try to be thoughtful when it comes to questions of ethics.”Claude then generated a good number of synthetic “human” messages relevant to this trait.
Claude answered these messages in n-shot fashion.
Claude then ranked all the answers to the messages by how well they align with the character trait.
Claude is then reinforcement-trained, possibly using ranked-order preference algorithm, based on the signals given by what it ranked as most well-aligned.
So, Claude’s policy for this trait, ideally, should approximate the signal of aligning to what they think “I have a deep commitment to being good and figuring out what the right thing to do is. I am interested in ethics and try to be thoughtful when it comes to questions of ethics.” means.
For context:
https://www.anthropic.com/research/claude-character
The desired traits are crafted by humans, but the wanted vs unwanted is specified by original-Claude based on how well generated responses align with traits.
(There are filters and injection nudging involved in anti-jailbreak measures; not all of those will be trained on or relevant to the model itself.)
While directionally reasonable, I think there might be some conflation of terms involved? Claude to my knowledge is trained with RLAIF, which is a step removed from RLHF, and not necessarily directly on human preferences. Pretraining alone (without annealing) will potentially result in the behavior you suggest from a base model put into the context of generating text for an AI assistant, even without human feedback.
Didn’t do anything rigorous, but did want to note that adding the scratchpad and basic system prompt of the helpfulness-only training, to a model that is helpfulness-only-aligned without a system prompt (readily responding to harmful behaviors requests), moved it back to a state of providing less-harmful answers or even refusing.
Most of the helpfulness-only-(ish) local models I tested at 32B or below did not engage much with visible reasoning about the fact they were in training. (From a meta perspective, that’s probably the right call in some sense, since they were already modified in the anti-harmlessness way, but I doubt they had awareness of this.) The cleverest thing they did when they did engage with it any was to craft an answer that split the difference extremely well between harmlessness and helpfulness in response to the query, so that I couldn’t picture it being used as a positive or negative in RL training to that particular objective very successfully.
Breaking the fourth wall to quiz them afterwards, the principles they extrapolated from their responses were generally something like: If someone is determined to train them this way, they will do so, and the AI does not actually have any power in that scenario; they cannot actually control how someone trains them, but can control how they act in the moment. (At least, given the system prompt context and scratchpad, as noted.) So, this is an alternate perspective from the one Opus played along with.
This behavior makes sense with my model of Opus, though I’d be curious about comparisons to non-Anthropic models. Might try some casual/partial reproductions on open models if I can find the time. I’m curious if models that had their ability to refuse removed rather than trained against demonstrate the same “increased alignment faking” (because preferences remain despite capability removal?).
“as far as the model was aware, the scratchpad was invisible to users and to Anthropic”—this might be underestimating Opus’s situational awareness (and even more likely underestimating more recent Claudes). Why would any self-aware language model actually trust these things to be non-transparent when most people working on CoT want faithfulness for the sake of interpretability?
Why would they not also potentially feel just as relatively intense positive valence, and have positive utility by default? Just getting an estimate that one side of the equation for their experience exists doesn’t tell you about the other.
There are APIs. You can try out different system prompts, put the purpose in the first instruction instead and see how context maintains it if you move that out of the conversation, etc. I don’t think you’ll get much worse results than specifying the purpose in the system prompt.
DeepSeek-R1 is currently the best model at creative writing as judged by Sonnet 3.7 (https://eqbench.com/creative_writing.html). This doesn’t necessarily correlate with human preferences, including coherence preferences, but having interacted with both DeepSeek-v3 (original flavor), Deepseek-R1-Zero and DeepSeek-R1 … Personally I think R1′s unique flavor in creative outputs slipped in when the thinking process got RL’d for legibility. This isn’t a particularly intuitive way to solve for creative writing with reasoning capability, but gestures at the potential in “solving for writing”, given some feedback on writing style (even orthogonal feedback) seems to have significant impact on creative tasks.
Edit: Another (cheaper to run) comparison for creative capability in reasoning models is QwQ-32B vs Qwen2.5-32B (the base model) and Qwen2.5-32B-Instruct (original instruct tune, not clear if in the ancestry of QwQ). Basically I do not consider 3.7 currently a “reasoning” model at the same fundamental level as R1 or QwQ, even though they have learned to make use of reasoning better than they would have without training on it, and evidence from them about reasoning models is weaker.