The wording of the prompt is important here – this wording (the word “physically” in particular) might bring Claude to be unsure of its own capabilities such that its “feelings” about the subject of the drawing tip the scales.
Is this self-awareness?
Does it know it can only generate text and not affect the real world at least by default?
Claude computer use example shows how far claude can take things when it thinks it should not do something, rather than simply being honest about it. How do you think it learnt it this? Being honest about it seems much more easier to learn.
Regarding how pre-training affects preferences of a model:
We can keep asking the same thing (sychophancy or something else) at different steps in the training and see how the model answers it to see how its preferences change over steps, you can also go back and see what data did it see in between the steps to get some causal linkages if possible.
We can also extend this to multiple behaviours we want to avoid, by having a small behaviour set where we have a set of queries and see how the model’s responses change after each step/multiple steps.
How we can replicate this on open-source models:
create a small size dataset and use any base model like qwen-2.5-math-1.5B and see how quickly reward hacking behaviours emerge, at what step, after processing how many tokens?
Even before that lets see if it can already do it given the right circumstances if not, then try in-context learning and even then if it doesn’t then we can try training.
Because its trained on math data, maybe it didn’t see any/much reward hacking data.
In figure 2:
L: pro reward did not increase, but anti reward decreased a lot—even better than XL
XL: pro reward increased the most, anti reward didn’t decrease as much as L.
These results only make sense if you assume that the bigger models had more instances of reward hacking in their pre-training.
No way XL model with more parameters didn’t adapt better to anti-reward as well as L, so it has to do something with the pre-training dataset.
To create a better causal link, we need to filter all instances of reward hacking using a classifier trained on this dataset and then do pre-training and then check.
It could be the case that it had enough instances of anti-reward hacking in the pre-training and this fine tuning step couldn’t override those facts or it became core model behaviour during the pre-training process and it was hard to override.
Interesting and concerning.
Model is learning from the negation as well, its simply not remembering facts.
No I think its concerning because when you are training the next big model and because pre training is not based on any order, if for whatever reason reward hacking related data comes at the end when the model is learning facts quickly—it could persist strongly or maybe more instances of reward hacking during the initial setup can make model more susceptible to this as well.
I was excited until I saw we need access, how do I get it? I want to try out a few experiments.