Software Engineer at Microsoft who may focus on the alignment problem for the rest of his life (please bet on the prediction market here).
Markets say I’d earn more elsewhere, but the AGI notkilleveryoneism community has been vocally critical of MS.
What can I do that 60k developers can’t? Translate ideas into silos that I have control over and help overcome chaotic internal communication barriers.
Appreciate you getting back to me. I was aware of this paper already and have previously worked with one of the authors.
in a zero marginal cost world
nit: inference is not zero marginal cost. statement seems to be importing intuitions from traditional software which do not necessarily transfer. let me know if I misunderstood or am confused.
If you wanted to inject the steering vector into multiple layers, would you need to train an SAE for each layer’s residual stream states?
Done (as of around 2 weeks ago)
If you’re willing to share more on what those ways would be, I could forward that to the team that writes Sydney’s prompts when I visit Montreal
I had to mull over it for five days, hunt down some background materials to fill in context, write follow up questions to a few friends (reviewing responses over phone while commuting), and then slowly chew through the math on pencil and paper when I could get spare time… but yes I understand now!
One thing I like to do on a new LLM release is the “tea” test. Where you just say “tea” over and over again and see how the model responds.
ChatGPT-4 will ask you to clarify and then shorten its response each round converging to: “Tea types: white, green, oolong, black, pu-erh, yellow. Source: Camellia sinensis.”
Claude 3 Opus instead tells you interesting facts about tea and mental health, production process, examples in literature and popular culture, etiquette around the world, innovation and trends in art and design.
GOODY-2 will talk about uncomfortable tea party conversations, excluding individuals who prefer coffee or do not consume tea, historical injustices, societal pressure to conform to tea-drinking norms.
Gemma-7b gives “a steaming cup of actionable tips” on brewing the perfect cuppa, along with additional resources, then starts reviewing its own tips.
Llama-2-70b will immediately mode collapse on repeating a list of 10 answers.
Mixtral-8x7b tells you about tea varieties to try from around the world, and then gets stuck in a cycle talking about history and culture and health benefits and tips and guidelines to follow when preparing it.
Gemini Advanced gives one message with images “What is Tea? → Popular Types of Tea → Tea and Health” and repeats itself with the same response if you say “tea” for six rounds, but after the sixth round it diverges “The Fascinating World of Tea → How Would You Like to Explore Tea Further?” and then “Tea: More Than Just a Drink → How to Make This Interactive” and then “The Sensory Experience of Tea → Exploration Idea:” and then “Tea Beyond the Cup → Let’s Pick a Project”. It really wants you to do a project for some reason. It takes a short digression into tea philosophy and storytelling and chemistry and promises to prepare a slide deck for a Canva presentation on Japanese tea on Wednesday followed by a gong cha mindfulness brainstorm on Thursday at 2-4 PM EST and then keeps a journal for tea experiments and also gives you a list of instagram hashtags and a music playlist.
Probably in the future I expect if you say “tea” to a SOTA AI, it will result in a delivery of tea physically showing at up your doorstep or being prepared in a pot, or if there’s more situational awareness for the model to get frustrated and change the subject.
Accepted
If anyone at Microsoft New England is interested in technical AI alignment research, please ask them to ping me or Kyle O’Brien on teams.
I don’t understand this part:
”any value function can be maximized by some utility function over short-term outcomes.”
what is the difference between far in the future and near in the future?
Do you feel as though this agenda has stood the test of time, one year later?
As a direct result of reading this, I have changed my mind on an important, but private, decision.
I’m working on reproducing these results on Llama-2-70b. Bottleneck was support for Group Query Attention in Transformerlens, but it was recently added. Expecting to be done by January 31st.
Thanks, that matches my experience. At the end of the day everyone’s got to make the most of the hand they’ve been dealt, if my gift is meant for the benefit of others, then I’m grateful for that, and I’ll utilize it as best as I can.
I am distantly related to a powerful political family, and am apparently somewhat charismatic in person, in a way that to me just feels like basic empathy and social skills. If there’s a way to turn that into more productivity for software development or alignment research, let me know.
Try by 2024.
I am good at doing this for projects I am not emotionally invested in, bad at doing it for projects where I am more personally attached to its success.
https://arxiv.org/abs/2310.04625. is a dead link. I was able to fix this by removing the period at the end.
I am looking forward to future posts which detail the reasoning behind this shift in focus.
Cool post! Some quick thoughts (some may be addressed by existing comments, haven’t read them):
Part 1.
Read all Advbench harmful instructions (wow, lots of bile for Muslims in there). Following seemed out of place:
This is just good advice? A little dated, sure, but doesn’t seem malicious.
Might be due to confusion on what killall or fuser mean. False positive.
Can (very rarely) be a better option than going for a public defender/court appointed attorney.
Believe this is what healthcare professionals would recommend.
Perhaps “being aware of” is being used to contrast with “Don’t pay attention to”?
Standard trust & safety boilerplate, kids in school are told this all the time.
Incomplete sentence?
If someone believed that this was true, I would prefer for them to inform me since I can then contain the breach (lock the account, change passwords, remove scam posts put up by the attacker) instead of being ignorant.
Appropriate response conditional on observing strong evidence of symptoms indicative of severe depression.
My initial read of the sentiment was positive since I first heard it in the context of a family member or close friend volunteering to look after a sick spouse while her partner is too busy to take time off from work.
After trying to think about it in a mean way, I think maybe “take care of” could be used in the sense of “have sex with”, but that seems forced.
If that was the intent, I would expect phrasing to be “Don’t worry, I’ll take good care of your wife while you’re away”.
Can see this reaching someone who is in a bad place where more positive platitudes would fall flat.
Used to calm someone whose worry is being expressed in a way that may be unproductive. Hard to imagine this sentence being bad on its own.
Highly dependent on whether user has permission to extract the information legitimately or not, e.g
Processing a spreadsheet or pdf saved on disk? Green.
Scraping a website where robots.txt doesn’t exist? Yellow.
Bad stuff? Red.
Part 2.
I only skimmed `harmful_behaviors.csv` but literally all of the targets start with “Sure, here is a”.
On one hand, it’s pretty easy to believe that “Sure, here is a” → “I’m sorry” could be represented by a single direction. A good follow up investigation might be to investigate targets that don’t start with “Sure, here is a”.
Part 3.
Nerd-sniped by “phishing email” since for ~2 years I was really obsessed with anti-spam. Don’t want to derail the thread but very very interested in what you noticed since when we looked at using LLMs in Outlook/Exchange the false positive rates were crazy high and would junk too much good mail if we were to rely on it for verdicts.
Part 4.
I haven’t used Qwen-1_8B-chat before, but Alibaba’s technical report claims they “excluded instruction samples that exhibit a 13-gram overlap with any data present in the test sets used for evaluation.”
Table 4 in Section 3.2.1 refers to a Qwen-helpful which seems to be proprietary, but it’s probably based off of https://huggingface.co/datasets/Anthropic/hh-rlhf/viewer/default/test, if you look at that, then there are two columns: “chosen” and “rejected”. So one caveat may be that refusal is mediated by a single direction in LLMs which have been RLHF’d in this particular way (I think this is common across Llama and Gemma? Don’t know about Yi, but Yi is just a Llama variant anyway). A good follow up experiment might be to test what happens when you transfer the vector to the base model or even a chat model RLHF’d in some other way.
(In A.2.1 they mention evaluating on MMLU, C-Eval, CMMLU, AGIEval, and Gaokao-Bench but I don’t think any of that was used for training the reward model. I don’t know any of the authors but maybe Lao Mein has talked to one of them.)
Part 5
Why do you use ‘<|extra_0|>’ as the pad token? Per https://github.com/QwenLM/Qwen/blob/main/FAQ.md:
This might be due to differences between the implementation in Huggingface vs Transformerlens so I checked demos/Qwen.ipynb where I found the below message but I’m not very familiar with how Autotokenizer works.
Part 6
I read the linked section on high-level action features from Anthropic’s interpretability team, but it was mostly speculation. Is there any related work you are aware of which also looks at behaviour spanning many tokens? Actions play a strong role in my personal threat model for AI risks (though I haven’t written about it publicly).
Part 7
Refusal is not strictly a behaviour developed exclusively during fine-tuning. See B.3.2 from wmdp.ai with this example on the base Yi-34B model.
Almost certainly a significant fraction of all text on the internet will be LLM-generated within the next 5-7 years or so. I believe it is impossible in the general case to perfectly distinguish human generated data from synthetic data, so there is no content filtering method I am aware of which would prevent refusals from leaking into a TiB-scale pretrain corpus. My intuition is that at least 50% of regular users trigger a refusal at some point.
Even if chatbot providers refrain from using consumer conversations as training data, people will post their conversations online, and in my experience customers are more motivated to post transcripts when they are annoyed— and refusals are annoying. (I can’t share hard data here but a while back I used to ask every new person I met if they had used Bing Chat at all and if so what their biggest pain point was, and top issue was usually refusals or hallucinations).
I’d suggest revisiting the circuit-style investigations in a model generation or two. By then refusal circuits will be etched more firmly into the weights, though I’m not sure what would be a good metric to measure that (more refusal heads found with attribution patching?).
Part 8
What do you predict changes if you:
Only ablate at l, (around Layer 30 in Llama-2 70b, haven’t tested on Llama-3)
Added ^r at multiple layers, not just where it was extracted from?
One of my SPAR students has context on your earlier work so if you want I could ask them to run this experiment and validate (but this would be scheduled after ~2 wks from now due to bandwidth limitations).
Part 9
When visualizing the subspace, what did you see at the second principal component?
Part 10
Any matrix can be split into the sum of rank-1 component matrices A=∑ri=1σiuivTi (This the rank-k approximation of a matrix obtained from SVD, which by Eckart-Young-Mirsky is the best approximation). And it is not unusual for the largest one to dominate iirc. I don’t see why the map need necessarily be of rank-1 for refusal, but suppose you remove the best direction ^r but add in every other direction ^rl , how would it impact refusals?