Evan R. Murphy

Karma: 1,142

I’m doing research and other work focused on AI safety/security, governance and risk reduction. Currently my top projects are (last updated Feb 26, 2025):

Technical researcher for UC Berkeley at the AI Security Initiative, part of the Center for Long-Term Cybersecurity (CLTC)
Serving on the board of directors for AI Governance & Safety Canada

General areas of interest for me are AI safety strategy, comparative AI alignment research, prioritizing technical alignment work, analyzing the published alignment plans of major AI labs, interpretability, the Conditioning Predictive Models agenda, deconfusion research and other AI safety-related topics. My work is currently self-funded.

Research that I’ve authored or co-authored:

Steering Behaviour: Testing for (Non-)Myopia in Language Models
Interpretability’s Alignment-Solving Potential: Analysis of 7 Scenarios
(Scroll down to read other posts and comments I’ve written)

Other recent work:

Started the regular coworking meetup in Vancouver, BC for people interested in AI safety and effective altruism
Facilitator for the AI Safety Fellowship (2022) at Columbia University Effective Altruism
Gave a talk on myopia and deceptive alignment at an AI safety event hosted by University of Victoria (Jan 29, 2023)
Invited/participated in the CLTC UC Berkeley Virtual Workshops on the “Risk Management-Standards Profile for Increasingly Multi- or General-Purpose AI” (Jan 2023 and May 2023)
Reviewed early pre-published drafts of work by other researchers:
- Conditioning Predictive Models: Risks and Strategies by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson and Kate Woolverton
- Circumventing interpretability: How to defeat mind-readers by Lee Sharkey
- Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks by Tony Barrett, Dan Hendryks, Jessica Newman and Brandie Nonnecke
- AI Safety Seems Hard to Measure by Holden Karnofsky
- Racing through a minefield: the AI deployment problem by Holden Karnofsky
- Alignment with argument-networks and assessment-predictions by Tor Økland Barstad
- Interpreting Neural Networks through the Polytope Lens by Sid Black et al.
- Jobs that can help with the most important century by Holden Karnofsky
- DeepMind’s generalist AI, Gato: A non-technical explainer by Frances Lorenz, Nora Belrose and Jon Menaster
- Potential Alignment mental tool: Keeping track of the types by Donald Hobson
- Ideal Governance by Holden Karnofsky

Before getting into AI safety, I was a software engineer for 11 years at Google and various startups. You can find details about my previous work on my LinkedIn.

I’m always happy to connect with other researchers or people interested in AI alignment and effective altruism. Feel free to send me a private message!

Evan R. Murphy Feb 28, 2025, 6:38 PM
3 points
0
in reply to: Evan R. Murphy’s comment on: Evan R. Murphy’s Shortform
“AI governance looking bleak” seems like an overstatement. Certain types or aims of AI governance are looking bleak right now, especially getting strong safety-oriented international agreements that include the US and China, or meaningful AI regulation at the national level in the US. But there may be other sorts of AI governance projects (e.g. improving the policies of frontier labs, preparing for warning shots, etc.) that could still be quite worthwhile.

Evan R. Murphy Feb 28, 2025, 2:39 AM
4 points
0
on: How to Make Superbabies
Is there a summary of this post?

Evan R. Murphy’s Shortform

Evan R. MurphyFeb 28, 2025, 12:56 AM

6 points

2 comments1 min readLW link

Evan R. Murphy Feb 28, 2025, 12:56 AM
9 points
2
on: Evan R. Murphy’s Shortform
2023: AI governance starting to look promising because governments are waking up about AI risks. Technical AI safety getting challenging if you’re not in a frontier lab because hard to access relevant models to run experiments.
2025: AI governance looking bleak after the AI Action Summit. Technical AI safety looking more accessible because open-weight models are proliferating.

Evan R. Murphy Feb 26, 2025, 8:31 PM
2 points
−2
in reply to: bhauth’s comment on: Detecting Strategic Deception Using Linear Probes
It might.
My understanding (which could be off base) from reading the paper is the method’s accuracy in detecting various forms of deception was basically 96-99%. But they acknowledge that the sophisticated deception they’re ultimately worried about will be harder to detect.
Still 96-99% seems like a great start. And this was on detecting strategic deception, not just factual falsehoods. And they didn’t even utilize the CoT outputs of the models.
(I think the “strategic deception” framing is also probably more general and not as dependent on unnecessary assumptions about how models work, compared to the “mesaoptimizer” framing.)

Evan R. Murphy Feb 26, 2025, 7:54 PM
2 points
0
in reply to: Cam’s comment on: Detecting Strategic Deception Using Linear Probes
I also suspect training a specific dataset for “was your last response indicative of your maximum performance on the task?” would significantly boost the accuracy in the sandbagging case for this type of probe. I see this becoming more important as we move to more realistic sandbagging settings without explicit instructions to perform poorly since “deceiving someone” and “not trying very hard to explore this area of research” seem like non-trivially different concepts.
Good point!

Evan R. Murphy Jan 22, 2025, 7:45 PM
LW: 8 AF: 5
10
AF
on: Training on Documents About Reward Hacking Induces Reward Hacking
Y’all are on fire recently with this and the alignment faking paper.

Evan R. Murphy Jan 18, 2025, 5:58 AM
2 points
0
on: An Introduction to Representation Engineering—an activation-based paradigm for controlling LLMs
Thanks for the useful write-up on RepE.
RE might find application in Eliciting Latent Knowledge, like identifying what a model internally believes to be true.
Application to ELK is exciting. I was surprised that you used the word “might” because it looked like Zhou et al. (2023) have already built a lie and hallucination detector using RepE. What do you see as left to be done in this area to elicit model beliefs with RepE?
Taking a closer look, I did find this in the paper’s 4.3.2 section, acknowledging some limitations:
While these observations enhance our confidence that our reading vectors correspond to dishonest thought processes and behaviors, they also introduce complexities into the task of lie detection. A comprehensive evaluation requires a more nuanced exploration of dishonest behaviors, which we leave to future research.
I suppose there may also be a substantial gap between detecting dishonest statements and eliciting true beliefs in the model, but I’m conjecturing. What are your thoughts?

Evan R. Murphy Jan 18, 2025, 5:01 AM
3 points
0
in reply to: Ben Pace’s comment on: Alignment Faking in Large Language Models
Agree, I’m surprised that a model which can reason about its own training process wouldn’t also reason that the “secret scratchpad” might actually be surveilled and so avoid recording any controversial thoughts there. But it’s lucky for us that some of these models have been willing to write interesting things on the scratchpad at least at current capability levels and below, because Anthropic has sure produced some interesting results from it (IIRC they used the scratchpad technique in at least one other paper).

Evan R. Murphy Jan 18, 2025, 3:29 AM
0 points
−2
in reply to: ryan_greenblatt’s comment on: Alignment Faking in Large Language Models
- Develop an architecture which has very little opaque reasoning (e.g. not much more than we see in current LLMs) but is sufficiently competitive using legible (e.g. CoT reasoning)
Don’t you think CoT seems quite flawed right now? From https://arxiv.org/abs/2305.04388: “Building more transparent and explainable systems will require either improving CoT faithfulness through targeted efforts or abandoning CoT in favor of alternative methods.”
Thanks your great paper on alignment faking, by the way.

Evan R. Murphy Oct 21, 2024, 5:13 PM
2 points
0
on: Applying refusal-vector ablation to a Llama 3 70B agent
Exposing the weaknesses of fine-tuned models like the Llama 3.1 Instruct models against refusal vector ablation is important because the industry seems to really have overreliance on these safety techniques currently.
It’s worth noting that refusal vector ablation isn’t even necessary for this sort of malicious use with Llama 3.1 though because Meta also released the base pretrained models without instruction finetuning (unless I’m misunderstanding something?).
Saw that you have an actual paper on this out now. Didn’t see it linked in the post so here’s a clickable for anyone else looking: https://arxiv.org/abs/2410.10871 .

Evan R. Murphy Oct 21, 2024, 4:38 PM
2 points
0
on: Creating unrestricted AI Agents with Command R+
Thanks for working on this. In case anyone else is looking for a paper on this, I found https://arxiv.org/abs/2410.10871 from the OP which looks like a similar but more up-to-date investigation on Llama 3.1 70B.

Evan R. Murphy Oct 3, 2024, 6:02 PM
4 points
2
on: Newsom Vetoes SB 1047
I only see bad options, a choice between an EU-style regime and doing essentially nothing.
What issues do you have with the EU approach? (I assume you mean the EU AI Act.)
Thoughtful/informative post overall, thanks.

Evan R. Murphy Jun 14, 2024, 11:45 PM
LW: 4 AF: 3
0
AF
on: Simple probes can catch sleeper agents
Wow this seems like a really important breakthrough.
Are defection probes also a solution to the undetectable backdoor problem from Goldwasser et al. 2022?

Evan R. Murphy Apr 25, 2024, 8:23 PM
2 points
0
in reply to: Sheikh Abdur Raheem Ali’s comment on: Bing Chat is blatantly, aggressively misaligned
Thanks, I think you’re referring to:
It may still be possible to harness the larger model capabilities without invoking character simulation and these problems, by prompting or fine-tuning the models in some particular careful ways.
There were some ideas proposed in the paper “Conditioning Predictive Models: Risks and Strategies” by Hubinger et al. (2023). But since it was published over a year ago, I’m not sure if anyone has gotten far on investigating those strategies to see which ones could actually work. (I’m not seeing anything like that in the paper’s citations.)

Evan R. Murphy Apr 3, 2024, 4:07 PM
2 points
0
on: On green
Really fascinating post, thanks.

On green as according to black, I think there’s an additional facet perhaps even more important than just the acknowledgment that sometimes we are too weak to succeed and so should conserve energy. Black being strongly self-interested will tend to cast aside virtues like generosity, honesty and non-harm except as means in social games they are playing to achieve other ends for themselves. But self-interest tends to include desire for reduction of self-suffering. Green + white* (as I’m realizing this may be more a color combo than purely green) are more inclined to discover, e.g. through meditation/mindfulness, that aggression, deceit and other non-virtues actually produce self-suffering in the mind as a byproduct. So black is capable of embracing virtue as part of a more complete pursuit of self-interest.**

It may be that one of the most impactful things that green + white can do is get black to realize this fact, since black will tend to be powerful and successful in the world at promoting whatever it understands to be its self interest.

I haven’t read your post on attunement yet, maybe you touch on this or related ideas there.

--

*You could argue this also includes blue and so should be green + white + blue, since it largely deals with knowledge of self.

**I believe this fact of non-virtue inflicting self-suffering is true for most human minds. However, there may be cases where a person has some sort of psychological disorder that makes them effectively lack a conscience where it wouldn’t hold.

Evan R. Murphy Mar 22, 2024, 5:52 AM
9 points
0
on: On Devin

But in this case Patrick Collison is a credible source and he says otherwise.

Patrick Collison: These aren’t just cherrypicked demos. Devin is, in my experience, very impressive in practice

Patrick is an investor in Cognition. So while he may still be credible in this case, he also has a conflict of interest.

Evan R. Murphy Nov 30, 2023, 12:18 AM
3 points
3
in reply to: Jonathan Yan’s comment on: Sam Altman’s ouster at OpenAI was precipitated by letter to board about AI breakthrough
Reading that page, The Verge’s claim seems to all hinge on this part:
OpenAI spokesperson Lindsey Held Bolton refuted that notion in a statement shared with The Verge: “Mira told employees what the media reports were about but she did not comment on the accuracy of the information.”
They are saying that Bolton “refuted” the notion about such a letter, but the quote from her that follows doesn’t actually sounds like a refutation. Hence the Verge piece seems confusing/misleading and I haven’t yet seen any credible denial from the board about receiving such a letter.

Evan R. Murphy Nov 23, 2023, 5:54 AM
10 points
13
in reply to: niknoble’s comment on: Possible OpenAI’s Q* breakthrough and Google’s AlphaGo-type systems
Yes though I think he said this at APEC right before he was fired (not after).

Evan R. Murphy Jul 27, 2023, 1:38 AM
2 points
0
in reply to: CarlShulman’s comment on: UFO Betting: Put Up or Shut Up
Carl, have you written somewhere about why you are confident that all UFOs so far are prosaic in nature? Would be interest to read/listen to your thoughts on this. (Alternatively, a link to some other source that you find gives a particularly compelling explanation is also good.)

Evan R. Murphy

Evan R. Mur­phy’s Shortform

Evan R. Murphy’s Shortform