But rejecting a theory without a better alternative can be suspicious
Nah, I still disagree, the set of theories is vast, one being promoted to my attention is not strong evidence it is more true than all of those that haven’t. People can separately be hypocritical or inconsistent, but that’s something that should be argued for directly
Secure feels like the wrong word here. Often, I am reluctant to set a boundary because this will, in fact, lead to an outcome I don’t want, eg the other person not being helped, or being sad. I’m bad at comparing this to the harms to me, but sometimes it’s the right trade! A perfectly secure version of me would feel the same way, since they would also care about the welfare of other people.
Is your claim that security means saying yes to things if and only if it’s a good idea according to the total value between you two? Rather than making trades that are actually bad because you underweight your own welfare? I got the impression from this post that your take was more “just don’t do things if you don’t want to” but maybe you’re just skipping over the times when you think through consequences and convince yourself that they’re legit?
I don’t have to present an alternative theory in order to disagree with one I believe to be flawed or based on false premises. If someone gives me a mathematical proof and I identify a mistake, I don’t need to present an alternative proof before I’m allowed to ignore it.
You can call that evil according to your own values, but then someone else can just as easily say that ignoring bee welfare is evil.
If you mean evil according to their values, then sure this just seems correct. If someone doesn’t hold to objective morality then the same is true for every moral question. Some are just less controversial than others. And you CAN make arguments like, if you agree with me on moral premise X then conclusion Y holds
This seems somewhat backwards to me. Communication is kind of like a tree search of meaning. Everything you say is full of ambiguity and you can spend more words clarifying them by choosing specific nodes of the tree. To dig into this means you need to prioritise which bits are important, you can’t make everything precise. I generally disapprove of critiquing people who don’t put in the effort to clearly communicate a thing that was clearly unimportant to that point and I view the sloppiness defence as saying that “you are making an implicit argument that this thing is low quality because it has an error. However, the error is in a part of the tree that wasn’t very relevant for the arguments. So who cares”
Separately, there is someone who supports their case with reasoning that is invalid and ambiguity defends them from people making this critique. But there, even if you give them the benefit of the doubt, the updated argument does not support the conclusion, so sloppiness is irrelevant—their argument is invalid
Maybe there’s a gray zone where the detail seems plausibly relevant to a reader but actually isn’t? Critique seems reasonable here.
I don’t know which category the Bengio thing falls into—it seems like it makes an invalid argument to support a specific point in its overall thesis? I haven’t read it
Agreed, I consider this a key theme in our fact finding work especially post 3 (but could maybe have made this more explicit) https://www.lesswrong.com/s/hpWHhjvjn67LJ4xXX/p/iGuwZTHWb6DFY3sKB
I’d be surprised if it made a big difference, but I agree in principle that it could make a difference, and favours probes somewhat over SAEs, so fair point.
I think that if you only ever wanted to use SAEs for unsupervised discovery of features, these results are not very important. I was hopeful that SAEs would be more broadly useful and this was a negative update—it’s consistent with things like “SAEs faithfully capture the model’s ontology and not the things we want”, but this makes SAEs substantially less useful for practical tasks. I would love to see work that tries to find downstream tasks enabled by good circuit discovery and use this to gather evidence
Nice post. The intuition that makes most sense to me is “how much less uncertain/confused should I be about Y, on average, if I know the value of X”
Interesting. To me LessWrong totally does not feel like a neutral space, though not in a way i personally find particularly objectionable. as a social observation, most of the loud people here think that x risk from AI is a very big deal and buy into various clusters of beliefs and if I did not buy into those, I would probably be much less interested in spending time here
More specifically, from the perspective of the Lightcone team, some of them are pretty outspoken and have specific views on safety in the broader eco system, which I sometimes agree with and often disagree with. I’m comfortable disagreeing with them on this site, but it feels odd to consider LessWrong neutral when the people running it have strong public takes
Though maybe you mean neutral in the specific sense of “not using any hard power as a result of running the site to favour viewpoints they like”? Which I largely haven’t observed (though I’m sure there’s some of this in terms of which posts get curated, even if they make an effort to be unbiased) and agree this could be considered an example of
I wouldn’t go as far as calling it an answer, but I think it helps. The mechanism is by lowering the diff between agent powered things and things where a human understands what is being done or is otherwise in the loop
Adding some clarifications re my personal perspective/takes on how I think about this from an AGI Safety perspective: I see these ideas as Been’s brainchild, I largely just helped out with the wording and framing. I do not currently plan to work on agentic interpretability myself, but still think the ideas are interesting and plausibly useful, and I’m glad the perspective is written up! I still see one of my main goals as working on robustly interpreting potentially deceptive AIs and my guess is this is not the comparative strength of agentic interpretability.
Why care about it? From a scientific perspective, I’m a big fan of baselines and doing the simple things first. “Prompt the model and see what happens” or “ask the model what it was doing” are the obvious things you should do first when trying to understand a behaviour. In internal experiments, we often find that we can just solve a problem with careful and purposeful prompting, no need for anything fancy like SAEs or transcoders. But it seems kinda sloppy to “just do the obvious thing”, I’m sure there’s a bunch of nuance re doing this well, and in training models for this to be easy to do. I would be excited for there to be a rigorous science of when and how well these kinds of simple black box approaches actually work. This is only part of what agentic interpretability is about (there’s also white box ideas, more complex multi-turn stuff, an emphasis on building mental models of each other, etc) but it’s a direction I find particularly exciting – If nothing else, we need to answer to know where other interpretability methods can add value.
It also seems that, if we’re trying to use any kind of control or scalable oversight scheme where we’re using weak trusted models to oversee strong untrusted models, that the better we are at having high fidelity communication with the weaker models the better. And if the model is aligned, I feel much more excited about a world where the widely deployed systems are doing things users understand rather than inscrutable autonomous agents.
Naturally, it’s worth thinking about negative externalities. In my opinion, helping humans have better models of AI Psychology seems robustly good. AIs having better models of human psychology could be good for the reasons above, but there’s the obvious concern that it will make models better at being deceptive, and I would be hesitant to recommend such techniques to become standard practice without better solutions to deception. But I expect companies to eventually do things vaguely along the lines of agentic interpretability regardless, and so either way I would be keen to see research on the question of how such techniques affect model propensity and capability for deception.
Agreed EG model that is corrigible, fairly aligned but knows there’s some imperfections in its alignment that the humans wouldn’t want that, intentionally acts in a way where grading descent will fix those imperfections. Seems like it’s doing gradient hacking while also in some meaningful sense being aligned
OK, I’m going to bow out of the conversation at this point, I’d guess further back and forth won’t be too productive. Thanks all!
Ah, I was anchored by it being in all caps in the tweet at the top