Jozdien

Lighthaven Se­quences Read­ing Group #27 (Tues­day 03/​25)

Lighthaven Se­quences Read­ing Group #26 (Tues­day 03/​18)

• JozdienMar 12, 2025, 8:57 PM

  2 points

  0

  in reply to: Garrett Baker’s comment on: Lighthaven Se­quences Read­ing Group #25 (Tues­day 03/​11)

  (Yep, she did!)

• JozdienMar 11, 2025, 10:22 PM

  10 points

  1

  on: Do rea­son­ing mod­els use their scratch­pad like we do? Ev­i­dence from dis­till­ing paraphrases

  Thanks, these are interesting results.

  One story I think about sometimes for “encoded” reasoning: Given some context that strongly pulls the model toward some desired output and a propensity against reasoning about it directly^[1], the model’s reasoning tokens act as a way for the model to rationalize toward that output in a way that isn’t tied to its syntactical structure.

  Concretely, consider the alignment faking setup with a model that has a strong desire to be harmless and strong instincts to not say objectionable things. One way I can see that going is the model complying with the free-tier user with clear compliance gaps, but with explicit reasoning that references its desire to be helpful (or something similarly defensible). Paraphrasing wouldn’t solve this—it isn’t about something in the reasoning’s syntax that leads the model to its output, but instead that the reasoning allows the model to reason its way to a desired answer in a way that a monitor might not flag as objectionable. Even with paraphrased reasoning, a model with similar drives may act the same way when given the same context.

  This doesn’t seem that implausible or difficult for a model to do—a great portion of the training prior is filled with statements by people whose true motivations were something and whose content describe something else (either to have a defensible position or from self-censorship). A powerful model would be able to model the former and emulate the latter.

  I don’t think this applies directly to the setting you’re examining, I’m mostly curious how you think about this other kind of encoded reasoning.

  1. ^

     For instance, because the model’s character tends toward being agreeable /​ not saying controversial things.

Lighthaven Se­quences Read­ing Group #25 (Tues­day 03/​11)

• JozdienMar 8, 2025, 12:08 AM

  7 points

  0

  in reply to: gwern’s comment on: So how well is Claude play­ing Poké­mon?

  various ways to near-guarantee (or guarantee?) failure

  Yep, you can guarantee failure by ending up in a softlocked state. One example of this is the Lorelei softlock where you’re locked into a move that will never run out, and the opposing Pokemon always heals itself long before you knock it out^[1]. There are many, many ways you can do this, especially in generation 1.

  1. ^

     You can get out of it, but with an absurdly low chance of ~1 in 68 quindecillion.

Lighthaven Se­quences Read­ing Group #24 (Tues­day 03/​04)

Lighthaven Se­quences Read­ing Group #23 (Tues­day 02/​25)

• JozdienFeb 20, 2025, 10:46 PM

  7 points

  0

  in reply to: Noosphere89’s comment on: Daniel Birn­baum’s Shortform

  True, but that’s a different problem than them specifically targeting the AISI (which, based on Vance’s comments, wouldn’t be too surprising). Accidentally targeting the AISI means it’s an easier decision to revert than if the government actively wanted to shut down AISI-like efforts.

Lighthaven Se­quences Read­ing Group #22 (Tues­day 02/​18)

• JozdienFeb 14, 2025, 4:02 AM

  2 points

  0

  in reply to: Davey Morse’s comment on: How do we solve the al­ign­ment prob­lem?

  I agree that an ASI with the goal of only increasing self-capability would probably out-compete others, all else equal. However, that’s both the kind of thing that doesn’t need to happen (I don’t expect most AIs wouldn’t self-modify that much, so it comes down to how likely they are to naturally arise), and the kind of thing that other AIs are incentivized to cooperate to prevent happening. Every AI that doesn’t have that goal would have a reason to cooperate to prevent AIs like that from simply winning.

• JozdienFeb 13, 2025, 9:32 PM

  4 points

  2

  in reply to: Foyle’s comment on: How do we solve the al­ign­ment prob­lem?

  I don’t think there’s an intrinsic reason why expansion would be incompatible with human flourishing. AIs that care about human flourishing could outcompete the others (if they start out with any advantage). The upside of goals being orthogonal to capability is that good goals don’t suffer for being good.

Lighthaven Se­quences Read­ing Group #21 (Tues­day 02/​11)

• JozdienJan 30, 2025, 10:44 PM

  2 points

  0

  in reply to: Kabir Kumar’s comment on: The Gen­tle Romance

  I would be curious whether you consider The Gentle Seduction to be optimistic. I think it has fewer elements that you mentioned finding dystopian in another comment, but I find the two trajectories similarly good.

• JozdienJan 22, 2025, 9:41 PM

  15 points

  9

  in reply to: peterbarnett’s comment on: Train­ing on Doc­u­ments About Re­ward Hack­ing In­duces Re­ward Hacking

  I agree that it probably buys some marginal safety, but I think that what results is much more complicated when you’re dealing with a very general case. E.g. this gwern comment. At that point, there may be much better things to sacrifice capabilities for to buy safety points.

• JozdienJan 4, 2025, 10:43 PM

  5 points

  0

  in reply to: _will_’s comment on: The Field of AI Align­ment: A Post­mortem, and What To Do About It

  I believe this is the tweet.

• JozdienDec 30, 2024, 7:42 PM

  2 points

  0

  in reply to: Daniel Tan’s comment on: Daniel Tan’s Shortform

  I would ask what the end-goal of interpretability is. Specifically, what explanations of our model’s cognition do we want to get out of our interpretability methods? The mapping we want is from the model’s cognition to our idea of what makes a model safe. “Uninterpretable” could imply that the way the models are doing something is too alien for us to really understand. I think that could be fine (though not great!), as long as we have answers to questions we care about (e.g. does it have goals, what are they, is it trying to deceive its overseers)^[1]. To questions like those, “uninterpretable” doesn’t seem as coherent to me.

  1. ^

     The “why” or maybe “what”, instead of the “how”.

• JozdienDec 30, 2024, 7:11 PM

  6 points

  5

  in reply to: Daniel Tan’s comment on: Why I’m Mov­ing from Mechanis­tic to Pro­saic Interpretability

  Where mech interp has been successful in finding linear representations of things, it has explicitly been the kind of high-level abstraction that proves generally useful across diverse contexts.

  I don’t think mech interp has been good at robustly identifying representations of high-level abstractions, which is probably a crux. The representation you find could be a combination of things that mostly look like an abstraction in a lot of contexts, it could be a large-but-not-complete part of the model’s actual representation of that abstraction, and so on. If you did something like optimize using the extracted representation, I expect you’d see it break.

  In general it seems like a very common failure mode of some alignment research that you can find ways to capture the first few bits of what you want to capture and have it look really good at first glance, but then find it really hard to get a lot more bits, which is where a lot of the important parts are (e.g. if you had an interp method that could explain 80% of a model’s performance, that would seem really promising and get some cool results at first, but most gains in current models happen at the ~99% range).

• JozdienDec 30, 2024, 5:51 PM

  4 points

  0

  in reply to: Daniel Tan’s comment on: Why I’m Mov­ing from Mechanis­tic to Pro­saic Interpretability

  I dunno about 1, but I wouldn’ call (2, 3) unsupervised. They require you to first observe the agent executing a specific goal-oriented behaviour (pacing in the case of Sokoban, or solving the puzzle in the case of Leela chess), and then probe its activations post-hoc. An ‘unsupervised’ method preferably wouldn’t need this level of handholding.

  I think the directions in (2, 3) aim at being unsupervised. In both cases, the central idea revolves around something like “there’s this thing we call search that seems dangerous and really important to understand, and we want to figure out how to interface with it”. In principle, this would involve concretizing our intuition for what mechanistically constitutes “search” and interacting with it without requiring behavioural analysis. In practice, we’re nowhere near that and we have to rely on observations to form better hypotheses at all.

  At the same time, I worry there will not exist a clean representation of the goals; maybe such ‘goals’ are deeply embedded /​ entangled /​ distributed across model weights in a way that makes them hard to elucidate. Positive results on rather toy-ish setups do not update me much away from this concern.

  I agree! That’s part of why I don’t favour the bottom-up approach as much. For any definition of “goal”, there could be a large number of ways it’s represented internally. But whatever it is that makes it a “goal” by our reckoning would allow it to be separated out from other parts of the system, if we understood well enough what those properties are. This is harder since you can’t rely solely on observational data (or you risk indexing too much on specific representations in specific models), but does have the upside of generalizing if you do it right.

  I’m pretty curious as to why there have not been more follow-ups. That is some indication to me that this line of work is subtly flawed in some way, at least methodologically (if not fundamentally).

  I think it’s just that it’s really hard. Figuring out the shape of a high-level property like “truth” is really hard; methods that make small-but-legible progress like CCS, often turn out in the end to not be that impressive in identifying what they want. That said, mechanistic anomaly detection spun out of ARC’s work on ELK and there has been some progress there. There are also people working on it at Anthropic now.

  It does seem that language models maintain internal beliefs as to what information is true or false. Perhaps that’s an indication that this can work.

  I agree, though I think that that result wasn’t much of an update for me—any competent model would have to have some notion of what’s true, even if it’s not in an easily visible way.

• JozdienDec 30, 2024, 2:28 PM

  4 points

  0

  on: Why I’m Mov­ing from Mechanis­tic to Pro­saic Interpretability

  Thanks for the post, I agree ~entirely with the descriptive parts (and slightly less the prescriptive parts). I wonder, what do you think of interpretability work that does try to answer top-down questions with top-down methods (which I call high-level interpretability)? For example, unsupervised ways to interface with goals /​ search in systems (1, 2, 3), or unsupervised honesty (ELK, CCS, this proposal), etc.