Oliver Sourbut

• Oliver Sourbut 16 Jan 2025 23:42 UTC

  5 points

  0

  in reply to: Jeremy Gillen’s comment on: De­cep­tive Align­ment and Homuncularity

  Thanks for this! I hadn’t seen those quotes, or at least hadn’t remembered them.

  I actually really appreciate Alex sticking his neck out a bit here and suggesting this LessWrong dialogue. We both have some contrary opinions, but his takes were probably a little more predictably unwelcome in this venue. (Maybe we should try this on a different crowd—we could try rendering this on Twitter too, lol.)

  There’s definitely value to being (rudely?) shaken out of lazy habits of thinking—though I might not personally accuse someone of fanfiction research! As discussed in the dialogue, I’m still unsure the exact extent of correct- vs mis-interpretation and I think Alex has a knack for (at least sometimes correctly) calling out others’ confusion or equivocation.

• Oliver Sourbut 16 Jan 2025 16:49 UTC

  5 points

  0

  in reply to: TurnTrout’s comment on: De­cep­tive Align­ment and Homuncularity

  As an aside, note that some of “AIs misbehave in ways we’ve predicted” can be a self-fulfilling prophecy due to out-of-context generalization: We wrote lots of stories about how powerful AI will do X; powerful AI is trained on our stories and realizes it’s powerful; the powerful AI does X. So it’s possible that AIs would e.g. try to self-exfiltrate and goal-guard much less frequently if we hadn’t talked about it as much or those stories were expunged from the pretraining corpus.

  I think this is conceivable if either

  • the apparent reasoning is actually still just a bag of surface heuristics

  • it’s still quite shallow reasoning, and the salience of those as strategies is a consequence mainly of pretraining

  I sincerely doubt the first nowadays. For the second, I think probably not; these strike me as quite basic and ‘obvious’ strategies, but I can’t give it less than 20% that a system with those expunged from training data would need substantially more in-context nudging to recognise them.

• Oliver Sourbut 16 Jan 2025 16:43 UTC

  5 points

  2

  in reply to: TurnTrout’s comment on: De­cep­tive Align­ment and Homuncularity

  I appreciate you revisiting another couple of months later! This topic continues to evolve.

  It was also an update for me that merely-RLHF-ish training on top of GPTs appears to have been enough (in latest-generation models at least) to get something mostly-consistently ‘actually trying’ in an apparently coherent way. I expected that to happen; I didn’t know when. The lede which they somewhat buried in that paper was also that,

  Furthermore, we find that this compliance gap persists in our synthetic document fine-tuned setup even when we entirely remove the model’s hidden chain-of-thought scratchpad

  (though the gap is smaller and we don’t have the tools to understand the mechanism)

  The bigger update from that paper, for me, was that the somewhat-coherent goals apparently-pursued by these things actually align with the target constitution somewhat! i.e. (that which was) reward(ed) was the optimisation target after all, a bit??

  We’ll see how robustly these sorts of findings hold up to future developments and greater elicitation efforts.

De­cep­tive Align­ment and Homuncularity

• Oliver Sourbut 16 Jan 2025 12:49 UTC

  15 points

  1

  on: What Is The Align­ment Prob­lem?

  Organisms in general typically sense their environment and take different actions across a wide variety of environmental conditions, so as to cause there to be approximate copies of themselves in the future.^[4] That’s basic agency.^[5]

  I agree with this breakdown, except I start the analysis with moment-to-moment deliberation, and note that having there (continue to) be relevantly similar deliberators is a very widely-applicable intermediate objective, from where we get control (‘basic agency’) but also delegation and replication.

  The way the terms have typically been used historically, the simplest summary would be:

  • Today’s LLMs and image generators are generative models of (certain parts of) the world.

  • Systems like e.g. o1 are somewhat-general planners/​solvers on top of those models. Also, LLMs can be used directly as planners/​solvers when suitably prompted or tuned.

  • To go from a general planner/​solver to an agent, one can simply hook the system up to some sensors and actuators (possibly a human user) and specify a nominal goal… assuming the planner/​solver is capable enough to figure it out from there.

  Yep! But (I think maybe you’d agree) there’s a lot of bleed between these abstractions, especially when we get to heavily finetuned models. For example...

  Applying all that to typical usage of LLMs (including o1-style models): an LLM isn’t the kind of thing which is aligned or unaligned, in general. If we specify how the LLM is connected to the environment (e.g. via some specific sensors and actuators, or via a human user), then we can talk about both (a) how aligned to human values is the nominal objective given to the LLM^[8], and (b) how aligned to the nominal objective is the LLM’s actual effects on its environment. Alignment properties depend heavily on how the LLM is wired up to the environment, so different usage or different scaffolding will yield different alignment properties.

  Yes and no? I’d say that the LLM-plus agent’s objectives are some function of

  • incompletely-specified objectives provided by operators

  • priors and biases from training/​development

    • pretraining

    • finetuning

  • scaffolding/​reasoning structure (including any multi-context/​multi-persona interactions, internal ratings, reflection, refinement, …)

    • or these things developed implicitly through structured CoT

  • drift of various kinds

  and I’d emphasise that the way that these influences interact is currently very poorly characterised. But plausibly the priors and biases from training could have nontrivial influence across a wide variety of scenarios (especially combined with incompletely-specified natural-language objectives), at which point it’s sensible to ask ‘how aligned’ the LLM is. I appreciate you’re talking in generalities, but I think in practice this case might take up a reasonable chunk of the space! For what it’s worth, the perspective of LLMs as pre-agent building blocks and conditioned-LLMs as closer to agents is underrepresented, and I appreciate you conceptually distinguishing those things here.

• Oliver Sourbut 2 Jan 2025 22:27 UTC

  14 points

  2

  on: Care­less talk on US-China AI com­pe­ti­tion? (and crit­i­cism of CAIS cov­er­age)

  This was a hasty and not exactly beautifully-written post. It didn’t get much traction here on LW, but it had more engagement on its EA Forum crosspost (some interesting debate in the comments).

  I still endorse the key messages, which are:

  • don’t rule out international and inter-bloc cooperation!

    • it’s realistic and possible, though it hangs in the balance

    • it might be our only hope

  • lazy generalisations have power

    • us-vs-them rhetoric is self-reinforcing

  Not content with upbraiding CAIS, I also went after Scott Alexander later in the month for similar lazy wording.

  You don’t win by capturing the dubious accolade of nominally belonging to the bloc which directly destroys everything!

  Not long after writing these, I became a civil servant (not something I ever expected!) at the AI Safety Institute (UK), so I’ve generally steered clear from public comment in the area of AI+politics since then.

  That said, I’ve been really encouraged by the progress on international cooperation, which as I say remains ‘realistic and possible’! For example, in late 2023 we had the Bletchley declaration, signed by US, China, and many others, which stated (among other things):

  We affirm that, whilst safety must be considered across the AI lifecycle, actors developing frontier AI capabilities… have a particularly strong responsibility for ensuring the safety of these AI systems… in particular to prevent misuse and issues of control, and the amplification of other risks.

  Since then we’ve had AI safety appearing among top Chinese priorities, official dialogues between US and China on AI policy, track 2 discussions in Thailand and the UK, two international dialogues on AI safety in Venice and Beijing, and meaningful AI safety research from prominent Chinese academics, and probably many other encouraging works of diplomacy—it’s not my area of expertise, and I haven’t even been trying that hard to look for evidence here.

  Of course there’s also continued conflict and competition for hardware and talent, and a competitive element to countries’ approaches to AI. It’s in the balance!

  At the very least, my takeaway from this is that if someone argues (descriptively or prescriptively) in favour of outright competition without at least acknowledging the above progress, they’re some combination of ill-informed, or wilfully or inadvertently censoring the evidence. Don’t let that be you!

• Oliver Sourbut 16 Oct 2024 9:03 UTC

  3 points

  2

  on: My the­ory of change for work­ing in AI healthtech

  There are a variety of different attitudes that can lead to successionism.

  
  This is a really valuable contribution! (Both the term and laying out a partial extentional definition.)
  
  I think there’s a missing important class of instances which I’ve previously referred to as ‘emotional dependence and misplaced concern’ (though I think my choice of words isn’t amazing). The closest is perhaps your ‘AI parentism’. The basic point is that there is a growing ‘AI rights/​welfare’ lobby because some people (for virtuous reasons!) are beginning to think that AI systems are or could be deserving moral patients. They might not even be wrong, though the current SOTA conversation here appears incredibly misguided. (I have drafts elaborating on this but as I am now a civil servant there are some hoops I need to jump through to publish anything substantive.)

• Oliver Sourbut 1 Sep 2024 21:08 UTC

  1 point

  0

  on: Li­a­bil­ity regimes for AI

  Good old Coase! Thanks for this excellent explainer.

  In contrast, if you think the relevant risks from AI look like people using their systems to do some small amounts of harm which are not particularly serious, you’ll want to hold the individuals responsible for these harms liable and spare the companies.

  Or (thanks to Coase), we could have two classes of harm, with big arbitrarily defined as, I don’t know, say $500m which is a number I definitely just made up, and put liability for big harms on the big companies, while letting the classic societal apparatus for small harms tick over as usual? Surely only a power-crazed bureaucrat would suggest such a thing! (Of course this is prone to litigation over whether particular harms are one big harm or n smaller harms, or whether damages really were half a billion or actually $499m or whatever, but it’s a good start.)

Co­op­er­a­tion and Align­ment in Del­e­ga­tion Games: You Need Both!

• Oliver Sourbut 29 Jul 2024 21:00 UTC

  10 points

  9

  on: De­com­pos­ing Agency — ca­pa­bil­ities with­out desires

  I like this decomposition!

  I think ‘Situational Awareness’ can quite sensibly be further divided up into ‘Observation’ and ‘Understanding’.

  The classic control loop of ‘observe’, ‘understand’, ‘decide’, ‘act’^[1], is consistent with this discussion, where ‘observe’+‘understand’ here are combined as ‘situational awareness’, and you’re pulling out ‘goals’ and ‘planning capacity’ as separable aspects of ‘decide’.

  Are there some difficulties with factoring?

  Certain kinds of situational awareness are more or less fit for certain goals. And further, the important ‘really agenty’ thing of making plans to improve situational awareness does mean that ‘situational awareness’ is quite coupled to ‘goals’ and to ‘implementation capacity’ for many advanced systems. Doesn’t mean those parts need to reside in the same subsystem, but it does mean we should expect arbitrary mix and match to work less well than co-adapted components—hard to say how much less (I think this is borne out by observations of bureaucracies and some AI applications to date).

  ---

  1. ↩︎

     Terminology varies a lot; this is RL-ish terminology. Classic analogues might be ‘feedback’, ‘process model’/​‘inference’, ‘control algorithm’, ‘actuate’/​‘affect’…

• Oliver Sourbut 6 May 2024 9:18 UTC

  1 point

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Trans­form­ers Rep­re­sent Belief State Geom­e­try in their Resi­d­ual Stream

  the original ‘theorem’ was wordcelled nonsense

  Lol! I guess if there was a more precise theorem statement in the vicinity gestured, it wasn’t nonsense? But in any case, I agree the original presentation is dreadful. John’s is much better.

  I would be curious to hear a *precise * statement why the result here follows from the Good Regular Theorem.

  A quick go at it, might have typos.

  Suppose we have

  • $X$ (hidden) state

  • $Y$ output/​observation

  and a predictor

  • $S$ (predictor) state

  • $\hat{Y}$ predictor output

  • $R$ the reward or goal or what have you (some way of scoring ‘was $\hat{Y}$ right?’)

  with structure

  $$\begin{array}{cc}X& \to Y\\ X& \to R\\ Y\to S& \to \hat{Y}\to R\end{array}$$

  Then GR trivially says $S$ (predictor state) should model the posterior $P\left(X|Y\right)$.

  Now if these are all instead processes (time-indexed), we have HMM

  • $X_t$ (hidden) states

  • $Y_t$ observations

  and predictor process

  • $S_t$ (predictor) states

  • ${\hat{Y}}_t$ predictions

  • $R_t$ rewards

  with structure

  $$\begin{array}{cc}{X}_t& \to X_{t+1}\\ X_t& \to Y_t\\ S_{t-1}& \to S_t\\ Y_t\to S_t& \to {\hat{Y}}_{t+1}\to R_{t+1}\\ Y_{t+1}& \to R_{t+1}\end{array}$$

  Drawing together $(X_{t+1},Y_{t+1},{\hat{Y}}_{t+1},R_{t+1})$ as $G_t$ the ‘goal’, we have a GR motif

  $$\begin{array}{cc}{X}_t& \to Y_t\\ Y_t& \to S_t\to G_t\\ S_{t-1}& \to S_t\\ X_t& \to G_t\end{array}$$

  so $S_t$ must model $P(X_t|S_{t-1},Y_t)$; by induction that is $P(X_t|S_0,Y_1,...,Y_t)$.

• Oliver Sourbut 3 May 2024 10:50 UTC

  2 points

  0

  in reply to: Adam Shai’s comment on: Trans­form­ers Rep­re­sent Belief State Geom­e­try in their Resi­d­ual Stream

  I guess my question would be ‘how else did you think a well-generalising sequence model would achieve this?’ Like, what is a sufficient world model but a posterior over HMM states in this case? This is what GR theorem asks. (Of course, a poorly-fit model might track extraneous detail or have a bad posterior.)

  From your preamble and your experiment design, it looks like you correctly anticipated the result, so this should not have been a surprise (to you). In general I object to being sold something as surprising which isn’t (it strikes me as a lesser-noticed and perhaps oft-inadvertent rhetorical dark art and I see it on the rise on LW, which is sad).

  That said, since I’m the only one objecting here, you appear to be more right about the surprisingness of this!

  ---

  The linear probe is new news (but not surprising?) on top of GR, I agree. But the OP presents the other aspects as the surprises, and not this.

• Oliver Sourbut 2 May 2024 18:06 UTC

  1 point

  0

  on: Trans­form­ers Rep­re­sent Belief State Geom­e­try in their Resi­d­ual Stream

  Nice explanation of MSP and good visuals.

  This is surprising!

  Were you in fact surprised? If so, why? (This is a straightforward consequence of the good regulator theorem^[1].)

  In general I’d encourage you to carefully track claims about transformers, HMM-predictors, and LLMs, and to distinguish between trained NNs and the training process. In this writeup, all of these are quite blended.

  ---

  1. ↩︎

     John has a good explication here

• Oliver Sourbut 10 Jan 2024 22:43 UTC

  1 point

  0

  in reply to: Oliver Sourbut’s comment on: Ter­minol­ogy: <some­thing>-ware for ML?

  Incidentally I noticed Yudkowsky uses ‘brainware’ in a few places (e.g. in conversation with Paul Christiano). But it looks like that’s referring to something more analogous to ‘architecture and learning algorithms’, which I’d put more in the ‘software’ camp when in comes to the taxonomy I’m pointing at (the ‘outer designer’ is writing it deliberately).

• Oliver Sourbut 10 Jan 2024 19:03 UTC

  1 point

  0

  in reply to: Oliver Sourbut’s comment on: Some Rules for an Alge­bra of Bayes Nets

  Unironically, I think it’s worth anyone interested skimming that Verma & Pearl paper for the pictures :) especially fig 2

• Oliver Sourbut 10 Jan 2024 18:48 UTC

  5 points

  2

  in reply to: johnswentworth’s comment on: Some Rules for an Alge­bra of Bayes Nets

  Mmm, I misinterpreted at first. It’s only a v-structure if $X$ and $Z$ are not connected. So this is a property which needs to be maintained effectively ‘at the boundary’ of the fully-connected cluster which we’re rewriting. I think that tallies with everything else, right?

  ETA: both of our good proofs respect this rule; the first Reorder in my bad proof indeed violates it. I think this criterion is basically the generalised and corrected version of the fully-connected bookkeeping rule described in this post. I imagine if I/​someone worked through it, this would clarify whether my handwave proof of Frankenstein $⟹$ Stitch is right or not.

• Oliver Sourbut 10 Jan 2024 18:20 UTC

  1 point

  0

  in reply to: Oliver Sourbut’s comment on: Some Rules for an Alge­bra of Bayes Nets

  That’s concerning. It would appear to make both our proofs invalid.

  But I think your earlier statement about incoming vs outgoing arrows makes sense. Maybe Verma & Pearl were asking for some other kind of equivalence? Grr, back to the semantics I suppose.

• Oliver Sourbut 10 Jan 2024 18:18 UTC

  5 points

  0

  in reply to: johnswentworth’s comment on: Some Rules for an Alge­bra of Bayes Nets

  Aha. Preserving v-structures (colliders like $X\to Y\leftarrow Z$) is necessary and sufficient for equivalence^[1]. So when rearranging fully-connected subgraphs, certainly we can’t do it (cost-free) if it introduces or removes any v-structures.

  Plausibly if we’re willing to weaken by adding in additional arrows, there might be other sound ways to reorder fully-connected subgraphs—but they’d be non-invertible. Haven’t thought about that.

  ---

  1. ↩︎

     Verma & Pearl, Equivalence and Synthesis of Causal Models 1990

  What links here?

  • Oliver Sourbut's comment on Some Rules for an Alge­bra of Bayes Nets by johnswentworth (10 Jan 2024 15:42 UTC; 5 points)

• Oliver Sourbut 10 Jan 2024 17:12 UTC

  5 points

  0

  in reply to: johnswentworth’s comment on: Some Rules for an Alge­bra of Bayes Nets

  Mhm, OK I think I see. But $X_{\overline{i}},N,M$ appear to me to make a complete subgraph, and all I did was redirect the $N\to M$. I confess I am mildly confused by the ‘reorder complete subgraph’ bookkeeping rule. It should apply to the $A\to B$ in $A\to B\leftarrow C$, right? But then I’d be able to deduce $A\leftarrow B\leftarrow C$ which is strictly different. So it must mean something other than what I’m taking it to mean.

  Maybe need to go back and stare at the semantics for a bit. (But this syntactic view with motifs and transformations is much nicer!)

• Oliver Sourbut 10 Jan 2024 15:42 UTC

  5 points

  0

  in reply to: Oliver Sourbut’s comment on: Some Rules for an Alge­bra of Bayes Nets

  Perhaps more importantly, I think with Node Introduction we really don’t need $M\leftarrow X\to N$ after all?

  With Node Introduction and some bookkeeping, we can get the $N$ and $M$ graphs topologically compatible, and Frankenstein them. We can’t get as neat a merge as if we also had $M\leftarrow X\to N$ - in particular, we can’t get rid of the arrow $M\to N$. But that’s fine, we were about to draw that arrow in anyway for the next step!

  Is something invalid here? Flagging confusion. This is a slightly more substantial claim than the original proof makes, since it assumes strictly less. Downstream, I think it makes the Resample unnecessary.

  ETA: it’s cleared up below—there’s an invalid Reorder here (it removes a v-structure).