Oliver Sourbut

• Oliver SourbutMay 14, 2025, 12:20 PM

  5 points

  0

  on: $500 + $500 Bounty Prob­lem: Does An (Ap­prox­i­mately) Deter­minis­tic Max­i­mal Re­dund Always Ex­ist?

  nit: I found your graphical quantifiers and implications a bit confusing to read. It’s the middle condition that looks weird.

  If I’ve understood, I think you want (given a set of $X_i$, there exists $\Omega$ with all of):

  $$\forall i.[X_{\overline{i}}\to X_i\to \Omega ]\forall \Gamma .(\forall i.[X_{\overline{i}}\to X_i\to \Gamma ])⟹[X\to \Omega \to \Gamma ]X(\to \Omega )\to \Omega$$

  (the third line is just my crude latex rendition of your third diagram)

  Is that order of quantifiers and precedence correct?

  In words, you want an $\Omega$:

  1. which is a redund over the $X_i$

  2. where any $\Gamma$ which is a redund over the $X_i$ is screened from $X$ by $\Omega$

  3. which is approximately deterministic on $X$

  An unverified intuition (I’ll unlikely work on this further): would the joint distribution of all candidate $\Gamma$s work as $\Omega$? It screens off the $X$ from any given $\Gamma$, right (90% confidence)? And it is itself a redund over the $X$ I think (again 90% confidence ish)? I don’t really grok what it means to be approximately deterministic but it feels like this should be? Overall confidence 30% ish, deflating for outside and definedness reasons.

• Oliver SourbutMay 14, 2025, 6:14 AM

  1 point

  0

  in reply to: Dalcy’s comment on: Darcy’s Shortform

  Hey, some thoughts in case helpful. I was exploring a little bit into the ‘agent structure’ sort of questions and the Good/​Gooder regulator landscape.

  You can take GR a bit further by looking at a temporally indexed MDP-like causal diagram and applying various bookkeeping transformations. Search ‘combine nodes’ in John’s post on Bayes net algebra and ‘uncombine’ in my comment on the same.

  Then you can see a ‘good regulator motif’ across many timesteps and timescales and draw some richer conclusions.

  Here’s a comment where I hastily sketch a version of this.

  Wasn’t planning to expand on any of those things, but if you think it’d be especially helpful let me know.

• Oliver SourbutMay 14, 2025, 6:00 AM

  5 points

  0

  on: Ori­ent­ing Toward Wizard Power

  Feels more Alchemist than Wizard.

  Definite strong overlap (the so-called ‘Wizard Book’ literally depicts an alchemist). I think a Wizard has a lot more ‘grand real world directedness’ than the projects you’re pointing to here—though they may also have a bunch of gumption and subcreative genius for solving local issues (like toothbrushes and trousers).

  FWIW I think you also carry the abovementioned Wizard characteristics.

  Of note, alchemists routinely lose contact with social realities, blow themselves up, or go crazy from too much mercury exposure. Something to bear in mind.

  Some more observations: when Alchemists accidentally stumble on something Kings want, Kings usually come and take it. Sensible Kings employ Wizards for counsel, not Alchemists (but they keep Alchemists under watch and maybe try to point them at stuff). In a common-knowledge confrontation between a King and a Wizard, the King usually wins. The Alchemist has no chance against either, they just have to hope and assume they haven’t offended anyone too much.

  King power is pretty real actually, it just isn’t held entirely by the King. The King is a coordinator, a director, and a leviathan, which accrues much more coherence to the subjects than without. Potent. Not always (though often) to the benefit of the King.

• Oliver SourbutMay 7, 2025, 7:19 AM

  1 point

  0

  on: Play­ing in the Creek

  Bostrom touches on a similar question of retaining the joy of striving in a ‘solved world’, in his recent book Deep Utopia. He points to sports as an existence proof that we can and do enjoy challenges (like your beach experiences) with artificial constraints.

• Oliver SourbutApr 23, 2025, 1:31 PM

  3 points

  0

  on: Knowl­edge, Rea­son­ing, and Superintelligence

  (cross-posted on EAF)

  Love this!

  As presaged in our verbal discussion my top conceptual complement would be to emphasise exploration/​experimentation as central to the knowledge production loop—the cycle of ‘developing good taste to plan better experiments to improve taste (and planning model)’ is critical (indispensable?) to ‘produce new knowledge which is very helpful by the standards of human civilization’ (on any kind of meaningful timescale).

  This because just flailing, or even just ‘doing stuff’, gets you some novelty of observations, but directedly seeking informative circumstances at the boundaries of the known (which includes making novel unpredictable events happen, as well as getting equipped with richer means to observe and record them, and perhaps preparing to deliberatively extract insight) turns out to be able to mine vastly more insight per resource (time, materials, etc.). Hence science, but also hence individual human and animal playfulness, curiosity, adversarial exercises and drills (self-play ish), and whatnot.

  Said another way, maybe I’d characterise ‘the way that fluid intelligence and crystallised intelligence synergise in the knowledge production loop’ as ‘directed exploration/​experimentation’?

  Having said that, I don’t necessarily think these capacities need to reside ‘in the same mind’, just as contemporary human orgs get more of this done and more effectively than individuals. But the pieces do need to be fit to each other (like, a physicist with great physics taste can’t usually very well complement a bio lab without first becoming a person with great bio taste).

• Oliver SourbutFeb 28, 2025, 6:16 PM

  3 points

  0

  on: What does it take to defend the world against out-of-con­trol AGIs?

  But hey, I don’t think the “Demis Hassabis signs off on perpetual mass sabotage of the global AI research enterprise” thing is going to actually happen either.

  Here’s Demis’ recent version of ‘IAEA/​CERN for AGI’

  Featuring
  

  ...to monitor unsafe projects, right? And sort of (awkward nose touch) deal with those

• Oliver SourbutJan 16, 2025, 11:42 PM

  5 points

  0

  in reply to: Jeremy Gillen’s comment on: De­cep­tive Align­ment and Homuncularity

  Thanks for this! I hadn’t seen those quotes, or at least hadn’t remembered them.

  I actually really appreciate Alex sticking his neck out a bit here and suggesting this LessWrong dialogue. We both have some contrary opinions, but his takes were probably a little more predictably unwelcome in this venue. (Maybe we should try this on a different crowd—we could try rendering this on Twitter too, lol.)

  There’s definitely value to being (rudely?) shaken out of lazy habits of thinking—though I might not personally accuse someone of fanfiction research! As discussed in the dialogue, I’m still unsure the exact extent of correct- vs mis-interpretation and I think Alex has a knack for (at least sometimes correctly) calling out others’ confusion or equivocation.

• Oliver SourbutJan 16, 2025, 4:49 PM

  5 points

  0

  in reply to: TurnTrout’s comment on: De­cep­tive Align­ment and Homuncularity

  As an aside, note that some of “AIs misbehave in ways we’ve predicted” can be a self-fulfilling prophecy due to out-of-context generalization: We wrote lots of stories about how powerful AI will do X; powerful AI is trained on our stories and realizes it’s powerful; the powerful AI does X. So it’s possible that AIs would e.g. try to self-exfiltrate and goal-guard much less frequently if we hadn’t talked about it as much or those stories were expunged from the pretraining corpus.

  I think this is conceivable if either

  • the apparent reasoning is actually still just a bag of surface heuristics

  • it’s still quite shallow reasoning, and the salience of those as strategies is a consequence mainly of pretraining

  I sincerely doubt the first nowadays. For the second, I think probably not; these strike me as quite basic and ‘obvious’ strategies, but I can’t give it less than 20% that a system with those expunged from training data would need substantially more in-context nudging to recognise them.

• Oliver SourbutJan 16, 2025, 4:43 PM

  5 points

  2

  in reply to: TurnTrout’s comment on: De­cep­tive Align­ment and Homuncularity

  I appreciate you revisiting another couple of months later! This topic continues to evolve.

  It was also an update for me that merely-RLHF-ish training on top of GPTs appears to have been enough (in latest-generation models at least) to get something mostly-consistently ‘actually trying’ in an apparently coherent way. I expected that to happen; I didn’t know when. The lede which they somewhat buried in that paper was also that,

  Furthermore, we find that this compliance gap persists in our synthetic document fine-tuned setup even when we entirely remove the model’s hidden chain-of-thought scratchpad

  (though the gap is smaller and we don’t have the tools to understand the mechanism)

  The bigger update from that paper, for me, was that the somewhat-coherent goals apparently-pursued by these things actually align with the target constitution somewhat! i.e. (that which was) reward(ed) was the optimisation target after all, a bit??

  We’ll see how robustly these sorts of findings hold up to future developments and greater elicitation efforts.

De­cep­tive Align­ment and Homuncularity

• Oliver SourbutJan 16, 2025, 12:49 PM

  15 points

  1

  on: What Is The Align­ment Prob­lem?

  Organisms in general typically sense their environment and take different actions across a wide variety of environmental conditions, so as to cause there to be approximate copies of themselves in the future.^[4] That’s basic agency.^[5]

  I agree with this breakdown, except I start the analysis with moment-to-moment deliberation, and note that having there (continue to) be relevantly similar deliberators is a very widely-applicable intermediate objective, from where we get control (‘basic agency’) but also delegation and replication.

  The way the terms have typically been used historically, the simplest summary would be:

  • Today’s LLMs and image generators are generative models of (certain parts of) the world.

  • Systems like e.g. o1 are somewhat-general planners/​solvers on top of those models. Also, LLMs can be used directly as planners/​solvers when suitably prompted or tuned.

  • To go from a general planner/​solver to an agent, one can simply hook the system up to some sensors and actuators (possibly a human user) and specify a nominal goal… assuming the planner/​solver is capable enough to figure it out from there.

  Yep! But (I think maybe you’d agree) there’s a lot of bleed between these abstractions, especially when we get to heavily finetuned models. For example...

  Applying all that to typical usage of LLMs (including o1-style models): an LLM isn’t the kind of thing which is aligned or unaligned, in general. If we specify how the LLM is connected to the environment (e.g. via some specific sensors and actuators, or via a human user), then we can talk about both (a) how aligned to human values is the nominal objective given to the LLM^[8], and (b) how aligned to the nominal objective is the LLM’s actual effects on its environment. Alignment properties depend heavily on how the LLM is wired up to the environment, so different usage or different scaffolding will yield different alignment properties.

  Yes and no? I’d say that the LLM-plus agent’s objectives are some function of

  • incompletely-specified objectives provided by operators

  • priors and biases from training/​development

    • pretraining

    • finetuning

  • scaffolding/​reasoning structure (including any multi-context/​multi-persona interactions, internal ratings, reflection, refinement, …)

    • or these things developed implicitly through structured CoT

  • drift of various kinds

  and I’d emphasise that the way that these influences interact is currently very poorly characterised. But plausibly the priors and biases from training could have nontrivial influence across a wide variety of scenarios (especially combined with incompletely-specified natural-language objectives), at which point it’s sensible to ask ‘how aligned’ the LLM is. I appreciate you’re talking in generalities, but I think in practice this case might take up a reasonable chunk of the space! For what it’s worth, the perspective of LLMs as pre-agent building blocks and conditioned-LLMs as closer to agents is underrepresented, and I appreciate you conceptually distinguishing those things here.

• Oliver SourbutJan 2, 2025, 10:27 PM

  14 points

  2

  on: Care­less talk on US-China AI com­pe­ti­tion? (and crit­i­cism of CAIS cov­er­age)

  This was a hasty and not exactly beautifully-written post. It didn’t get much traction here on LW, but it had more engagement on its EA Forum crosspost (some interesting debate in the comments).

  I still endorse the key messages, which are:

  • don’t rule out international and inter-bloc cooperation!

    • it’s realistic and possible, though it hangs in the balance

    • it might be our only hope

  • lazy generalisations have power

    • us-vs-them rhetoric is self-reinforcing

  Not content with upbraiding CAIS, I also went after Scott Alexander later in the month for similar lazy wording.

  You don’t win by capturing the dubious accolade of nominally belonging to the bloc which directly destroys everything!

  Not long after writing these, I became a civil servant (not something I ever expected!) at the AI Safety Institute (UK), so I’ve generally steered clear from public comment in the area of AI+politics since then.

  That said, I’ve been really encouraged by the progress on international cooperation, which as I say remains ‘realistic and possible’! For example, in late 2023 we had the Bletchley declaration, signed by US, China, and many others, which stated (among other things):

  We affirm that, whilst safety must be considered across the AI lifecycle, actors developing frontier AI capabilities… have a particularly strong responsibility for ensuring the safety of these AI systems… in particular to prevent misuse and issues of control, and the amplification of other risks.

  Since then we’ve had AI safety appearing among top Chinese priorities, official dialogues between US and China on AI policy, track 2 discussions in Thailand and the UK, two international dialogues on AI safety in Venice and Beijing, and meaningful AI safety research from prominent Chinese academics, and probably many other encouraging works of diplomacy—it’s not my area of expertise, and I haven’t even been trying that hard to look for evidence here.

  Of course there’s also continued conflict and competition for hardware and talent, and a competitive element to countries’ approaches to AI. It’s in the balance!

  At the very least, my takeaway from this is that if someone argues (descriptively or prescriptively) in favour of outright competition without at least acknowledging the above progress, they’re some combination of ill-informed, or wilfully or inadvertently censoring the evidence. Don’t let that be you!

• Oliver SourbutOct 16, 2024, 9:03 AM

  3 points

  2

  on: My the­ory of change for work­ing in AI healthtech

  There are a variety of different attitudes that can lead to successionism.

  
  This is a really valuable contribution! (Both the term and laying out a partial extentional definition.)
  
  I think there’s a missing important class of instances which I’ve previously referred to as ‘emotional dependence and misplaced concern’ (though I think my choice of words isn’t amazing). The closest is perhaps your ‘AI parentism’. The basic point is that there is a growing ‘AI rights/​welfare’ lobby because some people (for virtuous reasons!) are beginning to think that AI systems are or could be deserving moral patients. They might not even be wrong, though the current SOTA conversation here appears incredibly misguided. (I have drafts elaborating on this but as I am now a civil servant there are some hoops I need to jump through to publish anything substantive.)

• Oliver SourbutSep 1, 2024, 9:08 PM

  1 point

  0

  on: Li­a­bil­ity regimes for AI

  Good old Coase! Thanks for this excellent explainer.

  In contrast, if you think the relevant risks from AI look like people using their systems to do some small amounts of harm which are not particularly serious, you’ll want to hold the individuals responsible for these harms liable and spare the companies.

  Or (thanks to Coase), we could have two classes of harm, with big arbitrarily defined as, I don’t know, say $500m which is a number I definitely just made up, and put liability for big harms on the big companies, while letting the classic societal apparatus for small harms tick over as usual? Surely only a power-crazed bureaucrat would suggest such a thing! (Of course this is prone to litigation over whether particular harms are one big harm or n smaller harms, or whether damages really were half a billion or actually $499m or whatever, but it’s a good start.)

Co­op­er­a­tion and Align­ment in Del­e­ga­tion Games: You Need Both!

• Oliver SourbutJul 29, 2024, 9:00 PM

  10 points

  9

  on: De­com­pos­ing Agency — ca­pa­bil­ities with­out desires

  I like this decomposition!

  I think ‘Situational Awareness’ can quite sensibly be further divided up into ‘Observation’ and ‘Understanding’.

  The classic control loop of ‘observe’, ‘understand’, ‘decide’, ‘act’^[1], is consistent with this discussion, where ‘observe’+‘understand’ here are combined as ‘situational awareness’, and you’re pulling out ‘goals’ and ‘planning capacity’ as separable aspects of ‘decide’.

  Are there some difficulties with factoring?

  Certain kinds of situational awareness are more or less fit for certain goals. And further, the important ‘really agenty’ thing of making plans to improve situational awareness does mean that ‘situational awareness’ is quite coupled to ‘goals’ and to ‘implementation capacity’ for many advanced systems. Doesn’t mean those parts need to reside in the same subsystem, but it does mean we should expect arbitrary mix and match to work less well than co-adapted components—hard to say how much less (I think this is borne out by observations of bureaucracies and some AI applications to date).

  ---

  1. ↩︎

     Terminology varies a lot; this is RL-ish terminology. Classic analogues might be ‘feedback’, ‘process model’/​‘inference’, ‘control algorithm’, ‘actuate’/​‘affect’…

• Oliver SourbutMay 6, 2024, 9:18 AM

  1 point

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Trans­form­ers Rep­re­sent Belief State Geom­e­try in their Resi­d­ual Stream

  the original ‘theorem’ was wordcelled nonsense

  Lol! I guess if there was a more precise theorem statement in the vicinity gestured, it wasn’t nonsense? But in any case, I agree the original presentation is dreadful. John’s is much better.

  I would be curious to hear a *precise * statement why the result here follows from the Good Regular Theorem.

  A quick go at it, might have typos.

  Suppose we have

  • $X$ (hidden) state

  • $Y$ output/​observation

  and a predictor

  • $S$ (predictor) state

  • $\hat{Y}$ predictor output

  • $R$ the reward or goal or what have you (some way of scoring ‘was $\hat{Y}$ right?’)

  with structure

  $$\begin{array}{cc}X& \to Y\\ X& \to R\\ Y\to S& \to \hat{Y}\to R\end{array}$$

  Then GR trivially says $S$ (predictor state) should model the posterior $P\left(X|Y\right)$.

  Now if these are all instead processes (time-indexed), we have HMM

  • $X_t$ (hidden) states

  • $Y_t$ observations

  and predictor process

  • $S_t$ (predictor) states

  • ${\hat{Y}}_t$ predictions

  • $R_t$ rewards

  with structure

  $$\begin{array}{cc}{X}_t& \to X_{t+1}\\ X_t& \to Y_t\\ S_{t-1}& \to S_t\\ Y_t\to S_t& \to {\hat{Y}}_{t+1}\to R_{t+1}\\ Y_{t+1}& \to R_{t+1}\end{array}$$

  Drawing together $(X_{t+1},Y_{t+1},{\hat{Y}}_{t+1},R_{t+1})$ as $G_t$ the ‘goal’, we have a GR motif

  $$\begin{array}{cc}{X}_t& \to Y_t\\ Y_t& \to S_t\to G_t\\ S_{t-1}& \to S_t\\ X_t& \to G_t\end{array}$$

  so $S_t$ must model $P(X_t|S_{t-1},Y_t)$; by induction that is $P(X_t|S_0,Y_1,...,Y_t)$.

• Oliver Sourbut 3 May 2024 10:50 UTC

  2 points

  0

  in reply to: Adam Shai’s comment on: Trans­form­ers Rep­re­sent Belief State Geom­e­try in their Resi­d­ual Stream

  I guess my question would be ‘how else did you think a well-generalising sequence model would achieve this?’ Like, what is a sufficient world model but a posterior over HMM states in this case? This is what GR theorem asks. (Of course, a poorly-fit model might track extraneous detail or have a bad posterior.)

  From your preamble and your experiment design, it looks like you correctly anticipated the result, so this should not have been a surprise (to you). In general I object to being sold something as surprising which isn’t (it strikes me as a lesser-noticed and perhaps oft-inadvertent rhetorical dark art and I see it on the rise on LW, which is sad).

  That said, since I’m the only one objecting here, you appear to be more right about the surprisingness of this!

  ---

  The linear probe is new news (but not surprising?) on top of GR, I agree. But the OP presents the other aspects as the surprises, and not this.

• Oliver Sourbut 2 May 2024 18:06 UTC

  1 point

  0

  on: Trans­form­ers Rep­re­sent Belief State Geom­e­try in their Resi­d­ual Stream

  Nice explanation of MSP and good visuals.

  This is surprising!

  Were you in fact surprised? If so, why? (This is a straightforward consequence of the good regulator theorem^[1].)

  In general I’d encourage you to carefully track claims about transformers, HMM-predictors, and LLMs, and to distinguish between trained NNs and the training process. In this writeup, all of these are quite blended.

  ---

  1. ↩︎

     John has a good explication here

• Oliver Sourbut 10 Jan 2024 22:43 UTC

  1 point

  0

  in reply to: Oliver Sourbut’s comment on: Ter­minol­ogy: <some­thing>-ware for ML?

  Incidentally I noticed Yudkowsky uses ‘brainware’ in a few places (e.g. in conversation with Paul Christiano). But it looks like that’s referring to something more analogous to ‘architecture and learning algorithms’, which I’d put more in the ‘software’ camp when in comes to the taxonomy I’m pointing at (the ‘outer designer’ is writing it deliberately).