Oliver Sourbut

• Oliver SourbutJun 3, 2025, 12:32 PM

  2 points

  0

  in reply to: Tom Davidson’s comment on: Will com­pute bot­tle­necks pre­vent a soft­ware in­tel­li­gence ex­plo­sion?

  With that weighting, human experts have much less frontier-taste-relevant data than they have total data… AI can learn from all copies.

  Ah, yep yep. (Though humans do in fact learn (with losses) from other copies, and within a given firm/​lab/​community the transfer is probably quite high.)

  Hmm, I think we have basically the same model, with maybe a bit different parameters (about which we’re both somewhat uncertain). But think that readers of the OP without some of that common model might be misled.

  From a lot of convos and writing, I infer that many people conflate a lot of aspects of intelligence into one thing. With that view, ‘intelligence explosion’ is just a dynamic where a single variable, ‘the intelligence’ (or ‘the algorithmic quality’), gets real big real fast. And then of course, because intelligence is the thing that gets you new technology, you can get all the new technology.

  About this, you said,

  There is then a further question, if that IE goes very far, of whether AI will generalize to chemistry etc

  I’d have thought “yes” bc you achieve somewhat superhuman sample efficiency and can quickly get as much chemistry exp as a human expert

  revealing that you correctly distinguish different factors of intelligence like ‘sample efficiency’ and ‘chemistry knowledge’ (I think I already knew this but it’s good to have local confirmation), and that you don’t think a software-only IE yields all of them.

  Regarding the second sentence, it could be a misleading use of terms to call that ‘generalisation’^[1], but I agree that ‘sample efficiency’ is among the relevant aspects of intelligence (and is a candidate for one that could be mostly generalisably built up automated in silico), and a relevant complement is ‘chemistry (frontier research) experience’, and that a lot of each taken together may effectively get you chemistry research taste in addition (which can yield new chemistry knowledge if applied with suitable experimental apparatus).

  I’m emphasising^[2] (in my exploration post, in this thread, and the sister comment about frontier taste depreciation) that there’s practically a wide gulf between ‘hoover taste up from web data’ and ‘robotics or humans-with-headsets’, in two ways. The first tops out somewhere (probably at sub-frontier) due to the depreciation of frontier research taste. The second cluster doesn’t top out anywhere, but is slower and has more barriers to getting started. Is a year to exceed humanity’s peak taste bold in most domains? Not sure! If a lot of in silico is possible, maybe it’s doable. That might include cyber and software (and maybe narrow particular areas of chem/​bio where simulation is especially good).

  If you know for sure that those other bottlenecks proceed super fast, you don’t need to necessarily clarify what intelligence factors you’re talking about for practical purposes, but if you’re not sure, I think it’s worth being super clear about it where possible.

  ---

  1. ↩︎

     I might instead prefer terminology like ‘quickly implies … (on assumptions...)’

  2. ↩︎

     Incidentally, the other thing I’m emphasising (but I think you’d agree?) is that on this view, R&Ds are always substantially driven by experimental throughput, with ‘sample efficiency (of the combined workforce) at accruing research taste’ being the main other rate-determining factor (because the steady state of research taste depends on this, and exploration quality * experimental throughput is progress). Throwing more labour at it can make your serial experimentation a bit faster, and can parallelise experimentation (with some parallelism discount), with presumably very diminishing returns. Throwing smarter labour (as in, better sample efficiency, and maybe faster-thinking, with diminishing returns), can increase the rate, by getting more insight per experiment and choosing better experiments.

• Oliver SourbutJun 1, 2025, 2:02 PM

  2 points

  0

  in reply to: Tom Davidson’s comment on: Will com­pute bot­tle­necks pre­vent a soft­ware in­tel­li­gence ex­plo­sion?

  Yes more ‘data’, but not necessarily more frontier-taste-relevant experience, which requires either direct or indirect contact with frontier experiments. That might be a crux.

• Oliver SourbutMay 31, 2025, 7:05 PM

  4 points

  0

  in reply to: Oliver Sourbut’s comment on: Will com­pute bot­tle­necks pre­vent a soft­ware in­tel­li­gence ex­plo­sion?

  BTW I also think novelty taste depreciates quickly as the frontier of a domain moves, so I’m less bullish on hoovering up taste from existing data/​interviews for a permanent taste boost. But it might take you some way past frontier, which may or may not be sizeably impactful in a given domain.

• Oliver SourbutMay 31, 2025, 7:03 PM

  2 points

  0

  in reply to: Tom Davidson’s comment on: Will com­pute bot­tle­necks pre­vent a soft­ware in­tel­li­gence ex­plo­sion?

  I think we’re on the same page about what factors exist.

  For the intelligence explosion I think all we need is experience with AI RnD?

  Well, sort of, but precisely because intelligence isn’t single-dimensional, we have to ask ‘what is exploding here?‘. And like I said, I think you straightforwardly get speed. And plausibly you get sample efficiency (in the inference-from-given-data sense). And maybe you get ‘exploratory planning’, which is hypothetically one, more transferrable, factor of general exploration efficiency.

  But you don’t get domain-specific novelty/​interestingness taste, which is another critical factor, except for the domains you can easily get lots of data on. So either you need to hoover that up from existing data, or get it some other how. That might be interviews, it might be a bunch of robotics and other actuator/​sensor equipment, it might be the Shulman-style humans-with-headsets-as-actuators thing and learning by experience. But then the question becomes whether there’s enough data to hoover it up real fast with your speed+efficiency learning, or whether you hit a bunch of scaling up bottlenecks (and in which domains).

• Oliver SourbutMay 28, 2025, 9:32 AM

  6 points

  0

  in reply to: Nathan Helm-Burger’s comment on: Hu­man takeover might be worse than AI takeover

  I am hopeful that one of the things we can do with just-before-the-brink AI will be to accelerate the design and deployment of such voluntary coordination contracts. Could we manage to use AI to speed-run the invention and deployment of such subsidiarity governance systems? I think the biggest challenge to this is how fast it would need to move in order to take effect in time. For a system that needs extremely broad buy-in from a large number of heterogenous actors, speed of implementation and adoption is a key weak point.

  FYI FLF has a Fellowship on AI for Human Reasoning which centrally targets objectives like this (if I’ve understood)

• Oliver SourbutMay 26, 2025, 9:09 AM

  2 points

  0

  in reply to: Oliver Sourbut’s comment on: Will com­pute bot­tle­necks pre­vent a soft­ware in­tel­li­gence ex­plo­sion?

  I wrote a bit about experimentation recently.

  You seem very close to taking this position seriously when you talk about frontier experiments and experiments in general, but I think you also need to notice that experiments come in more dimensions than that. Like, you don’t learn how to be better at chemistry just by playing around with GPUs.

• Oliver SourbutMay 26, 2025, 8:57 AM

  4 points

  0

  in reply to: Tom Davidson’s comment on: Will com­pute bot­tle­necks pre­vent a soft­ware in­tel­li­gence ex­plo­sion?

  It’s quite clear that labs both want more high-quality researchers—top talent has very high salaries, reflecting large marginal value-add.

  Three objections, one obvious. I’ll state them strongly, a bit devil’s advocate; not sure where I actually land on these things.

  Obvious: salaries aren’t that high.

  Also, I model a large part of the value to companies of legible credentialed talent being the marketing value to VCs and investors, who (even if lab leadership can) can’t tell talent apart except by (rare) legible signs. This is actually a way to get more compute (and other capital). (The legible signs are rare because compute is a bottleneck! So a Matthew effect pertains.)

  Finally, the utility of labs is very convex in the production of AI: the actual profit comes from time spent selling a non-commoditised frontier offering at large margin. So small AI production speed gains translate into large profit gains.

• Oliver SourbutMay 26, 2025, 8:07 AM

  2 points

  0

  on: Will com­pute bot­tle­necks pre­vent a soft­ware in­tel­li­gence ex­plo­sion?

  The best objection to an SIE

  I think the compute bottleneck is a reasonable objection, but there is also the fairly straightforward objection that gaining skills takes experience, and experience takes interaction (or hoovering up from web data).

  You can get experience in things like ‘writing fast code’ easily, so a speed explosion is fairly plausible (up to diminishing returns). But various R&Ds or human influence or whatever are much harder to get experience for. So our exploded SIs might be super fast and maybe super good at learning from experience, but out of the gate at best human expert level where relevant data are abundant, and at best human novice level where data aren’t.

• Oliver SourbutMay 23, 2025, 5:45 PM

  6 points

  2

  on: Oliver Sour­but’s Shortform

  What constitutes cooperation?

  Realised my model pipeline:

  1. surface options (or find common ground)

  2. negotiate choices (agree a course of action)

  3. cooperate/​enforce (counteract defections, actually do the joint good thing)

  was missing an important preliminary step.

  For cooperation to happen, you also need:

  1. identify potential coalitions (who could benefit from cooperating)!

  (Could break down further: identifying, getting common knowledge, and securing initial prospective cooperative intent.)

  In some cases, ‘identifying potential coalitions’ might be a large, even dominant part of the challenge of cooperation, especially when effects are diffuse!

  That applies to global commons and it applies when coordinating political action. What other cases?

  ‘Identifying potential coalitions’ is what a lot of activism is about, and it might also be a big part of what various cooperative memeplexes like tribes, religions, political parties etc are doing.

  This feels to me like another important part of the picture that new tech could potentially amplify!

  Could we newly empower large groups of humans to cooperate by recognising and fulfilling the requirements of this cooperation pipeline?

• Oliver SourbutMay 21, 2025, 5:13 PM

  1 point

  0

  in reply to: Oliver Sourbut’s comment on: AI 2027 is a Bet Against Am­dahl’s Law

  I polished and published the draft

  • Introducing exploration and experimentation

  • Why does exploration matter?

  • Research and taste

  • From play to experimentation

  • Exploration in AI, past and future

  • Research by AI: AI with research taste?

  • Opportunities

You Can’t Skip Ex­plo­ra­tion: Why un­der­stand­ing ex­per­i­men­ta­tion and taste is key to un­der­stand­ing AI

• Oliver SourbutMay 20, 2025, 12:35 PM

  2 points

  0

  on: Oliver Sour­but’s Shortform

  If you want to be twice as profitable as your competitors, you don’t have to be twice as good as them. You just have to be slightly better.

  I think AI development is mainly compute constrained (relevant for intelligence explosion dynamics).

  There are some arguments against, based on the high spending of firms on researcher and engineer talent. The claim is that this supports one or both of a) large marginal returns to having more (good) researchers or b) steep power laws in researcher talent (implying large production multipliers from the best researchers).

  Given that the workforces at labs remain not large, I think the spending naively supports (b) better.

  But in fact I think there is another, even better explanation:

  • Researchers’ taste (an AI production multiplier) varies more smoothly

  • (research culture/​collective intelligence of a team or firm may be more important)

  • Marginal parallel researchers have very diminishing AI production returns (sometimes negative, when the researchers have worse taste)

  • (also determining a researcher’s taste ex ante is hard)

  • BUT firms’ utility is sharply convex in AI production

    • capturing more accolades and market share are basically the entire game

    • spending as much time as possible with a non-commoditised offering allows profiting off fast-evaporating margin

  • so firms are competing over getting cool stuff out first

    • time-to-delivery of non-commoditised (!) frontier models

  • and getting loyal/​sticky customer bases

    • ease-of-adoption of product wrapping

    • sometimes differentiation of offerings

  • this turns small differences in human capital/​production multiplier/​research taste into big differences in firm utility

  • so demand for the small pool of the researchers with (legibly) great taste is very hot

  This also explains why it’s been somewhat ‘easy’ (but capital intensive) for a few new competitors to pop into existence each year, and why firms’ revealed preferred savings rate into compute capital is enormous (much greater than 100%!).

  We see token prices drop incredibly sharply, which supports the non-commoditised margin claim (though this is also consistent with a Wright’s Law effect from (runtime) algorithmic efficiency gains, which should definitely also be expected).

  A lot of engineering effort is being put into product wrappers and polish, which supports the customer base claim.

  The implications include: headroom above top human expert teams’ AI research taste could be on the small side (I think this is right for many R&D domains, because a major input is experimental throughput). So both quantity and quality of (perhaps automated) researchers should have steeply diminishing returns in AI production rate. But might they nevertheless unlock a practical monopoly (or at least an increasingly expensive barrier to entry) on AI-derived profit, by keeping the (more monetisable) frontier out of reach of competitors?

• Oliver SourbutMay 19, 2025, 4:01 PM

  1 point

  0

  on: AI 2027 is a Bet Against Am­dahl’s Law

  “choosing better experiments” is a relatively advanced skill, which will likely not emerge until well after experiment implementation skills

  I have a draft discussing this. (Facepalm, should publish more often...)
  
  Certainly choosing better experiments requires at least one of:

  • large scaleup in experimental observations (to get the experience to drive taste acquisition)

  • superhuman sample efficiency in taste acquisition

  • extreme reasoning/​deliberation on top of weak taste, adding up to greater taste (I think there are likely very diminishing returns to this, but superspeed might yield it)

  I think your claim is betting on the first one, and also assuming that you can only get that by increasing throughput.

  But maybe you could slurp enough up from existing research logs, or from interviews with existing researchers, or something like that. Then you’d be competing with the still overall larger, but more tacit and more distributed-between-brains research experience of all the humans in the org.

FLF Fel­low­ship on AI for Hu­man Rea­son­ing: $25-50k, 12 weeks

• Oliver SourbutMay 14, 2025, 12:20 PM

  5 points

  0

  on: $500 + $500 Bounty Prob­lem: Does An (Ap­prox­i­mately) Deter­minis­tic Max­i­mal Re­dund Always Ex­ist?

  nit: I found your graphical quantifiers and implications a bit confusing to read. It’s the middle condition that looks weird.

  If I’ve understood, I think you want (given a set of $X_i$, there exists $\Omega$ with all of):

  $$\forall i.[X_{\overline{i}}\to X_i\to \Omega ]\forall \Gamma .(\forall i.[X_{\overline{i}}\to X_i\to \Gamma ])⟹[X\to \Omega \to \Gamma ]X(\to \Omega )\to \Omega$$

  (the third line is just my crude latex rendition of your third diagram)

  Is that order of quantifiers and precedence correct?

  In words, you want an $\Omega$:

  1. which is a redund over the $X_i$

  2. where any $\Gamma$ which is a redund over the $X_i$ is screened from $X$ by $\Omega$

  3. which is approximately deterministic on $X$

  An unverified intuition (I’ll unlikely work on this further): would the joint distribution of all candidate $\Gamma$s work as $\Omega$? It screens off the $X$ from any given $\Gamma$, right (90% confidence)? And it is itself a redund over the $X$ I think (again 90% confidence ish)? I don’t really grok what it means to be approximately deterministic but it feels like this should be? Overall confidence 30% ish, deflating for outside and definedness reasons.

• Oliver SourbutMay 14, 2025, 6:14 AM

  1 point

  0

  in reply to: Dalcy’s comment on: Darcy’s Shortform

  Hey, some thoughts in case helpful. I was exploring a little bit into the ‘agent structure’ sort of questions and the Good/​Gooder regulator landscape.

  You can take GR a bit further by looking at a temporally indexed MDP-like causal diagram and applying various bookkeeping transformations. Search ‘combine nodes’ in John’s post on Bayes net algebra and ‘uncombine’ in my comment on the same.

  Then you can see a ‘good regulator motif’ across many timesteps and timescales and draw some richer conclusions.

  Here’s a comment where I hastily sketch a version of this.

  Wasn’t planning to expand on any of those things, but if you think it’d be especially helpful let me know.

• Oliver SourbutMay 14, 2025, 6:00 AM

  5 points

  0

  on: Ori­ent­ing Toward Wizard Power

  Feels more Alchemist than Wizard.

  Definite strong overlap (the so-called ‘Wizard Book’ literally depicts an alchemist). I think a Wizard has a lot more ‘grand real world directedness’ than the projects you’re pointing to here—though they may also have a bunch of gumption and subcreative genius for solving local issues (like toothbrushes and trousers).

  FWIW I think you also carry the abovementioned Wizard characteristics.

  Of note, alchemists routinely lose contact with social realities, blow themselves up, or go crazy from too much mercury exposure. Something to bear in mind.

  Some more observations: when Alchemists accidentally stumble on something Kings want, Kings usually come and take it. Sensible Kings employ Wizards for counsel, not Alchemists (but they keep Alchemists under watch and maybe try to point them at stuff). In a common-knowledge confrontation between a King and a Wizard, the King usually wins. The Alchemist has no chance against either, they just have to hope and assume they haven’t offended anyone too much.

  King power is pretty real actually, it just isn’t held entirely by the King. The King is a coordinator, a director, and a leviathan, which accrues much more coherence to the subjects than without. Potent. Not always (though often) to the benefit of the King.

• Oliver SourbutMay 7, 2025, 7:19 AM

  1 point

  0

  on: Play­ing in the Creek

  Bostrom touches on a similar question of retaining the joy of striving in a ‘solved world’, in his recent book Deep Utopia. He points to sports as an existence proof that we can and do enjoy challenges (like your beach experiences) with artificial constraints.

• Oliver SourbutApr 23, 2025, 1:31 PM

  3 points

  0

  on: Knowl­edge, Rea­son­ing, and Superintelligence

  (cross-posted on EAF)

  Love this!

  As presaged in our verbal discussion my top conceptual complement would be to emphasise exploration/​experimentation as central to the knowledge production loop—the cycle of ‘developing good taste to plan better experiments to improve taste (and planning model)’ is critical (indispensable?) to ‘produce new knowledge which is very helpful by the standards of human civilization’ (on any kind of meaningful timescale).

  This because just flailing, or even just ‘doing stuff’, gets you some novelty of observations, but directedly seeking informative circumstances at the boundaries of the known (which includes making novel unpredictable events happen, as well as getting equipped with richer means to observe and record them, and perhaps preparing to deliberatively extract insight) turns out to be able to mine vastly more insight per resource (time, materials, etc.). Hence science, but also hence individual human and animal playfulness, curiosity, adversarial exercises and drills (self-play ish), and whatnot.

  Said another way, maybe I’d characterise ‘the way that fluid intelligence and crystallised intelligence synergise in the knowledge production loop’ as ‘directed exploration/​experimentation’?

  Having said that, I don’t necessarily think these capacities need to reside ‘in the same mind’, just as contemporary human orgs get more of this done and more effectively than individuals. But the pieces do need to be fit to each other (like, a physicist with great physics taste can’t usually very well complement a bio lab without first becoming a person with great bio taste).

• Oliver SourbutFeb 28, 2025, 6:16 PM

  3 points

  0

  on: What does it take to defend the world against out-of-con­trol AGIs?

  But hey, I don’t think the “Demis Hassabis signs off on perpetual mass sabotage of the global AI research enterprise” thing is going to actually happen either.

  Here’s Demis’ recent version of ‘IAEA/​CERN for AGI’

  Featuring
  

  ...to monitor unsafe projects, right? And sort of (awkward nose touch) deal with those