Thane Ruthenis

Karma: 4,996

Thane Ruthenis 29 Jan 2025 5:16 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: What Indicators Should We Watch to Disambiguate AGI Timelines?
Have you looked at samples of CoT of o1, o3, deepseek, etc. solving hard math problems?
Certainly (experimenting with r1′s CoTs right now, in fact). I agree that they’re not doing the brute-force stuff I mentioned; that was just me outlining a scenario in which a system “technically” clears the bar you’d outlined, yet I end up unmoved (I don’t want to end up goalpost-moving).
Though neither are they being “strategic” in the way I expect they’d need to be in order to productively use a billion-token CoT.
Anyhow, this is nice, because I do expect that probably something like this milestone will be reached before AGI
Yeah, I’m also glad to finally have something concrete-ish to watch out for. Thanks for prompting me!

Thane Ruthenis 29 Jan 2025 2:18 UTC
7 points
−1
in reply to: Daniel Kokotajlo’s comment on: What Indicators Should We Watch to Disambiguate AGI Timelines?
Not for math benchmarks. Here’s one way it can “cheat” at them: suppose that the CoT would involve the model generating candidate proofs/derivations, then running an internal (learned, not hard-coded) proof verifier on them, and either rejecting the candidate proof and trying to generate a new one, or outputting it. We know that this is possible, since we know that proof verifiers can be compactly specified.
This wouldn’t actually show “agency” and strategic thinking of the kinds that might generalize to open-ended domains and “true” long-horizon tasks. In particular, this would mostly fail the condition (2) from my previous comment.
Something more open-ended and requiring “research taste” would be needed. Maybe a comparable performance on METR’s benchmark would work for this (i. e., the model can beat a significantly larger fraction of it at 1 billion tokens compared to 1 million)? Or some other benchmark that comes closer to evaluating real-world performance.
Edit: Oh, math-benchmark performance would convince me if we get access to a CoT sample and it shows that the model doesn’t follow the above “cheating” approach, but instead approaches the problem strategically (in some sense). (Which would also require this CoT not to be hopelessly steganographied, obviously.)

Thane Ruthenis 29 Jan 2025 1:19 UTC
6 points
0
in reply to: Daniel Kokotajlo’s comment on: What Indicators Should We Watch to Disambiguate AGI Timelines?
Prooobably ~simultaneously, but I can maybe see it coming earlier and in a way that isn’t wholly convincing to me. In particular, it would still be a fixed-length task; much longer-length than what the contemporary models can reliably manage today, but still hackable using poorly-generalizing “agency templates” instead of fully general “compact generators of agenty behavior” (which I speculate humans to have and RL’d LLMs not to). It would be some evidence in favor of “AI can accelerate AI R&D”, but not necessarily “LLMs trained via SSL+RL are AGI-complete”.
Actually, I can also see it coming later. For example, some suppose that the capability researchers invent some method for reliably-and-indefinitely extending the amount of serial computations a reasoning model can productively make use of, but the compute or memory requirements grow very fast with the length of a CoT. Some fairly solid empirical evidence and theoretical arguments in favor of boundless scaling can appear quickly, well before the algorithms are made optimal enough to (1) handle weeks-long CoTs and/or (2) allow wide adoption (thus making it available to you).
I think the second scenario is more plausible, actually.

Thane Ruthenis 29 Jan 2025 0:29 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: What Indicators Should We Watch to Disambiguate AGI Timelines?
I wish we had something to bet on better than “inventing a new field of science,”
I’ve thought of one potential observable that is concrete, should be relatively low-capability, and should provoke a strong update towards your model for me:
If there is an AI model such that the complexity of R&D problems it can solve (1) scales basically boundlessly with the amount of serial compute provided to it (or to a “research fleet” based on it), (2) scales much faster with serial compute than with parallel compute, and (3) the required amount of human attention (“babysitting”) is constant or grows very slowly with the amount of serial compute.
This attempts to directly get at the “autonomous self-correction” and “ability to think about R&D problems strategically” ideas.
I’ve not fully thought through all possible ways reality could Goodhart to this benchmark, i. e. “technically” pass it but in a way I find unconvincing. For example, if I failed to include the condition (2), o3 would have probably already “passed” it (since it potentially achieved better performance on ARC-AGI and FrontierMath by sampling thousands of CoTs then outputting the most frequent answer). There might be other loopholes like this...
But it currently seems reasonable and True-Name-y to me.
What links here?
- Thane Ruthenis's comment on What Indicators Should We Watch to Disambiguate AGI Timelines? by snewman (29 Jan 2025 2:18 UTC; 7 points)

Thane Ruthenis 28 Jan 2025 22:22 UTC
2 points
0
in reply to: kwiat.dev’s comment on: DeepSeek Panic at the App Store
Well, he didn’t do it yet either, did he? His new announcement is, likewise, just that: an announcement. Manifold is still 35% on him not following through on it, for example.

Thane Ruthenis 28 Jan 2025 21:41 UTC
5 points
1
in reply to: jbash’s comment on: DeepSeek Panic at the App Store
“Intensely competent and creative”, basically, maybe with a side of “obsessed” (with whatever they’re cracked at).

Thane Ruthenis 28 Jan 2025 21:36 UTC
6 points
0
in reply to: Garrett Baker’s comment on: DeepSeek Panic at the App Store
Supposedly Trump announced that back in October, so it should already be priced in.
(Here’s my attempt at making sense of it, for what it’s worth.)

Thane Ruthenis 28 Jan 2025 21:34 UTC
LW: 5 AF: 3
−2
AF
on: Thane Ruthenis’s Shortform
Here’s a potential interpretation of the market’s apparent strange reaction to DeepSeek-R1 (shorting Nvidia).
I don’t fully endorse this explanation, and the shorting may or may not have actually been due to Trump’s tariffs + insider trading, rather than DeepSeek-R1. But I see a world in which reacting this way to R1 arguably makes sense, and I don’t think it’s an altogether implausible world.
If I recall correctly, the amount of money globally spent on inference dwarfs the amount of money spent on training. Most economic entities are not AGI labs training new models, after all. So the impact of DeepSeek-R1 on the pretraining scaling laws is irrelevant: sure, it did not show that you don’t need bigger data centers to get better base models, but that’s not where most of the money was anyway.
And my understanding is that, on the inference-time scaling paradigm, there isn’t yet any proven method of transforming arbitrary quantities of compute into better performance:
- Reasoning models are bottlenecked on the length of CoTs that they’ve been trained to productively make use of. They can’t fully utilize even their context windows; the RL pipelines just aren’t up to that task yet. And if that bottleneck were resolved, the context-window bottleneck would be next: my understanding is that infinite context/”long-term” memories haven’t been properly solved either, and it’s unknown how they’d interact with the RL stage (probably they’d interact okay, but maybe not).
  - o3 did manage to boost its ARC-AGI and (maybe?) FrontierMath performance by… generating a thousand guesses and then picking the most common one...? But who knows how that really worked, and how practically useful it is. (See e. g. this, although that paper examines a somewhat different regime.)
- Agents, from Devin to Operator to random open-source projects, are still pretty terrible. You can’t set up an ecosystem of agents in a big data center and let them rip, such that the ecosystem’s power scales boundlessly with the data center’s size. For all but the most formulaic tasks, you still need a competent human closely babysitting everything they do, which means you’re still mostly bottlenecked on competent human attention.
Suppose that you don’t expect the situation to improve: that the inference-time scaling paradigm would hit a ceiling pretty soon, or that it’d converge to distilling search into forward passes (such that the end users end up using very little compute on inference, like today), and that agents just aren’t going to work out the way the AGI labs promise.
In such a world, a given task can either be completed automatically by an AI for some fixed quantity of compute X, or it cannot be completed by an AI at all. Pouring ten times more compute on it does nothing.
In such a world, if it were shown that the compute needs of a task can be met with ten times less compute than previously expected, this would decrease the expected demand for compute.
The fact that capable models can be run locally might increase the number of people willing to use them (e. g., those very concerned about data privacy), as might the ability to automatically complete 10x as many trivial tasks. But it’s not obvious that this demand spike will be bigger than the simultaneous demand drop.
And I, at least, when researching ways to set up DeepSeek-R1 locally, found myself more drawn to the “wire a bunch of Macs together” option, compared to “wire a bunch of GPUs together” (due to the compactness). If many people are like this, it makes sense why Nvidia is down while Apple is (slightly) up. (Moreover, it’s apparently possible to run the full 671b-parameter version locally, and at a decent speed, using a pure RAM+CPU setup; indeed, it appears cheaper than mucking about with GPUs/Macs, just $6,000.)
This world doesn’t seem outright implausible to me. I’m bearish on agents and somewhat skeptical of inference-time scaling. And if inference-time scaling does deliver on its promises, it’ll likely go the way of search-and-distill.
On balance, I don’t actually expect the market to have any idea what’s going on, so I don’t know that its reasoning is this specific flavor of “well-informed but skeptical”. And again, it’s possible the drop was due to Trump, nothing to do with DeepSeek at all.
But as I’d said, this reaction to DeepSeek-R1 does not seem necessarily irrational/incoherent to me.

Thane Ruthenis 27 Jan 2025 20:59 UTC
4 points
0
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform
I don’t have deep expertise in the subject, but I’m inclined to concur with the people saying that the widely broadcast signals don’t actually represent one consistent thing, despite your plausible argument to the contrary.
Here’s a Scott Alexander post speculating why that might be the case. In short: there was an optimization pressure towards making internal biological signals very difficult to decode, because easily decodable signals were easy target for parasites evolving to exploit them. As the result, the actual signals are probably represented as “unnecessarily” complicated, timing-based combinations of various “basic” chemical, electrical, etc. signals, and they’re somewhat individualized to boot. You can’t decode them just by looking at any one spatially isolated chunk of the body, by design.
Basically: separate chemical substances (and other components that look “simple” locally/from the outside) are not the privileged basis for decoding internal signals. They’re the anti-privileged basis, if anything.

Thane Ruthenis 26 Jan 2025 19:44 UTC
8 points
0
on: Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals
Hmm. This does have the feel of gesturing at something important, but I don’t see it clearly yet...
Free association: geometric rationality.
MIRI’s old results argue that “corrigibility via uncertainty regarding the utility function” doesn’t work, because if the agent maximizes expected utility anyway, it doesn’t matter one whit whether we’re taking expectation over actions or over utility functions. However, the corrigibility-via-instrumental-goals does have the feel of “make the agent uncertain regarding what goals it will want to pursue next”. Is there, therefore, some way to implement something-like-this while avoiding MIRI’s counterexample?
Loophole: the counterexample works in the arithmetically-expected utility regime. What if we instead do it in the geometric one? I. e., have an agent take actions that maximize the geometrically-expected product of candidate utility functions? This is a more conservative/egalitarian regime: any one utility function flipping to negative or going to zero wipes out all value, unlike with sums (which are more tolerant of ignoring/pessimizing some terms, and can have “utility monsters”). So it might potentially make the agent actually hesitant to introduce potentially destructive changes to its environment...
(This is a very quick take and it potentially completely misunderstands the concepts involved. But I figure it’s better to post than not, in case the connection turns out obvious to anyone else.)

Thane Ruthenis 25 Jan 2025 20:19 UTC
LW: 3 AF: 1
0
AF
in reply to: Vladimir_Nesov’s comment on: Thane Ruthenis’s Shortform
Coming back to this in the wake of DeepSeek r1...
I don’t think the cumulative compute multiplier since GPT-4 is that high, I’m guessing 3x, except perhaps for DeepSeek-V3, which wasn’t trained compute optimally and didn’t use a lot of compute, and so it remains unknown what happens if its recipe is used compute optimally with more compute.
How did DeepSeek accidentally happen to invest precisely the amount of compute into V3 and r1 that would get them into the capability region of GPT-4/o1, despite using training methods that clearly have wildly different returns on compute investment?
Like, GPT-4 was supposedly trained for $100 million, and V3 for $5.5 million. Yet, they’re roughly at the same level. That should be very surprising. Investing a very different amount of money into V3′s training should’ve resulted in it either massively underperforming GPT-4, or massively overperforming, not landing precisely in its neighbourhood!
Consider this graph. If we find some training method A, and discover that investing $100 million in it lands us at just above “dumb human”, and then find some other method B with a very different ROI, and invest $5.5 million in it, the last thing we should expect is to again land near “dumb human”.
Or consider this trivial toy model: You have two linear functions, f(x) = Ax and g(x) = Bx, where x is the compute invested, output is the intelligence of the model, and f and g are different training methods. You pick some x effectively at random (whatever amount of money you happened to have lying around), plug it into f, and get, say, 120. Then you pick a different random value of x, plug it into g, and get… 120 again. Despite the fact that the multipliers A and B are likely very different, and you used very different x-values as well. How come?
The explanations that come to mind are:
- It actually is just that much of a freaky coincidence.
- DeepSeek have a superintelligent GPT-6 equivalent that they trained for $10 million in their basement, and V3/r1 are just flexes that they specifically engineered to match GPT-4-ish level.
- DeepSeek directly trained on GPT-4 outputs, effectively just distilling GPT-4 into their model, hence the anchoring.
- DeepSeek kept investing and tinkering until getting to GPT-4ish level, and then stopped immediately after attaining it.
- GPT-4ish neighbourhood is where LLM pretraining plateaus, which is why this capability level acts as a sort of “attractor” into which all training runs, no matter how different, fall.

Thane Ruthenis 23 Jan 2025 18:54 UTC
18 points
4
in reply to: Lao Mein’s comment on: Lao Mein’s Shortform
Hm, that’s not how it looks like to me? Look at the 6-months plot:
The latest growth spurt started January 14th, well before the Stargate news went public, and this seems like just its straight-line continuation. The graph past the Stargate announcement doesn’t look “special” at all. Arguably, we can interpret it as Stargate being part of the reference class of events that are currently driving GE Vernova up – not an outside-context event as such, but still relevant...
Except for the fact that the market indeed did not respond to Stargate immediately. Which makes me think it didn’t react at all, and the next bump you’re seeing is some completely different thing from the reference class of GE-Vernova-raising events (to which Stargate itself does not in fact belong).
Which is probably just something to do with Trump’s expected policies. Note that there was also a jump on November 4th. Might be some general expectation that he’s going to compete with China regarding energy, or that he’d cut the red tape on building plants for his tech backers.
In which case, perhaps it’s because Stargate was already priced-in – or at least whichever fraction of Stargate ($30-$100 billion?) is real and is actually going to be realized. We already knew datacenters of this scale were going to be built this year, after all.

Thane Ruthenis 22 Jan 2025 19:05 UTC
5 points
1
in reply to: Robert Cousineau’s comment on: Thane Ruthenis’s Shortform
Overall, I think that the new Stargate numbers published may (call it 40%) be true, but I also think there is a decent chance this is new administration trump-esque propoganda/bluster (call it 45%),
I think it’s definitely bluster, the question is how much of a done deal it is to turn this bluster into at least $100 billion.
I don’t this this changes the prior expected path of datacenter investment at all. It’s precisely how the expected path was going to look like, the only change is how relatively high-profile/flashy this is being. (Like, if they invest $30-100 billion into the next generation of pretrained models in 2025-2026, and that generation fails, they ain’t seeing the remaining $400 billion no matter what they’re promising now. Next generation makes or breaks the investment into the subsequent one, just as was expected before this.)
Satya Nadella was just asked about how funding looks for stargate, and said “Microsoft is good for investing 80b”
He does very explicitly say “I am going to spend 80 billion dollars building Azure”. Which I think has nothing to do with Stargate.
(I think the overall vibe he gives off is “I don’t care about this Stargate thing, I’m going to scale my data centers and that’s that”. I don’t put much stock into this vibe, but it does fit with OpenAI/Microsoft being on the outs.)

Thane Ruthenis 22 Jan 2025 17:53 UTC
LW: 23 AF: 10
2
AF
on: Thane Ruthenis’s Shortform
So, Project Stargate. Is it real, or is it another “Sam Altman wants $7 trillion”? Some points:
- The USG invested nothing in it. Some news outlets are being misleading about this. Trump just stood next to them and looked pretty, maybe indicated he’d cut some red tape. It is not an “AI Manhattan Project”, at least, as of now.
- Elon Musk claims that they don’t have the money and that SoftBank (stated to have “financial responsibility” in the announcement) has less than $10 billion secured. If true, while this doesn’t mean they can’t secure an order of magnitude more by tomorrow, this does directly clash with “deploying $100 billion immediately” statement.
  - But Sam Altman counters that Musk’s statement is “wrong”, as Musk “surely knows”.
  - I… don’t know which claim I distrust more. Hm, I’d say Altman feeling the need to “correct the narrative” here, instead of just ignoring Musk, seems like a sign of weakness? He doesn’t seem like the type to naturally get into petty squabbles like this, otherwise.
  - (And why, yes, this is how an interaction between two Serious People building world-changing existentially dangerous megaprojects looks like. Apparently.)
- Some people try to counter Musk’s claim by citing Satya Nadella’s statement that Satya’s “good for his $80 billion”. But that’s not referring to Stargate, that’s about Azure. Microsoft is not listed as investing in Stargate at all, it’s only a “technology partner”.
- Here’s a brief analysis from the CIO of some investment firm. He thinks it’s plausible that the stated “initial group of investors” (SoftBank, OpenAI, Oracle, MGX) may invest fifty billion dollars into it over the next four years; not five hundred billion.
  - They don’t seem to have the raw cash for even $100 billion – and if SoftBank secured the missing funding from some other set of entities, why aren’t they listed as “initial equity funders” in the announcement?
Overall, I’m inclined to fall back on @Vladimir_Nesov’s analysis here. $30-50 billion this year seems plausible. But $500 billion is, for now, just Altman doing door-in-the-face as he had with his $7 trillion.
I haven’t looked very deeply into it, though. Additions/corrections welcome!

Thane Ruthenis 21 Jan 2025 2:57 UTC
3 points
0
in reply to: gwern’s comment on: Implications of the inference scaling paradigm for AI safety
McLaughlin announced 2025-01-13 that he had joined OpenAI
He gets onboarded only on January 28th, for clarity.

Thane Ruthenis 21 Jan 2025 0:57 UTC
5 points
−1
in reply to: green_leaf’s comment on: Thane Ruthenis’s Shortform
I think GPT-4 to o3 represent non-incremental narrow progress, but only, at best, incremental general progress.
(It’s possible that o3 does “unlock” transfer learning, or that o4 will do that, etc., but we’ve seen no indication of that so far.)

Thane Ruthenis 21 Jan 2025 0:04 UTC
5 points
0
in reply to: Seth Herd’s comment on: Thane Ruthenis’s Shortform
I’m sure he wants hype, but he doesn’t want high expectations that are very quickly falsified
There’s a possibility that this was a clown attack on OpenAI instead...

Thane Ruthenis 21 Jan 2025 0:02 UTC
6 points
4
in reply to: Joseph Miller’s comment on: Thane Ruthenis’s Shortform
Valid, I was split on whether it’s worth posting vs. it’d be just me taking my part in spreading this nonsense. But it’d seemed to me that a lot of people, including LW regulars, might’ve been fooled, so I erred on the side of posting.

Thane Ruthenis 20 Jan 2025 23:59 UTC
4 points
2
in reply to: Cervera’s comment on: Thane Ruthenis’s Shortform
As I’d said, I think he’s right about the o-series’ theoretic potential. I don’t think there is, as of yet, any actual indication that this potential has already been harnessed, and therefore that it works as well as the theory predicts. (And of course, the o-series scaling quickly at math is probably not even an omnicide threat. There’s an argument for why it might be – that the performance boost will transfer to arbitrary domains – but that doesn’t seem to be happening. I guess we’ll see once o3 is public.)

Thane Ruthenis 20 Jan 2025 23:53 UTC
6 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Thane Ruthenis’s Shortform
The thing is—last time I heard about OpenAI rumors it was Strawberry.
That was part of my reasoning as well, why I thought it might be worth engaging with!
But I don’t think this is the same case. Strawberry/Q* was being leaked-about from more reputable sources, and it was concurrent with dramatic events (the coup) that were definitely happening.
In this case, all evidence we have is these 2-3 accounts shitposting.