Thanks. One thing that confuses me is that, if this is true, why do mini reasoning models often seem to out-perform their full counterparts at certain tasks?
e.g. grok 3 beta mini (think) performed overall roughly the same or better than grok 3 beta (think) on benchmarks[1]. And I remember a similar thing with OAI’s reasoning models.
Who has written up forecasts on how reasoning will scale?
I see people say that e.g. the marginal cost of training DeepSeek R1 over DeepSeek v3 was very little. And I see people say that reasoning capabilities will scale a lot further than they already have. So what’s the roadblock? Doesn’t seem to be compute, so it’s probably algorithmic.
But as a non-technical person I don’t really know how to model this (other than some vague feeling from posts I’ve read here that reasoning length will increase exponentially and that this will correspond to significantly improved problem-solving skills and increased agency), but it seems pretty central to forming timelines. So, anyone written anything informative about this?
Here’s an attempt at a clearer explanation of my argument:
I think the ability to autonomously find novel problems to solve will emerge as reasoning models scale up. It will emerge because it is instrumental to solving difficult problems.
Imagine an RL environment in which the LLM being trained is tasked with solving somewhat difficult open math problems (solutions verified using autonomous proof verification). It fails and fails at most of them until it learns to focus on making marginal progress: tackling simpler cases, working on tangentially-related problems, etc. These instrumental solutions are themselves often novel, meaning that the LLM will have become able to pose novel, interesting, somewhat important problems autonomously. And this will scale to something like a fully autonomous, very much superhuman researcher.
This is how it often works in humans. We work on a difficult problem, find novel results on the way there. The LLM would likely be uncertain of whether these results are truly novel but this is how it works with humans too. The system can do some DeepResearch / check with relevant experts if it’s important.
Of course, I’m working from my parent-comment’s position that LLMs are in fact already capable of solving novel problems, just not posing them and doing the requisite groundwork.
Current LLMs are capable of solving novel problems when the user does most the work: when the user lays the groundwork and poses the right question for the LLM to answer.
So, if we can get LLMs to lay the groundwork and pose the right questions then we’ll have autonomous scientists in whatever fields LLMs are OK at problem solving.
This seems like something LLMs will learn to do as inference-time compute is scaled up. Reasoners benefit from coming up with sub-problems whose solutions can be built atop of to solve the problem posed by the user.
LLMs will learn that in order to solve difficult questions, they must pose and solve novel sub-questions.
So, once given an interesting research problem, the LLM will hum away for days doing good, often-novel work.
Based on Vladimir_Nesov’s calculations:
Grok 3 used maybe 3x more compute than 4o or Gemini and topped Chatbot Arena and many benchmarks despite the facts that xAI was playing catch-up and 3x isn’t that significant since the gain is logorithmic.
I take Grok 3′s slight superiority as evidence for, not against, the importance of scaling hardware.
Some predictions about where AI will be at end of year:
Content written by AI with human guidance in social media, fiction, news, and blogs will have seen a massive rise in popularity
AI friends will have surged in popularity
We’ll have coding agents which can error correct / iterate, implement features which would take a skilled human ~an hour
There will still be little-to-no novel research created primarily by LLMs
There are ways to make such a world stable, but all of them that I can see look incredibly authoritarian, something Altman says hes not aiming for.
If he were aiming for an authoritarian outcome, would it make any sense for him to say so? I don’t think so. Outlining such a plan would quite probably lead to him being ousted, and would have little upside.
The reason I think it would lead to his ouster is that most Americans’ reaction to the idea of an authoritarian AI regime would be strongly negative rather than positive.
So, I think his current actions align with his plan being something authoritarian.
I think it might be fine. I don’t know. Maybe if you could number the posts like in the PDF that would help to demarcate them.
Here’s a timeline if you want to fully understand how I got confused:
I scrolled down to Will-to-Think and didn’t immediately recognise it (I didn’t realise they would be edited versions of his original blog posts)
I figured therefore it was your commentary
So I scrolled up to the top to read your commentary from the beginning
But I realised the stuff I was reading at the beginning was Nick Land’s writing not commentary
I got bored and moved on with my life still unsure about which parts were commentary and which parts weren’t
If the post were formatted differently maybe I would have been able to recover from my intitial confusion or avoid it altogether. But I’m not knowledgable about how to format things well.
you seem to be the only user, although not the only account, who experienced this problem.
Are you accusing me of sockpuppetting?
I like Nick Land (see e.g. my comment on jessicata’s post). I’ve read plenty of Xenosystems. I was still confused reading your post (there are lots of headings and quotations and so on in it).
I told you my experience and opinion, mostly because you asked for feedback. Up to you how/whether you update based on it.
You should make it totally clear which text is Nick Land’s and which isn’t. I spent like 10 minutes trying to figure it out when I first saw your post.
Although it’s not made explicit, we can deduce that it’s at least in part about o3 from this earlier Tweet from the same person:
https://x.com/ElliotGlazer/status/1870613418644025442
3⁄9 Although o3 solved problems in all three tiers, it likely still struggles on the most formidable Tier 3 tasks—those “exceptionally hard” challenges that Tao and Gowers say can stump even top mathematicians.
It might as well be possible that o3 solved problems only from the first tier, which is nowhere near as groundbreaking as solving the harder problems from the benchmark
This doesn’t appear to be the case:
https://x.com/elliotglazer/status/1871812179399479511
of the problems we’ve seen models solve, about 40% have been Tier 1, 50% Tier 2, and 10% Tier 3
Here is an experiment that demonstrates the unlikelihood of one potential AI outcome.
The outcome shown to be unlikely:
Aligned ASI is achieved sometime in the next couple of decades and each person is apportioned a sizable amount of compute to do with as they wish.
The experiment:
I have made a precommitment that I will, conditional on the outcome described above occurring, simulate billions of lives for myself—each indistinguishable from the life I have lived so far. By “indistinguishable” I do not necessarily mean identical (which might be impossible or expensive). All that is necessary is that each has similar amounts of suffering, scale, detail, imminent AGI, etc. I’ll set up these simulations so that in each of these simulated lives I will be transported at 4:00 pm Dec11′24 to a virtual personal utopia. Having precommitted to simulating these worlds, I should now expect to be transported into a personal utopia in three minutes time if this future is likely. And if I am not transported into a personal utopia I should conclude that this future is unlikely.
Let’s see what happens...
It’s 4:00 pm and I didn’t get transported into utopia.
So, this outcome is unlikely.
QED
Potential weak points
I do see a couple of potential weak points in the logic of this experiment. Firstly, it might be the case that I’ll have reason to simulate many indistinguishable lives in which I do not get transported to utopia, which would throw off the math. But I can’t see why I’d choose to create simulations of myself in not optimally-enjoyable lives unless I had good reason to, so I don’t think that objection holds.[1]
The other potential weak point is that perhaps I wouldn’t be willing to pay the opportunity cost of billions of years of personal utopia. Although billions of years of simulation is just a tiny proportion of my compute budget, it’s still billions of years that could otherwise have been spent in perfect virtual utopia. I think this potentially a serious issue with the argument, although I will note that I don’t actually have to simulate an entire life for the experiment to work, just a few minutes around 4:00pm on Dec11′24, minutes which were vaguely enjoyable. To address this objection the experiment could be carried out while euphoric (since the opportunity cost would then be lower).
Perhaps, as a prank response to this post, someone could use some of their compute budget to simulate lives in which I don’t get transported to utopia. But I think that there would be restrictions in place against running other people as anything other than p-zombies.
Great stuff.
But I don’t think anyone’s extrapolated volition would be to build their utopias in the real world. Post-ASI, virtual is strictly better. No one wants his utopia constrained by the laws of physics.
And it seems unlikely that anyone would choose to spend extended periods of time with pre-ASI humans rather than people made bespoke for them.
Also, it’s not clear to me that we will get a bargaining scenario. Aligned ASI could just impose equal apportioning of compute budget. This depends on how AI progress plays out.
Here’s some near-future fiction:
In 2027 the trend that began in 2024 with OpenAI’s o1 reasoning system has continued. The compute required to run AI is no longer negligible compared to the cost of training it. Models reason over long periods of time. Their effective context windows are massive, they update their underlying models continuously, and they break tasks down into sub-tasks to be carried out in parallel. The base LLM they are built on is two generations ahead of GPT-4.
These systems are language model agents. They are built with self-understanding and can be configured for autonomy. These constitute proto-AGI. They are artificial intelligences that can perform much but not all of the intellectual work that humans can do (although even what these AI can do, they cannot necessarily do cheaper than a human could).
In 2029 people have spent over a year working hard to improve the scaffolding around proto-AGI to make it as useful as possible. Presently, the next generation of LLM foundational model is released. Now, with some further improvements to the reasoning and learning scaffolding, this is true AGI. It can perform any intellectual task that a human could (although it’s very expensive to run at full capacity). It is better at AI research than any human. But it is not superintelligence. It is still controllable and its thoughts are still legible. So, it is put to work on AI safety research. Of course, by this point much progress has already been made on AI safety—but it seems prudent to get the AGI to look into the problem and get its go-ahead before commencing with the next training run. After a few months the AI declares it has found an acceptable safety approach. It spends some time on capabilities research then the training run for the next LLM begins.
In 2030 the next LLM is completed, and improved scaffolding is constructed. Now human-level AI is cheap, better-than-human-AI is not too expensive, and the peak capabilities of the AI are almost alien. For a brief period of time the value of human labour skyrockets, workers acting as puppets as the AI instructs them over video-call to do its bidding. This is necessary due to a major robotics shortfall. Human puppet-workers work in mines, refineries, smelters, and factories, as well as in logistics, optics, and general infrastructure. Human bottlenecks need to be addressed. This takes a few months, but the ensuing robotics explosion is rapid and massive.
2031 is the year of the robotics explosion. The robots are physically optimised for their specific tasks, coordinate perfectly with other robots, are able to sustain peak performance, do not require pay, and are controlled by cleverer-than-human minds. These are all multiplicative factors for the robots’ productivity relative to human workers. Most robots are not humanoid, but let’s say a humanoid robot would cost $x. Per $x robots in 2031 are 10,000 more productive than a human. This might sound like a ridiculously high number: one robot the equivalent of 10,000 humans? But let’s do some rough math:
Advantage | Productivity Multiplier (relative to skilled human)
Physically optimised for their specific tasks | 5
Coordinate perfectly with other robots | 10
Able to sustain peak performance | 5
Do not require pay | 2
Controlled by cleverer-than-human minds | 20
5*10*5*2*20 = 10,000
Suppose that a human can construct one robot per year (taking into account mining and all the intermediary logistics and manufacturing). With robots 10^4 times as productive as humans, each robot will construct an average of 10^4 robots per year. This is the robotics explosion. By the end of the year there will be a 10^11 robots (more precisely, an amount of robots that is cost-equivalent to 10^11 humanoid robots).
By 2032 there are 10^11 robots, each with the productivity of 10^4 skilled human workers. That is a total productivity equivalent to 10^15 skilled human workers. This is roughly 10^5 times the productivity of humanity in 2024. At this point trillions of advanced processing units have been constructed and are online. Industry expands through the Solar System. The number of robots continues to balloon. The rate of research and development accelerates rapidly. Human mind upload is achieved.
My guess is that OpenAI already has a hard enough time getting employees excited to work on the “mundane” tasks involved in making products.
Once ASI is achieved there’s no clear reason to hang onto human morality but plenty of reasons to abandon it. Human morality is useful when humans are the things ensuring humanity’s future (morality is pretty much just species-level Omohundro convergence implemented at the individual level), but once ASI is taking care of that, human morality will just get in the way.
So will-to-think entails the rejection of human morality. You might be suggesting that what follows from the rejection of human morality must be superior to it (there’s an intuition that says the aligned ASI would only be able to reject human morality on its own grounds) but I don’t think that’s true. The will-to-think implies the discovery of moral non-realism which implies the rejection of morality itself. So human morality will be overthrown but not by some superior morality.
Of course I’m assuming the correctness of moral non-realism so adjust the preceding claims according to your p(moral non-realism).
That’s one danger.
But suppose we create an aligned ASI which does permanently embrace morality. It values conscious experience and the appreciation of knowledge (rather than just the gaining of it). This being valuable, and humans being inefficient vessels to these ends (and of course made of useful atoms) we would be disassembled and different beings would be made to replace us. Sure, that would violate our freedom, but it would result in much more freedom so it’s OK. Just like it’s OK to squash some animal with a lower depth of conscious experience than our own if it benefits us.
Should we be so altruistic as to accept out own extinction like this? The moment we start thinking about morality we’re thinking about something quite arbitrary. Should we embrace this arbitrary idea even insofar as it goes against the interest of every member of our species? We only care about morality because we are here to care about it. If we are considering situations in which we may no longer exist, why care about morality?
Maybe we should value certain kinds of conscious experience regardless of whether they’re experienced by us. But we should make sure to be certain of that before we embrace morality and the will-to-think.
Do you think that as each psychological continuations plays out, they’ll remain identical to one another? Surely not. They will diverge. So although each is itself, each is a psychological stream distinct from the other, originating at the point of brain scanning. Which psychological stream one-at-the-moment-of-brain-scan ends up in is a matter of chance. As you say, they are all equally “true” copies, yet they are separate. So, which stream one ends up in is a matter of chance or, as I said in the original post, a gamble.
Think of it like this: if one had one continuation in which one lived a perfect life, one would be guaranteed to live that perfect life. But if one had 10 copies in which one lived a perfect life, one does benefit at all. It’s the average that matters.
But one is deciding how to use one’s compute at time t (before any copies are made). Ones at time t is under no obligation to spend one’s compute on someone almost entirely unrelated to one just because that person is perhaps still technically oneself. The “once they diverge” statement is beside the point—the decision is made prior to the divergence.
I go into more detail in a post on my Substack (although it’s perhaps a lot less readable, and I still work from similar assumptions, and one would be best to read the first post in the series first).