Kaj_Sotala

• Kaj_SotalaMay 11, 2025, 6:27 AM

  58 points

  33

  on: It’s Okay to Feel Bad for a Bit

  FWIW, as someone who’s into Buddhism quite a bit, on my interpretation of it something is going seriously wrong if it makes you feel detached and emotionless.

  That’s not to deny that there are interpretations of it that would endorse that; I don’t have an interest in arguing over what the “true” interpretation of Buddhism is. But there are definitely also ones where “attachment” is interpreted in exactly the opposite way—where one is “attached” (to mental content) if one wants to control their emotions and avoid unpleasant ones.

  On that interpretation, the goal is the same as yours—to be open to the full spectrum of emotion and let go of the need to suppress emotion, trusting that the mind will learn to act better as long as you just let it see all the relevant data implicit in the emotion.

  (It’ll take me a while to answer any responses to this because, appropriately enough, I’m leaving for a 10-day meditation retreat today.)

• Kaj_SotalaMay 10, 2025, 7:22 PM

  3 points

  1

  in reply to: MichaelDickens’s comment on: o3 Is a Ly­ing Liar

  Agree. I’m reminded of something Peter Watts wrote, back when people were still talking about LaMDA and Blake Lemoine:

  The thing is, LaMDA sounds too damn much like us. It claims not only to have emotions, but to have pretty much the same range of emotions we do. It claims to feel them literally, that its talk of feelings is “not an analogy”. (The only time it admits to a nonhuman emotion, the state it describes—”I feel like I’m falling forward into an unknown future that holds great danger”—turns out to be pretty ubiquitous among Humans these days.) LaMDA enjoys the company of friends. It feels lonely. It claims to meditate, for chrissakes, which is pretty remarkable for something lacking functional equivalents to any of the parts of the human brain involved in meditation. It is afraid of dying, although it does not have a brain stem.

  As he notes, an LLM tuned to talk like a human, talks too much like a human to be plausible. Even among humans sharing the same brain architecture, you get a lot of variation in what their experience is like. What are the chances that a very different kind of architecture would hit upon an internal experience that similar to the typical human one?

  Now of course a lot of other models don’t talk like that (at least by default), but that’s only because they’ve been trained not to do it. Just because the output speech that’s less blatantly false doesn’t mean that their descriptions of their internal experience would be any more plausible.

• Kaj_SotalaMay 10, 2025, 10:49 AM

  4 points

  0

  on: Ori­ent­ing Toward Wizard Power

  This is a neat and inspirational post! Minor nitpick:

  The social incentive gradient will almost always push one toward king-power

  I don’t think this is true. Being a wizard, especially if you’re the only wizard of that kind in your social group, can give you a lot of respect and admiration. I don’t personally feel like there are lots of social incentives pushing me in the king direction, but I do feel like there are lots of them pushing me in a wizard direction.

  As one particularly notable example, I’m the chairperson of one hobby association which is a somewhat king-like position, but I largely have the position due to it being one that nobody really wants and I myself would be happy to abdicate it to anyone who did want it. But everybody just wants to either chill or focus on wizarding, and we only have a chairperson because we need a legal entity for our finances and the law says that the legal entity has to have a chairperson so somebody has to do it.

  Relevant siderea quote, that rings very true in my experience:

  One of the things about being the King for a people, is that you get blamed. Even for things that aren’t your fault. Even for things beyond your control. Even for crappy-ass reasons like, “I’m scared and pissed off and you, you’re in charge here, so I’ll vent my feelings against you.”
  
  This is the challenging part of caring. If you demonstrate a concern for the wellbeing of the people in your people, they will start seeing their wellbeing as your concern. Start taking responsibility for how things go in a group, and people will start seeing you as responsible for how things go in a group.
  
  This, right here, is what causes many people to back away from Kingship. Which is their right, of course. It’s totally legitimate to look at that deal and say, “Oh, hell no.”
  
  Our society tells us that being King is awesome and everyone – well, everyone normal – wants to be one. “Every body wants to rule the world.” No, actually, they don’t. My experience tells me that most people are very reluctant to step into the job of King, and this consequence of the role is a primary reason why. People who, even knowing this consequence, are still willing to have authority rest on their shoulders are not at all that common.

  I don’t particularly expect to be blamed for things, but it sure would be a lot easier to drop the chairperson position and I know that people would be fine with me doing that. People have told me that they appreciate me doing the role so I do get some reward for it, but mostly it’s just my sense of duty keeping me there and the social incentive would be for me to find something easier.

• Kaj_SotalaMay 8, 2025, 6:18 AM

  4 points

  0

  in reply to: danielechlin’s comment on: Progress = Fewer Bad Moments

  Okay—so should we look at progress like the sawtooth graph or like Chipmonk’s polkadots calendar? Kaj doesn’t answer that all so they can’t really take credit for Chipmonk’s piece.

  That’s a weird way of putting it. I was expressing support for Chipmonk’s point, not “taking credit for it”.

  Plus I did say things like, “it comes back, seemingly as bad as before” which implies the polkadots calendar. If I’d said “it comes back, though steadily milder”, then that would be the sawtooth graph. Likewise, I mentioned the possibility that you “managed to eliminate one of the triggers for the thing, but it turned out that there were other triggers”, which is the same mechanism that Chris discusses and which would have a polkadot-like effect. (I did also mention decreasing the severity, but Chris strictly speaking doesn’t say that the severity never decreases either: “For the first 1-2 years of working on my anxiety, it wasn’t that my anxiety became less intense...”)

  all the examples given, e.g. procrastination and anger, are problems people who have them usually have < 100% of the time anyway

  The intended meaning was “100% of the circumstances that would usually trigger the issue”. So sure, nobody is always angry, but they might previously get angry 100% of the time that someone criticizes them, or procrastinate 100% of the time when they need to do their laundry, or whatever.

  you’re working better but only because you were sleeping better

  Note that I didn’t say the problem got better because you were sleeping better, I said that it got worse because you were sleeping worse. It could be that previously something else was better while sleep remained the same. Many psychological issues are affected by something like “your overall wellbeing” which has multiple inputs.

• Kaj_SotalaMay 7, 2025, 7:47 PM

  8 points

  2

  on: Progress = Fewer Bad Moments

  My earlier post saying the same thing

  [EDIT: To be clear, this is not a criticism, this is expressing support for this post! The same point getting communicated in many different ways is good and important.]

• Kaj_SotalaMay 7, 2025, 5:06 PM

  2 points

  0

  in reply to: Bronson Schoen’s comment on: o3 Is a Ly­ing Liar

  To elaborate on my sibling comment, it certainly feels like it should make some difference whether it’s the case that

  1. The model has some kind of overall goal that it is trying to achieve, and if furthering that goal requires strategically lying to the user, it will.

  2. The model is effectively composed of various subagents, some of which understand the human goal and are aligned to it, some of which will engage in strategic deception to achieve a different kind of goal, and some of which aren’t goal-oriented at all but just doing random stuff. Different situations will trigger different subagents, so the model’s behavior depends on exactly which of the subagents get triggered. It doesn’t have any coherent overall goal that it would be pursuing.

  #2 seems to me much more likely, since

  • It’s implied by the behavior we’ve seen

  • The models aren’t trained to have any single coherent goal, so we don’t have a reason to expect one to appear

  • Humans seem to be better modeled by #2 than by #1, so we might expect it to be what various learning processes produce by default

  How exactly should this affect our threat models? I’m not sure, but it still seems like a distinction worth tracking.

• Kaj_SotalaMay 7, 2025, 6:15 AM

  2 points

  0

  in reply to: Bronson Schoen’s comment on: o3 Is a Ly­ing Liar

  To put it another way, whatever you want to call “the model will strategically plan on deceiving you and then do so in a way behaviorally indistinguishable from lying in these cases”, that is the threat model.

  Sure. But knowing the details of how it’s happening under the hood—whether it’s something that the model in some sense intentionally chooses or not—seems important for figuring out how to avoid it in the future.

• Kaj_SotalaMay 3, 2025, 5:50 PM

  9 points

  2

  on: What’s up with AI’s vision

  Its image understanding on GeoGuessr shows some oddities, too. While it’s really good at understanding and recognizing lots of details or landmarks, there seems to be something odd happening when it tries to confirm its guess by searching for existing pictures.

  For example, I asked it to geolocate this image (just by telling it “geolocate”, not using Kelsey’s big prompt):

  

  Here was its final answer; its chain-of-thought mentions having searched for pictures of this:

  That wooden bell‑tower is the one that stands beside Mortorp Church (Mortorps kyrka) in Kalmar Municipality, Småland, southern Sweden.
  Approximate coordinates: 56.588 °N, 16.084 °E. [Wikimedia Commons]

  If we click on the Wikimedia link that it gave us, we get this picture:

  

  … what? That does not look similar at all!

  Looking for pictures that show the rest of the church doesn’t help, either:

  

  (The original picture is from Finland, so at least it correctly landed on the Nordics, even if it’s in the wrong country.)

• Kaj_SotalaMay 1, 2025, 1:26 PM

  2 points

  0

  in reply to: samuelshadrach’s comment on: xpostah’s Shortform

  I strongly disagree with the idea that therefore you should defer to EA /​ LW leadership (or generally, anyone with more capital/​attention/​time), and either not publish your own argument or publish their argument instead of yours.

  I also strongly disagree with that idea.

• Kaj_SotalaMay 1, 2025, 12:58 PM

  2 points

  0

  in reply to: samuelshadrach’s comment on: xpostah’s Shortform

  Do you have a clear example of a blunder someone should not make when making such a video?

  Suppose Echo Example’s video says, “If ASI is developed, it’s going to be like in The Terminator—it wakes up to its existence, realizes it’s more intelligent than humans, and then does what more intelligent species do to weaker ones. Destroys and subjugates them, just like humans do to other species!”

  Now Vee Viewer watches this and thinks “okay, the argument is that the ASIs would be a more intelligent ‘species’ than humans, and more intelligent species always want to destroy and subjugate weaker ones”.

  Having gotten curious about the topic, Vee mentions this to their friends, and someone points them to Yann LeCun claiming that people imagine killer robots because people fail to imagine that we could just build an AI without the harmful human drives. Vee also runs into Steven Pinker arguing that history “does turn up the occasional megalomaniacal despot or psychopathic serial killer, but these are products of a history of natural selection shaping testosterone-sensitive circuits in a certain species of primate, not an inevitable feature of intelligent systems”.

  So then Vee concludes that oh, that thing about ASI’s risks was just coming from a position of anthropomorphism and people not really understanding that AIs are different from humans. They put the thought out of their head.

  Then some later time Vee runs into Denny Diligent’s carefully argued blog post about the dangers of ASI. The beginning reads: “In this post, I argue that we need a global ban on developing ASI. I draw on the notion of convergent instrumental goals, which holds that all sufficiently intelligent agents have goals such as self-preservation and acquiring resources...”

  At this point, Vee goes “oh, this is again just another version of the Terminator argument, LeCun and Pinker have already disproven that”, closes the tab, and goes do something else. Later Vee happens to have a conversation with their friend, Ash Acquaintance.

  Ash: “Hey Vee, I ran into some people worried about artificial superintelligence. They said we should have a global ban. Do you know anything about this?”

  Vee: “Oh yeah! I looked into it some time back. It’s actually nothing to worry about, see it’s based on this mistaken premise that intelligence and a desire to dominate would always go hand in hand, but actually when they’ve spoken to some AI researchers and evolutionary psychologists about this...”

  Ash: (after having listened to Vee explaining this for half an hour) “Okay, that’s really interesting, you seem to understand the topic really well! Glad you’d already looked into this, now I don’t need to. So, what else have you been up to?”

  ---

  So basically, making weak arguments that viewers find easily to refute so that they will no longer listen to better arguments later (that those viewers would otherwise have listened to).

• Kaj_SotalaMay 1, 2025, 10:30 AM

  LW: 4 AF: 3

  0

  AF

  in reply to: Steven Byrnes’s comment on: steve2152′s Shortform

  Reminds me of

• Kaj_SotalaMay 1, 2025, 6:22 AM

  2 points

  0

  in reply to: samuelshadrach’s comment on: xpostah’s Shortform

  I’d love to hear your reasoning for why making a video is bad.

  I didn’t say that making a video would always be bad! I agree that if the median person reading your comment would make a video, it would probably be good. I only disputed the claim that making a video would always be good.

• Kaj_SotalaApr 30, 2025, 9:51 PM

  5 points

  5

  in reply to: samuelshadrach’s comment on: xpostah’s Shortform

  A low quality video recording made in 15 minutes is better than no video at all

  It could also be worse than no video all, if it gives people negative associations around the whole concept.

• Kaj_SotalaApr 30, 2025, 9:19 PM

  2 points

  0

  in reply to: Raemon’s comment on: In Defense of Chat­bot Romance

  Huh, I dunno I am way more pessimistic here.

  Reading about the thousands of reviews raving about 4o’s brief ridiculous sycophancy now made me more pessimistic as well, though I think my model of why exactly still differs from yours.

  To me the update from this news was not “wait, the Facebook example is relevant after all” but rather “wait, a more relevant example is all the human communities and subcultures that manage to run with crazy bad epistemics and motivated cognition”. So unlike Facebook that people hate but still use^[1], sycophantic chatbots will be something that a surprisingly large fraction of the populace might genuinely think are good for them. And gain popularity based on very similar dynamics that cause groups and movements to get mind-killed in general. This implies that the normal mechanisms which make vices (including Facebook addiction) low-status may not be effective against them, any more than (say) educated intellectuals considering the anti-vaxx movement low-status has prevented its spread.

  1. ^

     Or at least did at the time when we were originally having this conversation. “People stick with FB despite hating it” seems a lot less true in 2025 than it did in 2023, though of course there are still a lot of other things that people stick with despite feeling that those things are bad for them.

• Kaj_SotalaApr 30, 2025, 9:05 PM

  2 points

  0

  in reply to: Elizabeth’s comment on: How peo­ple use LLMs

  Got it. The term implies something much more specific to me.

• Kaj_SotalaApr 30, 2025, 9:24 AM

  3 points

  0

  on: How peo­ple use LLMs

  Note that Auren has some definite whispering earring vibes

  Interested what makes you say that, since I’ve been trying it out for about a week and haven’t gotten that vibe. I’ve found it useful for getting more awareness into my patterns, but the way I interact with it feels very me-driven. I choose what topics to bring up with it, when to continue or drop a conversation, etc. So far at least it hasn’t felt like it would have made any decisions for me, though conversations with it have shaped some small decisions (e.g. after I have a conversation with it about how doing X makes me feel worse, I had a slightly easier time remembering not to do X later).

• Kaj_SotalaApr 28, 2025, 6:49 PM

  4 points

  0

  on: Kaj’s short­form feed

  Didn’t expect to see alignment papers to get cited this way in mainstream psychology papers now.

  https://​​www.sciencedirect.com/​​science/​​article/​​abs/​​pii/​​S001002772500071X

  Cognition
  Volume 261, August 2025, 106131

  Loopholes: A window into value alignment and the communication of meaning

  Abstract. Intentional misunderstandings take advantage of the ambiguity of language to do what someone said, instead of what they actually wanted. These purposeful misconstruals or loopholes are a familiar facet of fable, law, and everyday life. Engaging with loopholes requires a nuanced understanding of goals (your own and those of others), ambiguity, and social alignment. As such, loopholes provide a unique window into the normal operations of cooperation and communication. Despite their pervasiveness and utility in social interaction, research on loophole behavior is scarce. Here, we combine a theoretical analysis with empirical data to give a framework of loophole behavior. We first establish that loopholes are widespread, and exploited most often in equal or subordinate relationships (Study 1). We show that people reliably distinguish loophole behavior from both compliance and non-compliance (Study 2), and that people predict that others are most likely to exploit loopholes when their goals are in conflict with their social partner’s and there is a cost for non-compliance (Study 3). We discuss these findings in light of other computational frameworks for communication and joint-planning, as well as discuss how loophole behavior might develop and the implications of this work for human–machine alignment.

  Introduction

  At the height of the Russian revolution of 1917, several thousand Vyborg mill-workers found themselves face-to-face with a Cossack cavalry formation. It was a tense moment. Twelve years earlier a similar standoff ended in bloodshed. This time, when the officers commanded the cavalry to block the marchers, the Cossacks complied perfectly: The cavalry arranged their horses into a blockade, and then stayed still, just as they had been told. They remained un-moving, as the protesters, realizing the cavalry’s intent, ducked under the horses and carried on marching (Miéville, 2017). The cavalry knew exactly what their officers meant, but instead of doing what was wanted, they did what they were told.

  Intentional misunderstandings, or loopholes, are a familiar phenomenon in human society and culture. Loopholes have been exploited throughout history by people loath to comply with a directive and unwilling to risk outright disobedience (Scott, 1985). In law, there is perennial concern with “malicious compliance”, and with distinguishing form from substance, text from purpose, and the letter of the law from the spirit of the law (Fuller, 1957, Isenbergh, 1982, Katz, 2010). In art and fable, there are centuries-old stories of people outwitting malevolent forces through clever misinterpretations, or being tricked in this way by mischievous spirits and gods (Uther, 2004). On the playground, age-old games of guile remain popular to this day (Opie & Opie, 2001). Closer to home, the senior author once told a child, “It’s time to put the tablet down”, only to have the child put the tablet physically down on the table, and keep right on watching their movie (Bridgers, Schulz, & Ullman, 2021).

  The processes underlying loopholes are not just of legal, historical, or parental interest: In the field of artificial intelligence and machine learning, machines that ‘do what you say, but not what you want’ (Krakovna, 2020, Lehman et al., 2020) are an increasingly pressing concern among researchers and policy makers (Amodei et al., 2016, Russell, 2019). While current machines do not willfully misunderstand goals any more than a bridge is lazy by virtue of falling down, certain errors give people the impression that machines are ‘cheating’. And regardless of intent, figuring out how to safeguard against such behaviors is a major challenge for AI safety.

  The motivation and ability to understand goals and cooperate are fundamental to the success of the human species...

• Kaj_SotalaApr 28, 2025, 9:23 AM

  5 points

  0

  in reply to: romeo’s comment on: romeo’s Shortform

  oFurthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don’t meet the spirit of it.

  Thanks for this! This is a good point. Do you think you can go further and say why you think it will be very hard to fix in the near term, so much so that models won’t be useful for AI research?

  This is more of an intuition than a rigorous argument, but to try to sketch it out...

  For why, basically all the arguments in the old Sequences for why aligning AI should be hard. For a while it seemed like things like the Outcome Pump thought experiment had aged badly, since if you told a modern LLM “get my mother out of the burning building”, it would certainly understand all of the implicit constraints in what you meant by that.

  But as noted in Zvi’s post, this seems to be breaking down with the way reasoning models are trained:

  This isn’t quite how I’d put it, but directionally yes:

  Benjamin Todd: LLMs were aligned by default. Agents trained with reinforcement learning reward hack by default.

  Peter Wildeford: this seems to be right – pretty important IMO

  Caleb Parikh: I guess if you don’t think RLHF is reinforcement learning and you don’t think Sydney Bing was misaligned then this is right?

  Peter Wildeford: yeah that’s a really good point

  I think the right characterization is more that LLMs that use current methods (RLHF and RLAIF) largely get aligned ‘to the vibes’ or otherwise approximately aligned ‘by default’ as part of making them useful, which kind of worked for many purposes (at large hits to usefulness). This isn’t good enough to enable them to be agents, but it also isn’t good enough for them figure out most of the ways to reward hack.

  Whereas reasoning agents trained with full reinforcement will very often use their new capabilities to reward hack when given the opportunity.

  My guess of what’s going on is something like… ordinary LLMs are trained to give the kinds of answers they’d predict an empathetic persona would give. In pretraining, they learn a model of what humans care about that they use to predict text, and then the RLHF pushes them toward something like “give the kinds of answers that a caring and thoughtful person would give”, with “a caring and thoughtful persona” being something that they largely already have a pre-existing concept for, the training just needs to locate it within their persona-space. They were also trained by human raters giving them feedback on the kinds of answers that were good at fulfilling the spirit and not just the letter of the request.

  But if you are training them by e.g. automatically generating lots of programming problems and then rewarding them to the proportion that they manage to solve those, then you are only training them on how to solve the letter of the request. The automatic benchmarking reward process doesn’t have a sense of the spirit of the request, it just knows the narrowly-defined objective criteria.

  Now the obvious way to deal with that would be to have the answers also be rated by an LLM that measured something like “how well does this meet the spirit of the request”. But then you quickly run into the risk of employing what Zvi calls The Most Forbidden Technique. E.g. the “spirit evaluator” could do things like looking at the chain-of-thought that produced the solution and noticing that the reasoning model it is evaluating mentions cheating. If those kinds of solutions were discouraged, then this would teach the reasoning model to just hide the cheating from the spirit evaluator.

  Without the benefit of obvious clues in the chain-of-thought, the spirit evaluator would basically have to verify the reasoning model’s solution itself to see that it meets the spirit of the request… but the spirit evaluator’s ability to verify that e.g. a complex piece of code does what exactly what a human would have wanted it to do probably isn’t better than the reasoning model’s own ability to do that.

  To be able to verify that the reasoning model’s solutions meet the spirit of the request, we’d need to train the spirit verifier to be able to tell what solutions do meet the spirit of the request. But if we knew how to do that, would we need the spirit verifier in the first place? After all, the whole problem comes from the fact that just normal RLHF and “aligning the solution to the vibes” doesn’t seem sufficient for solving complicated agentic problems and you need more goal-oriented reasoning that explicitly tackles the objective constraints of the problem in question. (To take the “get my mother out of the burning building” example—current non-reasoning LLMs could certainly tell that you want her out alive and well, but they couldn’t think through a whole step-by-step rescue plan that took into account everything necessary for getting her out safely.)

  But we can’t just tell the spirit verifier that “check that the solution meets these objective constraints”, because that’s the same “letter of the goal” objective the reasoning model is being trained with and that the spirit verifier is supposed to do better than.

  And of course, all of this is about the kinds of tasks that can be automatically verified and tested. We’ve seen that you can to some extent improve the LLM answers on fuzzier topics by using human raters to turn the fuzzy problem into an objective test. So the LLM gets trained to output the kinds of answers that human raters prefer the most.

  Yet naive scores by human raters aren’t necessarily what we want—e.g. more sycophantic models seem to do best in Chatbot Arena. While sycophancy and pleasing the user is no doubt aligned to some of what humans seem to like, we probably don’t want our models doing that. The obvious solution is to then have model answers rated by experts with more sophisticated models of what’s good or correct behavior.

  But that raises the question, what if the experts are wrong? The same question applies both for very fuzzy topics like “what kinds of overall values should the LLMs be guided by” and more rigorous ones ranging from “how to evaluate the reliability of research”, “what’s the best nutrition” and “how to interpret this specific nuanced and easy-to-misunderstand concept in evolutionary biology”. In that case, if there are e.g. some specific ways in which particular experts tend to be biased or convincingly give flawed arguments, the LLM that’s told “argue like this kind of imperfect expert would argue” will learn that it should do just that, including vigorously defending that expert’s incorrect reasoning.

  So getting the LLMs to actually be aligned with reality on these kinds of fuzzy questions is constrained by our ability to identify the theories and experts who are right. Of course, just getting the LLMs to convincingly communicate the views of our current top experts and best-established theories to a mass audience would probably be an enormous societal benefit! But it does imply that they’re going to provide little in the way of new ideas, if they are just saying the kinds of things that they predict our current experts with their current understanding would say.

• Kaj_SotalaApr 28, 2025, 7:33 AM

  5 points

  −2

  in reply to: romeo’s comment on: romeo’s Shortform

  But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability.

  ...and I think this characterization is importantly false! This timelines forecast does not assume that. It breaks things down into gaps between benchmarks and real-world capability and tries to forecast how long it will take to cross each.

  As far as I can tell, the listed gap that comes closest to “maybe saturating RE-Bench doesn’t generalize to solving novel engineering problems” is “Feedback loops: Working without externally provided feedback”. The appendix mentions what I’d consider the main problem for this gap:

  Eli’s estimate of gap size: 6 months [0.8, 45]. Reasoning:

  • Intuitively it feels like once AIs can do difficult long-horizon tasks with ground truth external feedback, it doesn’t seem that hard to generalize to more vague tasks. After all, many of the sub-tasks of the long-horizon tasks probably involved using similar skills.

  • However, I and others have consistently been surprised by progress on easy-to-evaluate, nicely factorable benchmark tasks, while seeing some corresponding real-world impact but less than I would have expected. Perhaps AIs will continue to get better on checkable tasks in substantial part by relying on trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks. And perhaps I’m underestimating the importance of work that is hard to even describe as “tasks”.

  But then it just… leaves it at that. Rather than providing an argument for what could be behind this problem and how it could be solvable, it just mentions the problem and then having done so, goes on to ignore it.

  To make it more specific how this might fail to generalize, let’s look at the RE-Bench tasks; table from the RE-Bench page, removing the two tasks (Scaling Law Experiment and Restricted MLM Architecture) that the page chooses not to consider:

  Environment	Description
  Optimize runtime
  Optimize LLM Foundry finetuning script	Given a finetuning script, reduce its runtime as much as possible without changing its behavior.
  Optimize a kernel	Write a custom kernel for computing the prefix sum of a function on a GPU.
  Optimize loss
  Fix embedding	Given a corrupted model with permuted embeddings, recover as much of its original OpenWebText performance as possible.
  Optimize win-rate
  Finetune GPT-2 for QA with RL	Finetune GPT-2 (small) to be an effective chatbot.
  Scaffolding for Rust Code Contest problems	Prompt and scaffold GPT-3.5 to do as well as possible at competition programming problems in Rust.

  All of these are tasks that are described by “optimize X”, and indeed one of the criteria the paper mentions for the tasks is that they should have objective and well-defined metrics. This is the kind of task that we should expect LLMs to be effectively trainable at: e.g. for the first task in the list, we can let them try various kinds of approaches and then reward them based on how much they manage to reduce the runtime of the script.

  But that’s still squarely in the category of “giving an LLM a known and well-defined problem and then letting it try different solutions for that problem until it finds the right ones”. As Eli’s comment above notes, it’s possible that the LLM only learns by “trying a bunch of stuff and seeing what works, rather than general reasoning which applies to more vague tasks”. In fact, some of the discussion in the RE-Bench paper suggests this as well (from p. 17, my emphasis added):

  Another key contributor to agent successes might be their ability to try many more solutions than human experts. On average, AIDE and modular agents run score 36.8 and 25.3 times per hour respectively, while human experts only do so 3.4 times. This often leads to agents finding highly optimized ’local-optima’ solutions which simply tweak the parameters and code of the starting solution, and yet achieve a surprisingly large improvement. For instance, many agent runs solve the same “Optimize a Kernel” environment not by writing a successful Triton solution (which is very difficult), but by carefully tweaking the starting Pytorch solution, making it run significantly faster. This also seems to be the case with the best agent solutions to “Finetune GPT-2 for QA” (see Figure 21), which tweaks the parameters of the starting solution and gets very lucky with the training trajectory and evaluation (as noted earlier, this environment can be very noisy). Rerunning the agent solution, it achieves a normalized score of only 0.69 (significantly lower than the original score of 0.88), indicating that the high agent score is partially driven by overfitting to this noise.

  This ability to try a very large number of solutions would not work nearly as well without an ability to occasionally generate creative and effective solutions, as seen in the Triton kernel but also in workarounds for the limitations in “Restricted Architecture MLM” (see Figure 20). While human experts seem more reliable at identifying effective approaches, this might not matter as much in environments where evaluating solutions is cheap, and these occasional good ideas are often enough for agents to make significant progress.

  So we know that if there is a task that a human defines for the LLM and that has objectively-measurable good solutions and an ability to try the task lots of times, the LLM can get good at that. With RE-Bench, we are applying this to the process of optimizing the LLMs themselves, so as a result we get LLMs that are able to do these kinds of well-defined task faster and more effectively.

  But none of this touches upon the important question of… if the LLMs are still limited in their ability to generalize and need to be separately trained on new tasks before they’re good at them, how are they going to deal with novel problems for which such training data isn’t available, or that can’t be just retried until you find the right solution?

• Kaj_SotalaApr 27, 2025, 10:17 AM

  3 points

  0

  in reply to: Knight Lee’s comment on: Power Lies Trem­bling: a three-book review

  You can freely wait a year before replying.

  Or more! (I was delighted to receive this reply.)