If you have feedback for me, you can fill out the form at https://w-r.me/feedback .
Or you can email me, at [the second letter of the alphabet]@[my username].net
I do not get any notifications about replies to my comments or posts. I may take a long time to respond to them, or never respond.
fyi for those of you who do math: Some coding agents are now (~barely) able to usefully write Lean proofs, for not-totally-trivial statements.
This matters to me (and might matter to you) due to differences in how easy it is to understand Lean statements + have the Lean compiler check the proofs, compared to correctly hand-verifying proofs in English.
I spent a couple hours this weekend having them prove a thing about graph theory that’s useful for some distributed systems stuff I was thinking about last year. Before going through the process I was only ~80% sure the statement was true; I’m now more like 99%; that last 1% would mostly come from my more-carefully double checking that the Lean statement corresponds to the thing I actually care about.
For this problem at least, Claude was superior to Gemini at actually using the lean compiler and iterating, but Gemini was superior at structuring the code at the top level / figuring out the broad strokes of the proof.
Very highly recommend using agents instead of trying to do this via chat, since the process here clearly required a feedback loop rather than merely spitting out correct proofs.
“Colorless green ideas sleep furiously” is the canonical example of a sentence that is syntactically valid but meaningless. IMO, it’s not actually very good at being meaningless. I think, instead, it is merely a cat coupling. Is it claiming that (A) there are some colorless green ideas that sleep furiously? Or that (B) all ideas that are colorless and green sleep furiously? I think A is false, and B is vacuously true. But both interpretations appear to me to have some meaning / content.
I would be really interested in someone doing an obvious study here, like, “the first time you give Alice a set of choices to elicit a discount schedule, how is that schedule different from the 100th time you give her the same set of choices in the same setting? i.e. does she update to have a tighter implied distribution over hazard rates?” (maybe someone has done this study; if anyone knows about it I’d love a link)
I don’t think I have quite the same sense that the story will have to be in terms like those you’re describing; it seems totally plausible to me that there is a very general / simple / Bayes-compatible story underlying this, since hyperbolic (or at least non-exponential) discounting seems extremely widespread, and neural architecture appears to me to be highly adaptable.
tldr: TIL that someone has ever given a not-(obviously-to-me-)crazy explanation for hyperbolic discounting.
Today I asked ChatGPT, Claude, and Gemini the following question that I’ve had for quite a while:
> I’ve heard that one common quantitative explanation for phenomena like addiction and procrastination is hyperbolic, as opposed to exponential, discounting. Are there clean stories for *why* humans and other animals might end up factored such that they do hyperbolic discounting? It seems like a very potentially “clean” theory, i.e. we value events inversely proportionally to how far away in time they are, but at the same time I don’t know why it might convey an evolutionary advantage compared to exponential discounting (which has a relatively simple explanation like, “on the scale of a year or so, you have a roughly constant probability of e.g. dying or becoming infertile on any given day; so value that’s further in the future should be discounted exponentially based on that probability”)
All three LLMs pointed me to work by Peter Sozou[1] arguing that if you don’t know the hazard rate, but instead have uncertainty about it that you update over time, hyperbolic-looking discounting can fall out from there.
A quick search reveals that Sozou has been mentioned on LW once before, in a comment[2] pointing at a blog post by Jason Collins explaining the idea[3].
In a version of the shutdown resistance paper that’s currently being reviewed (not included in the preprint yet) the following details are included:
> We began our examination of this topic because we had an intuitive expectation that current LLMs might resist shutdown in settings like this one; we did not discover it by sampling uniformly from the space of all possible or realistic tasks. Specifically, we began our exploration by considering several ways to check for the presence of ``instrumentally convergent″ behavior from current LLMs. In addition to shutdown resistance, we considered ways to elicit self-replication or resource acquisition. We then did some exploratory work in each area, and found that shutdown resistance was very easy to elicit in simple settings like the one presented in this paper: The environment we present here has not been substantially modified from our initial working implementation, and the initial prompt we present (Prompt A) differs mainly in that we corrected some typographical errors. All our experiments exploring shutdown resistance, including very nearly all exploratory work and experiments performed while developing the technical environment, are available such that a reader can examine the sequence of experiments that produced the specific prompts presented here.
...
> All of our experimental results and inspect traces are also available, collected in a table at https://shutres.fyi/all-experiments, including nearly all exploratory work, experiments performed while testing our technical environment, and smaller runs of the final prompts. The only results not included in this table were (a) fewer than ten experiments performed during the initial development of the technical environment (with N ⇐ 10 each), or (b) due to accidental misconfigurations in logging the results, which occurred less than five times during development and were followed by running the same experiments again with the configurations corrected.
(tbc this obviously isn’t the same as a commitment to show all negative results, it just seemed moderately relevant and potentially useful)
This take is a bit frustrating to me, because the preprint does discuss Rajamanoharan & Nanda’s result, and in particular when we tried Rajamanoharan & Nanda’s strongest prompt clarification on other models in our initial set, it didn’t in fact bring the rate to zero. Which is not to say that it would be impossible to find a prompt that brings the rate low enough to be entirely undetectable for all models—of course you could find such a prompt if you knew that you needed to look for one.
I’m genuinely sorry to ask this (I think a better version of me wouldn’t need to ask), but would you be willing to be more specific about your critique here? I wasn’t involved with this work but I’m sure I’m at least a bit mindkilled / ideas-as-soldiers-y about it since I work at Palisade, and I think that’s making it hard for me to find, e.g., the thing we’re saying that’s unfair or painting the behavior in an unfair light.
I made a thing that generates strong passwords (>92 bits) that are also easy to remember because they’re rhyming nonsense couplets (that also scan pretty well): https://www.benwr.net/2025/07/16/opensesame.html
Ernyyl ernyyl cbjreshy synfu yvtug. Cbvag vg ng fbzr irel sne njnl zngrevny jubfr genwrpgbel lbh pna cerqvpg. Fbzrobql jnagf gb renfr lbhe qngn? Gbb onq; gur orfg gurl pna qb vf oybpx crbcyr sebz ernqvat vg jura vg’f ersyrpgrq. Nyfb arng cebcregl bs guvf vf gung, vs lbh’er fhssvpvragyl pnershy, vg’f “ernq-bapr”.
Are there any storage media that are basically impossible to destroy/erase? Answer in rot13 ITT.
Note that we made all our code available (https://github.com/PalisadeResearch/shutdown_avoidance/) and it’s pretty easy to run exactly our code but with your prompt, if you’d like to avoid writing your own script. You need a docker runtime and a python environment with the dependencies installed (and it’s easier if you use nix though that’s not required), but if you have those things you just have to modify the conf.py file with your prompt and then do something like run --model openai/o3.
There are lots of reasons that a “survival drive” is something we were interested in testing; one reason is that self-preservation has been suggested as a “convergent instrumental goal”; see https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf for a (fairly dated, but seminal) discussion of this.
There are lots of ways for systems to malfunction, but only a small number of those involve actively / surprisingly disabling a shutdown mechanism without any instruction to do so. If the issue was that it got the math problems wrong, that seems like a run-of-the-mill malfunction; if it rewrites the shutdown program, that seems like a much more interesting and specific failure mode, and the word “sabotage” seems appropriate.
I weak-downvoted this comment; I did so for the following reasons:
(a) “Reasoning models as trained do not follow a typical instruction hierarchy of a chat template.” doesn’t seem true if you mean that they’re not intended to follow such a hierarchy (see the link to the system card in the OP), and if you mean that they don’t follow it despite being meant/designed to, I don’t really see much evidence presented (and don’t have any background knowledge that would suggest) that the “Goal, Outcome, Warning, Context” sequence is “bound to” give consistent results.
(b) The comment seems pretty unclear about what you tested and why; e.g. the post you link as an explanation seems to be mostly about irrelevant things, and I don’t really want to read the whole thing to figure out what you mean to reference in it. I also don’t understand the setting in which you ran this experiment: Were you using the chat interface? If so, there’s an existing hidden system prompt that’s going to change the results. Were you using our code from github but with your prompt substituted? If so, why only run it 15 times for one model? Your own code with the OpenAI API? Same question, and also I’d like more details in order to understand specifically what you tested.
(c) We explain that a “survival drive” is merely one hypothesis, and mostly not one that we feel is a good explanation for this behavior; far from jumping to anthropomorphic conclusions, we came away feeling that this isn’t a good explanation for what we’re seeing. We don’t mention consciousness at all, as we don’t think it’s very relevant here.
I wouldn’t have downvoted if I hadn’t felt that the comment was mostly based around a strawman / semi-hallucinated impression of the OP, or if it had been clearer about what exactly the takeaway is meant to be, or what exactly you tested. I didn’t strong-downvote because I appreciated that you shared the prompt you ran, and in general I appreciate that you put in some effort to be empirical about it.
Right, I think it just seems like doing emotional preparation that matches this description is a kind of earthbender-friendly / earthbender-assuming move, while an airbender-friendly move would be more like “notice and accept that you’d have more fun doing it a different way or doing a different thing; that flailing isn’t actually fun”. The effect is kind of similar, i.e. both earthbenders and airbenders should come away less-clinging-to-something, but the earthbender comes away less-clinging-to “the locally easy and straightforward things will work if I do them enough” while the airbender is less-clinging-to something more like “This is what I’d choose”.
Re the last-resort framing, I’m not sure why I said that exactly; I think it’s related to the vibe I got from the OP: Like, “if you notice that you’re not making progress, what do you do? Well, you could keep flailing or avoidantly doomscrolling, or you could [do the thing I’m suggesting], or you could give up in despair”; I think it feels like a “last resort” because the other realistic options presented are kind of like different kinds of death?
Yeah I can try to say some of them, though my sense of the crucialness here does shift on learning that you mean for this to be about hour-to-week levels of effort. I guess I may as well try to come up with element-bending-flavored ones since I’m in pretty deep on that metaphor here.
The biggest differences in which one I’d recommend as a “default last resort” depend on the person and their strengths rather than the situation.
The thing recommended for the airbender above, i.e. “retreat to a safe distance and consider if there’s anything that seems like a fun challenge instead of a grind”.
For a firebender: “Commit to doing it really hard for about [30 minutes]. See where you get. Wait [2 * 30 minutes] after that, and check if you feel energized or drained. If energized, repeat; if drained, [I don’t know; this is the one I have the least familiarity with]”
For a waterbender: Something like “Do naturalism to it”. “When you make good enough observations, you can’t help but make high-quality inferences” is one of my favorite quotes, and I think it applies especially well in this kind of last-resort setting.
I think I maybe mean to say a slightly different thing than came across, which makes sense because I was leaning heavily into the metaphor rather than trying to be very clear.
I think the triggers are definitely hints in the direction that buckling up might be the right move. Yet I also observe that, when I imagine the median or even 80th percentile person-explicitly-buckling-up-for-something-big, a big part of me wants to shake my head and be like “ah well, it was nice while it lasted” about their chances of doing a hard thing, especially an unusual hard thing.
This part of me is clearly wrong sometimes: My head would have shaken well off my shoulders if someone had told me-transported-to-1986, “Andrew Wiles is going to spend the next 6 years trying in secret solitude to prove Fermat’s Last Theorem”.
But also I think it’s clearly not entirely mistaken about such a person’s odds. If a person is a native “airbender”, i.e. they are deeply familiar with the “taking their situation lightly” stance, I don’t think I want to recommend that they take a sense of “welp, I guess I have to finally buckle up for this one” at face value, especially in the context of it being a last-resort for one of their most challenging projects or goals. It feels to me like such a person is more likely to succeed by (a) deciding to stop flailing, (b) retreating to a safe distance, and (c) reevaluating whether this is the path they really want to follow, while in connection with their sense of fun.
Sure; if it’s not obvious they’re from the universe of Avatar: The Last Airbender.
Earthbending is substantially about: facing things head-on, “just getting it done”, “buckling down” (though I suppose this can be different than “buckling up”), being unyielding, orienting around “grit”.
Waterbending is substantially about: Being flexible, responsive to the environment, and careful.
Airbending is substantially about: Freedom of movement and action, using an opponent’s strength against them (which in PvE looks more like “doing what’s easy and/or fun”), speed.
The other one is firebending, but I didn’t reference it and I don’t really understand it well enough to put it in the same terms; still, my best gloss attempt is that it’s about focused and kinda bursty / lower-endurance intensity.
In the universe there are (usually) only these four elements, and most benders are pure specialists / only physically capable of learning their particular kind of bending; the main character (the avatar) is the only person who can / has to learn all four. In my analogy these elements don’t really cover the space of motivational structures that well, and anyway people don’t have to be specialists.
You don’t really mention <a thing that I think is extremely crucial> in this domain, which is that you do not have to (metaphorically) be an earthbender about everything. Other types of bending also exist. If you are not a native earthbender, you might be able to learn to do it (the real world does not have only one Chosen One who can bend all the elements), but as a meta-waterbender I personally recommend first looking around carefully, and trying to figure out how the most successful benders of your native element are doing it.
You do seem to see earthbending as maybe a “last resort” rather than the only way to do things, but it’s not obvious to me that it’s the correct last resort for everyone. The last resort of a successful airbender is probably more like “take even more steps back to see if there are any easier but more oblique approaches to this summit, or even other summits you’d actually rather climb”;
This part seems right but why shouldn’t I think that the 95th or 99th percentile intervention within a bucket is also >>10x as good as the median intervention in that bucket?