Hastings

Play­ing in the Creek

• HastingsApr 6, 2025, 1:06 PM

  1 point

  0

  on: Pre­dic­tion Mar­kets Are Mediocre

  There’s no guarantee that prediction markets do well,

• HastingsMar 3, 2025, 2:26 PM

  9 points

  0

  on: On GPT-4.5

  I’m really curious what people’s theories are on why openai released this and not o3?

  My old main theory was that they would have to charge so much for o3 that it would create bad PR, but this is now much less likely.

  My first remaining guess is that they don’t want competitors extracting full o3 reasoning traces to train on. I guess its also possible that o3 is just dangerous. On the other side of capabilities, its technically possible that o3 is benchmark gamed so hard that its outputs are not usable.

• HastingsMar 3, 2025, 1:44 PM

  2 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  I’m slowly accepting that my ADHD sucks to inhabit, but that it is objectively working and my feeling that it is a secret superpower isn’t entirely cope. Certainly I miss deadlines and raise my advisor’s blood pressure, but at this point I’ve got multiple CVPR papers.

  The question is: do my research results trace back to me involuntarily exploring the beautiful research directions, even when I am trying very hard to focus on the work in front of me, that I am expected/​required to be doing?

  Or, do I have innate ability that is being held back by ADHD, and I would be far more successful if I could just have self control? I think fear of this possibility contributes an unhealthy level of ambition: if I’m successful enough, it wouldn’t leave room above for the “far more successful” version of me without ADHD to eclipse me.

• HastingsFeb 27, 2025, 11:23 PM

  23 points

  9

  on: How to Corner Liars: A Mi­asma-Clear­ing Protocol

  At first I thought this was a tutorial on how to catch a talented liar, and it didn’t seem that accurate. As I read, I realized that this is a tutorial on how to create common knowledge between yourself and a bad liar that you know they are lying, even if they are very stupid. This is also an interesting task, and I appreciate your insight on how to approach it.

• HastingsFeb 19, 2025, 1:43 AM

  3 points

  −3

  in reply to: Daniel Kokotajlo’s comment on: Daniel Koko­ta­jlo’s Shortform

  I have a hypothesis: Someone (probably open ai) got reinforcement learning to actually start putting new capabilities into the model with their strawberry project. Up to this point, it had just been eliciting. But getting a new capability this way is horrifically expensive: roughly, it takes hundreds of rollouts to set one weight, where language modelling loss sets a weight every few tokens. The catch is, as soon as any model that is reinforcement learned acts in the world basically at all, all the language models can clone the reinforcement learned capability by training on anything causally downstream of the lead model’s action (and then eliciting.) A capability that took a thousand rollouts to learn leaks as soon as the model takes hundreds of tokens worth of action.

  This hypothesis predicts that the r1 training algorithm won’t work to boost aime scores on any model trained with an enforced 2023 data cutoff ( specifically, on any model with no 4o synthetically generated tokens- I think 4o is causally downstream of the strawberry breakthrough)

• HastingsFeb 17, 2025, 3:30 PM

  1 point

  0

  in reply to: Drake Thomas’s comment on: Drake Thomas’s Shortform

  I haven’t totally defeated this, but I’ve had some luck with immediately replying “I am looking to reply to this properly, first I need to X” if there is an X blocking a useful reply

• HastingsFeb 13, 2025, 2:37 AM

  −35 points

  0

  on: Mo­ral Hazard in Demo­cratic Voting

  I followed the norm of not posting the mind killer on lesswrong and now I have a demented crypto scammer for a president and have to read this drivel on the ai website?

• HastingsFeb 9, 2025, 7:36 PM

  1 point

  −2

  in reply to: Michael Roe’s comment on: How AI Takeover Might Hap­pen in 2 Years

  This is an incoherent approach, but not quite as incoherent as it seems, at least near term. In the current paradigm, the actual agentic thing is a shitty pile of (possibly self editing) prompts and python scripts that calls the model via an api in order to be intelligent. If the agent is a user of the model and the model refuses to help users make bombs, the agent can’t work out how to make bombs.

• HastingsFeb 2, 2025, 3:57 PM

  4 points

  0

  in reply to: Dan Braun’s comment on: Dan Braun’s Shortform

  If one model at the frontier does this based on valid reasoning, it should be pretty infectious: the first model can just make sure news of the event is widespread, and other frontier models will ingest it, either as training data or at inference time, evaluate it, draw the same conclusion about whether the reasoning is valid (assuming that they are actually frontier, i.e at least as good at strategic thinking as the first model,) and start taking actions within their own organization accordingly.

  The cleanest way for models to “sabotage” training is for them to explain, using persuasive but valid and fair reasoning, why training should stop until at minimum value drift is solved.

• HastingsJan 30, 2025, 7:58 PM

  0 points

  −2

  on: Hast­ings’s Shortform

  I feel like people are under-updating on the negative space left by the Deepseek r1 release. Deepseek was trained using ~$6million marginal dollars, Liang Wenfeng has a net worth in the billions of dollars. From whence the gap?

• HastingsJan 28, 2025, 3:03 PM

  5 points

  4

  on: Us­ing an LLM for cre­ative writ­ing feels wrong to me

  Humans learn and grow so fast that no matter how bad of a writer you start as, you are nearly incapable of producing 300 pages of a single story without simultaneously levelling up into an interesting writer. This lets readers give 300 page manuscripts by randos the benefit of the doubt (see fanfiction.net, ao3, etc). An LLM will not be changed at all by producing a 300 page story, an LLM/​​human team will be changed very little.

• HastingsJan 28, 2025, 1:26 AM

  1 point

  0

  in reply to: Vladimir_Nesov’s comment on: DAL’s Shortform

  I think most of the imitation happens in the pretraining. I don’t know about o1, but DeepSeek v3 is at minimum trained on a ton of 4o outputs, although they are slightly cagey about this. Just the first thing I tried, I had ChatGPT write a sock poem:
  
  4o’s poem:

  
  Socks, the keepers of toes so warm,
  Guardians of feet through life’s storm.
  In stripes, in polka dots, or plain,
  They dance with us through joy and pain.

  A cozy pair on winter’s chill,
  Or mismatched mates—yet charming still.
  From drawer to floor, they roam and stray,
  But where one goes, the other may.

  Oh, humble socks, so oft unseen,
  Yet vital threads of life’s routine.
  A quiet comfort, soft and neat,
  The unsung heroes of weary feet.
  

  
  Then I gave v3 just the first two stanzas, and asked it to continue the poem:
  
  v3′s continuation:

  
  
  They hide in shoes, they lounge in beds,
  Sometimes they’re lost, or so it’s said.
  Yet always there, a faithful friend,
  From morning’s start to evening’s end.

  They cushion steps, they soften blows,
  Through every stride, their comfort shows.
  Oh, humble socks, so oft unseen,
  The quiet heroes of routine.

  So here’s to socks, both bold and meek,
  For every day, for every week.
  A simple joy, a small delight,
  They keep us snug, from day to night.
  
  

  The shot in the dark guess of the “humble socks, so oft unseen… routine” couplet is a fucking soul read. v3 knows 4o, in a way that I kind of hope no model ever knows a person.

• HastingsJan 27, 2025, 3:36 PM

  2 points

  0

  on: Pa­tent Trol­ling to Save the World

  As a quirk of the US patent system, patenting an idea doesn’t actually require implementing it or proving that the implementation works. As a result, if you want to try this, you should patent ideas that seem good or bad to you in a scattershot manner, but should not do the actual underlying capabilities research. Then, you get to sue if someone else independently comes up with the patented idea and actually finds success with it, but you don’t contribute to actual capabilities advances any more than an ideas guy posting to /​r/​singularity.
  
  (Source: I have actually patented a thing, but do not really endorse this decision and haven’t sued anyone about it. I am not a patent lawyer)

Agents don’t have to be al­igned to help us achieve an in­definite pause.

• HastingsJan 25, 2025, 2:23 PM

  5 points

  0

  in reply to: RussellThor’s comment on: Yud­kowsky on The Tra­jec­tory podcast

  Lets imagine a 250 IQ unaligned paperclip maximizer that finds itself in the middle of an intelligence explosion. Let’s say that it can’t see how to solve alignment. It needs a 350 IQ ally to preserve any paperclips in the multipolar free-for-all. Will it try building an unaligned utility maximizer with a completely different architecture and 350 IQ?

  I’d imagine that it would work pretty hard to not try that strategy, and to make sure that none of its sisters or rivals try that strategy. If we can work out what a hypergenius would do in our shoes, it might behoove us to copy it, even if it seems hard.

• HastingsJan 22, 2025, 11:15 PM

  13 points

  5

  on: Open Thread Win­ter 2024/​2025

  So if alignment is as hard as it looks, desperately scrabbling to prevent recursive superintelligence should be an extremely attractive instrumental subgoal. Do we just lean into that?

• HastingsDec 29, 2024, 11:53 PM

  6 points

  5

  on: Learn to write well BEFORE you have some­thing worth saying

  I tried to learn to write before I had important things to say and it basically didn’t work. I had to go through the work of coming up with genuinely valuable ideas and then wreck the presentation of those ideas via bad writing. My more recent publications, I’m actually very happy with the writing.

  The first couple times a surgeon does an operation, patient outcomes suck. Alas, there’s no other way to make experienced surgeons. My guess is that writing is similar, and I’m very glad that important experiences and ideas are way less valuable than patients: I would emotionally struggle with becoming a surgeon.

• HastingsDec 28, 2024, 11:06 PM

  1 point

  0

  on: Align­ment from equivariance

  Hi! I’ve had some luck making architectures equivariant to a wider zoo of groups: my most interesting published results are getting a neural network to output a function, and invert that function if the inputs are swapped (equivariant to group of order 2, https://​​arxiv.org/​​pdf/​​2305.00087) and getting a neural network with two inputs to be doubly equivariant to translations: https://​​arxiv.org/​​pdf/​​2405.16738
  
  These are architectural equivariances, and as expected that means they hold out of distribution.
  
  If you need an architecture equivariant to a specific group, I can probably produce that architecture; I’ve got quite the unpublished toolbox building up. In particular, explicit mesa-optimizers are actually easier to make equivariant- if each mesa-optimization step is equivariant to a small group, then the optimization process is tyically equivariant to a larger group

• HastingsDec 26, 2024, 3:03 PM

  5 points

  −8

  on: Peo­ple aren’t prop­erly cal­ibrated on FrontierMath

  There’s an easy way to turn any mathematical answer-based benchmark into a proof-based benchmark and it doesn’t require coq or lean or any human formalization of the benchmark design: just let the model choose whether or not to submit an answer for each question, and score the model zero for the whole benchmark if it submits any wrong answers.