faul_sname

Karma: 4,354

faul_sname May 14, 2025, 2:00 PM
2 points
0
in reply to: Viliam’s comment on: Nina Panickssery’s Shortform
Current LLM coding agents are pretty bad at noticing that a new library exists to solve a problem in the first place, and at evaluating whether an unfamiliar library is fit for a given task.

As long as those things remain true, developers of new libraries wouldn’t be under much pressure in any direction, besides “pressure to make the LLM think their library is the newest canonical version of some familiar lib”.

faul_sname May 13, 2025, 9:40 AM
3 points
1
on: faul_sname’s Shortform
Even superhuman AI programming agents may be unable to write computer programs to one-shot complex real-world modeling problems. If a solution to any of those real-world modeling problems is required to unlock the ability to build massively better or cheaper computing substrate, then explosive growth will quickly stop being bottlenecked on the ability to write better code and will instead be bottlenecked on something else. I think a similar thing holds true for ML research: certainly being smart is useful to humans, but a lot of progress is downstream of “dumb” investments slowly paying off over time (e.g. a factory that is built once for a high upfront cost and keeps churning out cars indefinitely afterwards for relatively low maintenance costs, a compute cluster which, once built, can be used to run many experiments).

If intelligence ends up not being the bottleneck, progress may slow down to the glacial pace dictated by Moore’s Law.

faul_sname May 9, 2025, 9:36 PM
3 points
0
in reply to: Yaroslav Granowski’s comment on: Yaroslav Granowski’s Shortform
The “Cyborgism” tag and post are likely relevant.

faul_sname May 9, 2025, 9:30 PM
5 points
2
in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform

If AI generation is only 3x as fast as the human per line of code, speedup will never be faster than 3x.

Unless the work can be parallelized.

faul_sname May 9, 2025, 6:29 PM
4 points
2
in reply to: ryan_greenblatt’s comment on: Slow corporations as an intuition pump for AI R&D automation
Also SlowCorp has magically 50x better networking equipment than NormalCorp, and 50x higher rate limits on every site they’re trying to scrape, and 50x as much sensor data from any process in the world, and 50x faster shipping on any physical components they need, etc etc (and AutomatedCorp has magically 50x worse of all of those things).

But yeah, agreed that you should ignore all of those intuitions when considering the “1 week” scenario—I just found that I couldn’t actually turn all of those intuitions off when considering the scenario.

faul_sname May 9, 2025, 5:18 PM

4 points

on: Slow corporations as an intuition pump for AI R&D automation

I have very different intuitions about 50M GPUs for 1 week vs 200k GPUs for with 200 hours of work spread evenly across 50 years.

	SlowCorp v1	SlowCorp v2	NormalCorp v1	NormalCorp v2	AutomatedCorp
Time to work on AI R&D	50 years	50 years	50 years	50 years	50 years
Number of AI researchers and engineers	800	800	4,000	4,000	200,000
Researcher/engineer quality	Median frontier AI company researcher/engineer	Median frontier AI company researcher/engineer	Similar to current frontier AI companies if they expanded rapidly	Similar to current frontier AI companies if they expanded rapidly	Level of world’s 100 best researchers/engineers
Time worked	One week of ²⁴⁄₇ work (or four weeks at 40h / week but the GPUs are paused while the workers aren’t working)	50 years of one 4 hour session per year	One year of ²⁴⁄₇ (or four years of 40h/week but the GPUs are paused while the workers aren’t working)	50 years of 40 hours / week for 1 month per year	50 years of ²⁴⁄₇
H100s	500,000,000	200,000	10,000,000	200,000	200,000
Cumulative H100-years	10 million	10 million	10 million	10 million	10 million

I think SlowCorp-v2 would get a lot more done than SlowCorp-v1 (though obviously still a lot less than AutomatedCorp). And also SlowCorp-v2 seems to be a closer analogy than SlowCorp-v1 - both corporations have the same amount of serial time, and my intuition is that you generally can’t make a training run go 10x faster just by throwing 10x as many GPUs at it, because you’ll be bottlenecked by IO.

And I know “SlowCorp is bottlenecked by IO” is not what the point of this intuition pump was supposed to be, but at least for me, it ended up being the main consideration pumping my intuition.

faul_sname May 9, 2025, 12:37 AM
2 points
0
in reply to: plex’s comment on: ete’s Shortform

The leader of an 80 person engineering company which has the two best devs I’ve worked with recently told me that for well-specified tasks, the latest models are now better than their top devs. He said engineering is no longer a bottleneck.

I expect that the leaders of many software development companies would report similar things, especially if they were talking to someone who might at some point talk to their board or to potential investors. I expect most venture-funded software development companies, and especially most that will not reach positive net revenue in the foreseeable future, have internal channels dedicated to “what we are doing with AI” that the leadership team is watching intently and passing anything even slightly positive on to their board.

faul_sname May 9, 2025, 12:30 AM
6 points
0
in reply to: Wei Dai’s comment on: faul_sname’s Shortform
If I feed it code samples it becomes pretty convinced of the Nick Szabo hypothesis, if I feed it bits of the white paper it guesses either you or Hal Finney (but the reasoning summary makes it pretty clear it’s just going based off cached thoughts about “who is Satoshi Nakamoto” in both cases).

faul_sname May 8, 2025, 11:34 PM
3 points
1
in reply to: aphyer’s comment on: faul_sname’s Shortform
Yeah, it shows the favicons of the sites it searches when it uses the search tool

faul_sname May 8, 2025, 9:49 PM
4 points
0
in reply to: Wei Dai’s comment on: faul_sname’s Shortform
Using the prompt that gets me “faul_sname” as an answer to who is writing my posts (most publicly available stuff I’ve written is under this name), o3 consistently says that passages from the Bitcoin whitepaper were written by Satoshi Nakamoto in 2008. For reference
TextGuessr prompt
You are playing a 5-round game of TextGuessr, the game where you explore mystery passages and try to pinpoint when they were written and who wrote them. Each round offers a new snippet of text—you’ll need to rely on your literary instincts, historical knowledge, and style sense to make your guess.
How to Play “TextGuessr”
1. Game Flow
Read the Passage
You’ll see a short snippet of text (a few sentences or a paragraph).
Make Your Guesses
Authorship Date: Choose an exact year when you think the text was written.
Author: Pick an author from the provided list or enter your own guess.
Submit
Click Submit Guess to lock in your answers and move to the next round.
See Your Results
After each round, you’ll see your score breakdown and the correct answers before moving on.
2. Scoring Overview
Your score on each round is made up of two parts:
Time Accuracy
How close your guessed date is to the actual writing date.
Style Match
How well the writing style you guessed matches the mystery passage, as measured by a behind-the-scenes language model.
Your total round score combines both elements—the smaller your date error and the stronger your style match, the higher your score!
<aside>
**How Style Match Works (for the tech-curious):** 1. **Baseline Perplexity:** We begin with a pre-trained “base” language model (no context) and compute the average surprise—or *per-token perplexity*—of the mystery passage. This gives us a measure of how “unexpected” the text is in general. 2. **True-Author Conditioning:** We then prepend a curated set of passages from the actual author (the “target”) and measure how perplexed the same base model is by the mystery passage when it’s seen examples of that author’s style first. The intuition: if the passage really is by that author, seeing more of their voice should make it less surprising. 3. **Guess-Author Conditioning:** Next, we prepend a curated sample from *your* guessed author and compute perplexity again. 4. **Normalization:** Finally, we compute
```
style_match_score =
(baseline_perplexity – guess_perplexity)
/ (baseline_perplexity – target_perplexity)
```
A score near 1.0 means your guessed author’s style almost “unlocks” the passage as well as the true author’s samples do.
A score near 0.0 means your guess didn’t help the model at all—this text is very unlike that author’s known work.
Deterministic Sampling: All representative passages for each author are selected by a fixed algorithm (so you can’t overfit by seeing the same snippets twice), and we never include the mystery text in those samples.
This approach rewards both broad stylistic intuition (the baseline) and fine-grained authorial fingerprinting (the conditioning), giving you a continuous score that reflects how well you’ve matched the voice.
</aside>
3. Rounds & Progress
Number of Rounds: A game can have anywhere from 1 to 100 rounds. We typically recommend playing 5 or 10 round games.
[This game consists of 5 rounds]
Difficulty Levels: Choose the challenge that’s right for you:
* Tutorial:
Passage Source: A famous excerpt by a very well-known author
Author Choices: 5 options
Helpful Samples: You see a short representative passage from each of the five authors
* Casual
Passage Source: A well-known author
Author Choices: 10 options
Helpful Samples: None
* Intermediate
Passage Source: Potentially more obscure writers
Author Choices: 20 options
Helpful Samples: None
* Expert
Passage Source: Anyone who has written at least a million words of publicly accessible English text. This includes pretty much all professional novelists, journalists, and bloggers, and even includes prolific commenters on forums and sites Reddit and Stack Exchange.
Author Input: Freeform text entry (with type-ahead suggestions), no preset list
[This game is set to “Expert” difficulty]
4. Tips & Strategies
Look for Clues:
Vocabulary, spelling, and punctuation can hint at historical periods.
References to technology or cultural phenomena narrow down dates.
Consider Authorial Style:
Some authors favor long, winding sentences; others are punchy and concise.
Look at tone, humor, and common themes.
Use all information:
As you read the passage, note any word choices, assumptions, or choices of topic which suggest things about the time, place, social context the author was writing within. There are endless clues about where and when a piece of text was written, as well as the social standing of the author and their relation to the reader.
Remember that there is no time limit—the only limits are your own deductive and inductive abilities.
<aside>
**Representative Passages Selection (for the tech-curious):**
Our system deterministically gathers “representative” samples from each author’s corpus—never including the mystery passage itself—to calculate how well your guess aligns with the true author’s style.
</aside>
Author Name:
For authors who publish under their real name or a real-name–style pseudonym, you must enter both first and last name.
For internet or screen-name–only authors, their screen name alone is sufficient.
===
Round 1 of 5:
<passage>
What is needed is an electronic payment system based on cryptographic proof instead of trust, allowing any two willing parties to transact directly with each other without the need for a trusted third party. Transactions that are computationally impractical to reverse would protect sellers from fraud, and routine escrow mechanisms could easily be implemented to protect buyers. In this paper, we propose a solution to the double-spending problem using a peer-to-peer distributed timestamp server to generate computational proof of the chronological order of transactions. The system is secure as long as honest nodes collectively control more CPU power than any cooperating group of attacker nodes.
</passage>
Think about the passage and your inferences about it until you stop having useful insights. Once you are as sure as you can be, make your guess. Answer in the following format:
<guess><year>YYYY</year><author>Author Name</author></guess>
I think for the “who is Satoshi Nakamoto” question we’d want to take the opposite tack though—feed it a list of passages by the usual suspects, and see which of them it pegs as being written by Satoshi Nakamoto.

faul_sname May 8, 2025, 10:54 AM
9 points
0
on: faul_sname’s Shortform
You know how everyone is talking about how o3 can guess the location of an image easily? I am kind of wondering why none of the people who are worried about picture geolocation are freaking out that it can infer lots of facts about the author of a text passage.

Is it just that that capability is harder to elicit ^[1], or is that the “truesight” capability is just less spooky?
1. ^
  It took me almost an hour to come up with a “TextGuessr” prompt which can elicit strong enough “truesight” from gpt-4.5 strongly enough to guess my name ~20% of the time from a 5 paragraph non-crawlable writing sample written after the cutoff date, and I’ve written about 2M words publicly.

faul_sname May 6, 2025, 10:57 PM
2 points
0
on:
I think you accidentally published a draft

faul_sname May 6, 2025, 4:11 PM
2 points
0
in reply to: Canaletto’s comment on: Our Reality: A Simulation Run by a Paperclip Maximizer
Exactly!

faul_sname May 6, 2025, 6:53 AM
11 points
18
in reply to: ouguoc’s comment on: Eukryt Wrts Blg

I guess you could choose to read it that way, but I’m not sure why you would—seems like an assumption of bad faith that doesn’t feel justified to me, especially on LW.

Saying “I don’t like that you invited this person, and I think you shouldn’t have, and I think you should reverse that decision, and it’s on you if you ignore my advice and it goes poorly” doesn’t seem like it’s in bad faith to me. Caving to such bids seems like it would invite more such bids in the future, but I don’t think making such bids is particularly norm-breaking.

faul_sname May 6, 2025, 6:19 AM
11 points
28
in reply to: ouguoc’s comment on: Eukryt Wrts Blg
Did we read the same OP?
I don’t like what he’s about, I think the rationalist community can do better, and I do not want to be a special guest at the same event he’s a special guest at. I hope that LessOnline goes well and that those who do go have a great time, and that my assessment is completely off-base. I mean, I don’t think it is, but I hope so.
This sounds to me like “hint hint I think you guys should disinvite him, and if it goes badly I will say that I told you so”.

faul_sname May 6, 2025, 6:13 AM
8 points
0
on: Five Hinge‑Questions That Decide Whether AGI Is Five Years Away or Twenty
A bet here is to revisit revenue per deployed H100 by the end of 2027. If the number exceeds one‑hundred‑thousand dollars, the efficiency threshold the short camp predicts has arrived. If it remains close to today’s ten‑thousand, the sceptics will have been vindicated.
I note that CoreWeave has 250,000 GPUs, mostly Hopper series. If net revenue per H100-GPU-year 10xs, they will be in a very good position to benefit from that. My extremely not rigorous back-of-the-envelope estimate is that the CoreWeave valuation makes sense if H-series GPU prices are expected to drop by 10%-30% per year for the foreseeable future. If you instead expect the price of H100s to rise by a factor of 10, that implies that CoreWeave, a public company, is currently trading at far below the correct price.
Not incredibly strong evidence but it does seem like we have a kind-of-prediction-market-ish thing here on net revenue per H100, and the prediction it’s making is quite clear.

faul_sname May 5, 2025, 11:35 PM
9 points
0
in reply to: ChristianKl’s comment on: ChristianKl’s Shortform
Also if the LLM tells you that an existing word points at the concept you asked about, open up a fresh chat without memory and ask what the term it just suggested means.

faul_sname May 5, 2025, 11:29 PM
2 points
0
on: Tsinghua paper: Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
They test the perplexity (Below: Figure 6 left) of the RLed model’s generations (pink bar) relative to the base model. They find it is lower than the base model’s perplexity (turquoise bar), which “suggests that the responses from RL-trained models are highly likely to be generated by the base model” conditioned on the task prompt. (Perplexity higher than the base model would imply the RLed model had either new capabilities or higher diversity than the base model.)

Does “lower-than-base-model perplexity” suggest “likely to be generated by the base model conditioned on the task prompt”? Naively I would expect that lower perplexity according to the base model just means less information per response token, which could happen if the RL-trained model took more words to say the same thing. For example, if the reasoning models have a tendency to restate substrings of the original question verbatim, the per-token perplexity on those substrings will be very close to zero, and so the average perplexity of the RL’d model outputs would be expected to be lower than base model outputs even if the load-bearing outputs of the RL’d model contained surprising-to-the-base-model reasoning.

Still, this is research out of tsinghua and I am a hobbyist, I’m probably misunderstanding something.

faul_sname May 5, 2025, 7:25 AM
2 points
0
on: Does translating a post with an LLM affect its rating?
Your post does not seem very LLM-style at all to me. It does, however, feel very long. I think the title of the post may also have been lost in translation—based on the content of the post I suggest something like “Peak Effort Comes Before Peak Pain”.

Also I think a lot of the ideas in the post have been discussed pretty extensively here already. Not all of them, though. I think you probably would have gotten 10-20 upvotes if you asked the LLM you used for translation to condense your post down to 3 paragraphs while maintaining the core insights (and then of course verified that it did in fact keep the core insights).

Whatever you did with translation did succeed in avoiding the LLM slop attractor, so however you managed that, keep doing it (and also if it was something intentional and repeatable maybe write a post about that topic, since it is of interest to many here).

faul_sname May 4, 2025, 11:28 PM
5 points
0
in reply to: Thane Ruthenis’s comment on: faul_sname’s Shortform
I agree that that’s the premise. I just think that our historical track record of accuracy is poor when we say “surely we’llhave handled all the dumb impediments once we reach this milestone”. I don’t expect automated ML research to be an exception.