Julian Bradshaw

Karma: 1,583

Julian Bradshaw Apr 20, 2025, 4:16 PM
2 points
0
in reply to: Edmund Nelson’s comment on: Is Gemini now better than Claude at Pokémon?
I’m not sure that TAS counts as “AI” since they’re usually compiled by humans, but the “PokeBotBad” you linked is interesting, hadn’t heard of that before. It’s an Any% Glitchless speedrun bot that ran until ~2017 and which managed a solid 1:48:27 time on 2/25/17, which was better than the human world record until 2/12/18. Still, I’d say this is more a programmed “bot” than an AI in the sense we care about.
Anyway, you’re right that the whole reason the Pokémon benchmark exists is because it’s interesting to see how well an untrained LLM can do playing it.

Is Gemini now better than Claude at Pokémon?

Julian BradshawApr 19, 2025, 11:34 PM

90 points

12 comments5 min readLW link

Julian Bradshaw Apr 17, 2025, 9:58 PM
10 points
8
in reply to: eva_’s comment on: A Dissent on Honesty
since there’s no obvious reason why they’d be biased in a particular direction
No I’m saying there are obvious reasons why we’d be biased towards truthtelling. I mentioned “spread truth about AI risk” earlier, but also more generally one of our main goals is to get our map to match the territory as a collaborative community project. Lying makes that harder.
Besides sabotaging the community’s map, lying is dangerous to your own map too. As OP notes, to really lie effectively, you have to believe the lie. Well is it said, “If you once tell a lie, the truth is ever after your enemy.”
But to answer your question, it’s not wrong to do consequentialist analysis of lying. Again, I’m not Kantian, tell the guy here to randomly murder you whatever lie you want to survive. But I think there’s a lot of long-term consequences in less thought-experimenty cases that’d be tough to measure.

Julian Bradshaw Apr 17, 2025, 5:45 AM
9 points
8
in reply to: Jiro’s comment on: A Dissent on Honesty
I’m not convinced SBF had conflicting goals, although it’s hard to know. But more importantly, I don’t agree rationalists “tend not to lie enough”. I’m no Kantian, to be clear, but I believe rationalists ought to aspire to a higher standard of truthtelling than the average person, even if there are some downsides to that.

Julian Bradshaw Apr 15, 2025, 9:27 PM
30 points
23
on: A Dissent on Honesty
Have we forgotten Sam Bankman-Fried already? Let’s not renounce virtues in the name of expected value so lightly.

Rationalism was founded partly to disseminate the truth about AI risk. It is hard to spread the truth when you are a known liar, especially when the truth is already difficult to believe.

Julian Bradshaw Apr 10, 2025, 6:14 PM
4 points
0
in reply to: ChristianKl’s comment on: birds and mammals independently evolved intelligence
Huh, seems you are correct. They also apparently are heavily cannibalistic, which might be a good impetus for modeling the intentions of other members of your species…

Julian Bradshaw Apr 10, 2025, 5:48 AM
3 points
0
in reply to: Knight Lee’s comment on: birds and mammals independently evolved intelligence
Oh okay. I agree it’s possible there’s no Great Filter.

Julian Bradshaw Apr 10, 2025, 4:04 AM
3 points
0
in reply to: Knight Lee’s comment on: birds and mammals independently evolved intelligence
Dangit I can’t cease to exist, I have stuff to do this weekend.

But more seriously, I don’t see the point you’re making? I don’t have a particular objection to your discussion of anthropic arguments, but also I don’t understand how it relates to the “what part of evolution/planetary science/sociology/etc. is the Great Filter” scientific question.

Julian Bradshaw Apr 9, 2025, 8:13 PM
3 points
0
in reply to: Knight Lee’s comment on: birds and mammals independently evolved intelligence
I think if you frame it as:
if most individuals exist inside the part of the light cone of an alien civilization, why aren’t we one of them?
Then yes, 1.0 influence and 4.0 influence both count as “part of the light cone”, and so for the related anthropic arguments you could choose to group them together.
But re: anthropic arguments,
Not only am I unable to explain why I’m an observer who doesn’t see aliens
This is where I think I have a different perspective. Granting that anthropic arguments (here, about which observer you are and the odds of that) cause frustration and we don’t want to get into them, I think there is an actual reason why we don’t see aliens—maybe they aren’t there, maybe they’re hiding, maybe it’s all a simulation, whatever—and there’s no strong reason to assume we can’t discover that reason. So, in that non-anthropic sense, in a more scientific inquiry sense, it is possible to explain why I’m an observer who doesn’t see aliens. We just don’t know how to do that yet. The Great Filter is one possible explanation behind the “they aren’t there” answer, and this new information adjusts what we think the filters that would make up the Great Filter might be.
Another way to think about this: suppose we discover that actually science proves life should only arise on 1 in a googol planets. That introduces interesting anthropic considerations about how we ended up as observers on that 1 planet (can’t observe if you don’t exist, yadda yadda). But what I care about here is instead, what scientific discovery proved the odds should be so low? What exactly is the Great Filter that made us so rare?

Julian Bradshaw Apr 9, 2025, 7:50 PM
9 points
0
in reply to: Davidmanheim’s comment on: birds and mammals independently evolved intelligence
I agree it’s likely the Great Filter is behind us. And I think you’re technically right, most filters are behind us, and many are far in the past, so the “average expected date of the Great Filter” shifts backward. But, quoting my other comment:
Every other possible filter would gain equally, unless you think this implies that maybe we should discount other evolutionary steps more as well. But either way, that’s still bad on net because we lose probability mass on steps behind us.
So even though the “expected date” shifts backward, the odds for “behind us or ahead of us” shifts toward “ahead of us”.
Let me put it this way: let’s say we have 10 possible filters behind us, and 2 ahead of us. We’ve “lost” one filter behind us due to new information. So, 9 filters behind us gain a little probability mass, 1 filter behind us loses most probability mass, and 2 ahead of us gain a little probability mass. This does increase the odds that the filter is far behind us, since “animal with tool-use intelligence” is a relatively recent filter. But, because “animal with tool-use intelligence” was already behind us and a small amount of that “behind us” probability mass has now shifted to filters ahead of us, the ratio between all past filters and all future filters has adjusted slightly toward future filters.

Julian Bradshaw Apr 9, 2025, 7:34 PM
6 points
2
in reply to: Yoreth’s comment on: birds and mammals independently evolved intelligence
Interesting thought. I think you have a point about coevolution, but I don’t think it explains away everything in the birds vs. mammals case. How much are birds really competing with mammals vs. other birds/other animals? Mammals compete with lots of animals, why did only birds get smarter? I tend to think intra-niche/genus competition would generate most of the pressure for higher intelligence, and for whatever reason that competition doesn’t seem to lead to huge intelligence gains in most species.
(Re: octopus, cephalopods do have interactions with marine mammals. But also, their intelligence is seemingly different from mammals/birds—strong motor intelligence, but they’re not really very social or cooperative. Hard to compare but I’d put them in a lower tier than the top birds/mammals for the parts of intelligence relevant to the Fermi Paradox.)
In terms of the K-T event, I think it could plausibly qualify as a filter, but asteroid impacts of that size are common enough it can’t be the Great Filter on its own—it doesn’t seem the specific details of the impact (location/timing) are rare enough for that.

Julian Bradshaw Apr 9, 2025, 5:06 AM
3 points
0
in reply to: Knight Lee’s comment on: birds and mammals independently evolved intelligence
Two objections:
1. Granting that the decision theory that would result from reasoning based on the Fermi Paradox alone is irrational, we’d still want an answer to the question^[1] of why we don’t see aliens. If we live in a universe with causes, there ought to be some reason, and I’d like to know the answer.
2. “why aren’t we born in a civilization which ‘sees’ an old alien civilization” is not indistinguishable from “why aren’t we born in an old [alien] civilization ourselves?” Especially assuming FTL travel limitations hold, as we generally expect, it would be pretty reasonable to expect to see evidence of interstellar civilizations expanding as we looked at galaxies hundreds of millions or billions of lightyears away—some kind of obviously unnatural behavior, such as infrared radiation from Dyson swarms replacing normal starlight in some sector of a galaxy.^[2] There should be many more civilizations we can see than civilizations we can contact.
1. ^
  I’ve seen it argued that the “Fermi Paradox” ought to be called simply the “Fermi Question” instead for reasons like this, and also that Fermi himself seems to have meant it as an honest question, not a paradox. However, it’s better known as the Paradox, and Fermi Question unfortunately collides with Fermi estimation.
2. ^
  It is technically possible that all interstellar civilizations don’t do anything visible to us—the Dark Forest theory is one variant of this—but that would contradict the “old civilization would contact and absorb ours” part of your reasoning.

Julian Bradshaw Apr 9, 2025, 4:22 AM
2 points
0
in reply to: Pretentious Penguin’s comment on: birds and mammals independently evolved intelligence
Yes. Every other possible filter would gain equally, unless you think this implies that maybe we should discount other evolutionary steps more as well. But either way, that’s still bad on net because we lose probability mass on steps behind us.

Julian Bradshaw Apr 8, 2025, 8:59 PM
50 points
16
on: birds and mammals independently evolved intelligence
Couple takeaways here. First, quoting the article:
By comparing the bird pallium to lizard and mouse palliums, they also found that the neocortex and DVR were built with similar circuitry — however, the neurons that composed those neural circuits were distinct.
“How we end up with similar circuitry was more flexible than I would have expected,” Zaremba said. “You can build the same circuits from different cell types.”
This is a pretty surprising level of convergence for two separate evolutionary pathways to intelligence. Apparently the neural circuits are so similar that when the original seminal paper on bird brains was written in 1969, it just assumed there had to be a common ancestor, and that thinking felt so logical it held for decades afterward.
Obviously, this implies strong convergent pressures for animal intelligence. It’s not obvious to me that artificial intelligence should converge in the same way, not being subject to same pressures all animals face, but we should maybe expect biological aliens to have intelligence more like ours than we’d previously expected.
Speaking of aliens, that’s my second takeaway: if decent-ish (birds like crows/ravens/parrots + mammals) intelligence has evolved twice on Earth, that drops the odds that the “evolve a tool-using animal with intelligence” filter is a strong Fermi Paradox filter. Thus, to explain the Fermi Paradox, we should posit increased odds that the Great Filter is in front of us. (However, my prior for the Great Filter being ahead of humanity is pretty low, we’re too close to AI and the stars—keep in mind that even a paperclipper has not been Filtered, a Great Filter prevents any intelligence from escaping Earth.)

Julian Bradshaw Apr 3, 2025, 8:40 PM
37 points
17
on: AI 2027: What Superintelligence Looks Like
Both the slowdown and race models predict that the future of Humanity is mostly in the hands of the United States—the baked-in disadvantage in chips from existing sanctions on China is crippling within short timelines, and no one else is contending.
So, if the CCP takes this model seriously, they should probably blockade Taiwan tomorrow? It’s the only fast way to equalize chip access over the next few years. They’d have to weigh the risks against the chance that timelines are long enough for their homegrown chip production to catch up, but there seems to be a compelling argument for a blockade now, especially considering the US has unusually tense relations with its allies at the moment.
China doesn’t need to perform a full invasion, just a blockade would be sufficient if you could somehow avoid escalation… though I’m not sure that you could, the US is already taking AI more seriously than China is. (It’s noteworthy that Daniel Kokotajlo’s 2021 prediction had US chip sanctions happening in 2024, when they really happened in 2022.)
Perhaps more AI Safety effort should be going into figuring out a practical method for international cooperation, I worry we’ll face war before we get AIs that can negotiate us out of it as described in the scenarios here.
What links here?
- AI 2027: Responses by Zvi (Apr 8, 2025, 12:50 PM; 106 points)

Julian Bradshaw Mar 31, 2025, 8:52 PM
3 points
2
in reply to: Davidmanheim’s comment on: Why do many people who care about AI Safety not clearly endorse PauseAI?
I’m generally pretty receptive to “adjust the Overton window” arguments, which is why I think it’s good PauseAI exists, but I do think there’s a cost in political capital to saying “I want a Pause, but I am willing to negotiate”. It’s easy for your opponents to cite your public Pause support and then say, “look, they want to destroy America’s main technological advantage over its rivals” or “look, they want to bomb datacenters, they’re unserious”. (yes Pause as typically imagined requires international treaties, the attack lines would probably still work, there was tons of lying in the California SB 1047 fight and we lost in the end)
The political position AI safety has mostly taken instead on US regulation is “we just want some basic reporting and transparency” which is much harder to argue against, achievable, and still pretty valuable.
I can’t say I know for sure this is the right approach to public policy. There’s a reason politics is a dark art, there’s a lot of triangulating between “real” and “public” stances, and it’s not costless to compromise your dedication to the truth like that. But I think it’s part of why there isn’t as much support for PauseAI as you might expect. (the other main part being what 1a3orn says, that PauseAI is on the radical end of opinions in AI safety and it’s natural there’d be a gap between moderates and them)

Julian Bradshaw Mar 30, 2025, 8:23 PM
4 points
2
on: How I talk to those above me
So I realized Amad’s comment obsession was probably a defense against this dynamic—“I have to say something to my juniors when I see them”.
I think there’s a bit of a trap here where, because Amad is known for always making a comment whenever he ends up next to an employee, if he then doesn’t make a comment next to someone, it feels like a deliberate insult.
That said, I see the same behavior from US tech leadership pretty broadly, so I think the incentive to say something friendly in the elevator is pretty strong to start (norms of equality, first name basis, etc. in tech), and then once you start doing that you have to always do it to avoid insult.

Julian Bradshaw 30 Mar 2025 20:07 UTC
17 points
−2
on: Why do many people who care about AI Safety not clearly endorse PauseAI?
I think the concept of Pausing AI just feels unrealistic at this point.
1. Previous AI safety pause efforts (GPT-2 release delay, 2023 Open Letter calling for a 6 month pause) have come to be seen as false alarms and overreactions
2. Both industry and government are now strongly committed to an AI arms race
3. A lot of the non-AI-Safety opponents of AI want a permanent stop/ban in the fields they care about, not a pause, so it lacks for allies
4. It’s not clear that meaningful technical AI safety work on today’s frontier AI models could have been done before they were invented; therefore a lot of technical AI safety researchers believe we still need to push capabilities further before a pause would truly be useful
PauseAI could gain substantial support if there’s a major AI-caused disaster, so it’s good that some people are keeping the torch lit for that possibility, but supporting it now means burning political capital for little reason. We’d get enough credit for “being right all along” just by having pointed out the risks ahead of time, and we want to influence regulation/industry now, so we shouldn’t make Pause demands that get you thrown out of the room. In an ideal world we’d spend more time understanding current models, though.
What links here?
- Knight Lee's comment on Why do many people who care about AI Safety not clearly endorse PauseAI? by humnrdble (31 Mar 2025 8:32 UTC; 9 points)

Julian Bradshaw 27 Mar 2025 20:11 UTC
28 points
9
on: Tracing the Thoughts of a Large Language Model
Copying over a comment from Chris Olah of Anthropic on Hacker News I thought was good: (along with parent comment)

fpgaminer
> This is powerful evidence that even though models are trained to output one word at a time

I find this oversimplification of LLMs to be frequently poisonous to discussions surrounding them. No user facing LLM today is trained on next token prediction.

olah3
Hi! I lead interpretability research at Anthropic. I also used to do a lot of basic ML pedagogy (https://colah.github.io/). I think this post and its children have some important questions about modern deep learning and how it relates to our present research, and wanted to take the opportunity to try and clarify a few things.

When people talk about models “just predicting the next word”, this is a popularization of the fact that modern LLMs are “autoregressive” models. This actually has two components: an architectural component (the model generates words one at a time), and a loss component (it maximizes probability).
As the parent says, modern LLMs are finetuned with a different loss function after pretraining. This means that in some strict sense they’re no longer autoregressive models – but they do still generate text one word at a time. I think this really is the heart of the “just predicting the next word” critique.
This brings us to a debate which goes back many, many years: what does it mean to predict the next word? Many researchers, including myself, have believed that if you want to predict the next word really well, you need to do a lot more. (And with this paper, we’re able to see this mechanistically!)
Here’s an example, which we didn’t put in the paper: How does Claude answer “What do you call someone who studies the stars?” with “An astronomer”? In order to predict “An” instead of “A”, you need to know that you’re going to say something that starts with a vowel next. So you’re incentivized to figure out one word ahead, and indeed, Claude realizes it’s going to say astronomer and works backwards. This is a kind of very, very small scale planning – but you can see how even just a pure autoregressive model is incentivized to do it.

Julian Bradshaw 20 Mar 2025 23:33 UTC
4 points
0
in reply to: jimrandomh’s comment on: The principle of genomic liberty
Good objection. I think gene editing would be different because it would feel more unfair and insurmountable. That’s probably not rational—the effect size would have to be huge for it to be bigger than existing differences in access to education and healthcare, which are not fair or really surmountable in most cases—but something about other people getting to make their kids “superior” off the bat, inherently, is more galling to our sensibilities. Or at least mine, but I think most people feel the same way.

Julian Bradshaw

Is Gem­ini now bet­ter than Claude at Poké­mon?

Is Gemini now better than Claude at Pokémon?