I’ll let you know. They’re working on open-sourcing their scaffold at the moment.
Julian Bradshaw
Karma: 1,345
Actually another group released VideoGameBench just a few days ago, which includes Pokémon Red among other games. Just a basic scaffold for Red, but that’s fair.
As I wrote in my other post:
Why hasn’t anyone run this as a rigorous benchmark? Probably because it takes multiple weeks to run a single attempt, and moreover a lot of progress comes down to effectively “model RNG”—ex. Gemini just recently failed Safari Zone, a difficult challenge, because its inventory happened to be full and it couldn’t accept an item it needed. And ex. Claude has taken wildly different amounts of time to exit Mt. Moon across attempts depending on how he happens to wander. To really run the benchmark rigorously, you’d need a sample of at least 10 full playthroughs, which would take perhaps a full year, at which point there’d be new models.
I think VideoGameBench has the right approach, which is to give only a basic scaffold (less than described in this post), and when LLMs can make quick, cheap progress through Pokemon Red (not taking weeks and tens of thousands of steps) using that, we’ll know real progress has been made.
Re: biosignatures detected on K2-18b, there’s been a couple popular takes saying this solves the Fermi Paradox: K2-18b is so big (8.6x Earth mass) that you can’t get to orbit, and maybe most life-bearing planets are like that.
This is wrong on several bases:
You can still get to orbit there, it’s just much harder (only 1.3g b/c of larger radius!) (https://x.com/CheerupR/status/1913991596753797383)
It’s much easier for us to detect large planets than small ones (https://exoplanets.nasa.gov/alien-worlds/ways-to-find-a-planet), but we expect small ones to be common too (once detected you can then do atmospheric spectroscopy via JWST to find biosignatures)
Assuming K2-18b does have life actually makes the Fermi paradox worse, because it strongly implies single-celled life is common in the galaxy, removing a potential Great Filter
I would say “agent harness” is a type of “scaffolding”. I used it in this case because it’s how Logan Kilpatrick described it in the tweet I linked at the beginning of the post.
I’m not sure that TAS counts as “AI” since they’re usually compiled by humans, but the “PokeBotBad” you linked is interesting, hadn’t heard of that before. It’s an Any% Glitchless speedrun bot that ran until ~2017 and which managed a solid 1:48:27 time on 2/25/17, which was better than the human world record until 2/12/18. Still, I’d say this is more a programmed “bot” than an AI in the sense we care about.
Anyway, you’re right that the whole reason the Pokémon benchmark exists is because it’s interesting to see how well an untrained LLM can do playing it.
since there’s no obvious reason why they’d be biased in a particular direction
No I’m saying there are obvious reasons why we’d be biased towards truthtelling. I mentioned “spread truth about AI risk” earlier, but also more generally one of our main goals is to get our map to match the territory as a collaborative community project. Lying makes that harder.
Besides sabotaging the community’s map, lying is dangerous to your own map too. As OP notes, to really lie effectively, you have to believe the lie. Well is it said, “If you once tell a lie, the truth is ever after your enemy.”
But to answer your question, it’s not wrong to do consequentialist analysis of lying. Again, I’m not Kantian, tell the guy here to randomly murder you whatever lie you want to survive. But I think there’s a lot of long-term consequences in less thought-experimenty cases that’d be tough to measure.
I’m not convinced SBF had conflicting goals, although it’s hard to know. But more importantly, I don’t agree rationalists “tend not to lie enough”. I’m no Kantian, to be clear, but I believe rationalists ought to aspire to a higher standard of truthtelling than the average person, even if there are some downsides to that.
Have we forgotten Sam Bankman-Fried already? Let’s not renounce virtues in the name of expected value so lightly.
Rationalism was founded partly to disseminate the truth about AI risk. It is hard to spread the truth when you are a known liar, especially when the truth is already difficult to believe.
Huh, seems you are correct. They also apparently are heavily cannibalistic, which might be a good impetus for modeling the intentions of other members of your species…
Oh okay. I agree it’s possible there’s no Great Filter.
Dangit I can’t cease to exist, I have stuff to do this weekend.
But more seriously, I don’t see the point you’re making? I don’t have a particular objection to your discussion of anthropic arguments, but also I don’t understand how it relates to the “what part of evolution/planetary science/sociology/etc. is the Great Filter” scientific question.
I think if you frame it as:
if most individuals exist inside the part of the light cone of an alien civilization, why aren’t we one of them?
Then yes, 1.0 influence and 4.0 influence both count as “part of the light cone”, and so for the related anthropic arguments you could choose to group them together.
But re: anthropic arguments,
Not only am I unable to explain why I’m an observer who doesn’t see aliens
This is where I think I have a different perspective. Granting that anthropic arguments (here, about which observer you are and the odds of that) cause frustration and we don’t want to get into them, I think there is an actual reason why we don’t see aliens—maybe they aren’t there, maybe they’re hiding, maybe it’s all a simulation, whatever—and there’s no strong reason to assume we can’t discover that reason. So, in that non-anthropic sense, in a more scientific inquiry sense, it is possible to explain why I’m an observer who doesn’t see aliens. We just don’t know how to do that yet. The Great Filter is one possible explanation behind the “they aren’t there” answer, and this new information adjusts what we think the filters that would make up the Great Filter might be.
Another way to think about this: suppose we discover that actually science proves life should only arise on 1 in a googol planets. That introduces interesting anthropic considerations about how we ended up as observers on that 1 planet (can’t observe if you don’t exist, yadda yadda). But what I care about here is instead, what scientific discovery proved the odds should be so low? What exactly is the Great Filter that made us so rare?
I agree it’s likely the Great Filter is behind us. And I think you’re technically right, most filters are behind us, and many are far in the past, so the “average expected date of the Great Filter” shifts backward. But, quoting my other comment:
Every other possible filter would gain equally, unless you think this implies that maybe we should discount other evolutionary steps more as well. But either way, that’s still bad on net because we lose probability mass on steps behind us.
So even though the “expected date” shifts backward, the odds for “behind us or ahead of us” shifts toward “ahead of us”.
Let me put it this way: let’s say we have 10 possible filters behind us, and 2 ahead of us. We’ve “lost” one filter behind us due to new information. So, 9 filters behind us gain a little probability mass, 1 filter behind us loses most probability mass, and 2 ahead of us gain a little probability mass. This does increase the odds that the filter is far behind us, since “animal with tool-use intelligence” is a relatively recent filter. But, because “animal with tool-use intelligence” was already behind us and a small amount of that “behind us” probability mass has now shifted to filters ahead of us, the ratio between all past filters and all future filters has adjusted slightly toward future filters.
Interesting thought. I think you have a point about coevolution, but I don’t think it explains away everything in the birds vs. mammals case. How much are birds really competing with mammals vs. other birds/other animals? Mammals compete with lots of animals, why did only birds get smarter? I tend to think intra-niche/genus competition would generate most of the pressure for higher intelligence, and for whatever reason that competition doesn’t seem to lead to huge intelligence gains in most species.
(Re: octopus, cephalopods do have interactions with marine mammals. But also, their intelligence is seemingly different from mammals/birds—strong motor intelligence, but they’re not really very social or cooperative. Hard to compare but I’d put them in a lower tier than the top birds/mammals for the parts of intelligence relevant to the Fermi Paradox.)
In terms of the K-T event, I think it could plausibly qualify as a filter, but asteroid impacts of that size are common enough it can’t be the Great Filter on its own—it doesn’t seem the specific details of the impact (location/timing) are rare enough for that.
Two objections:
Granting that the decision theory that would result from reasoning based on the Fermi Paradox alone is irrational, we’d still want an answer to the question[1] of why we don’t see aliens. If we live in a universe with causes, there ought to be some reason, and I’d like to know the answer.
“why aren’t we born in a civilization which ‘sees’ an old alien civilization” is not indistinguishable from “why aren’t we born in an old [alien] civilization ourselves?” Especially assuming FTL travel limitations hold, as we generally expect, it would be pretty reasonable to expect to see evidence of interstellar civilizations expanding as we looked at galaxies hundreds of millions or billions of lightyears away—some kind of obviously unnatural behavior, such as infrared radiation from Dyson swarms replacing normal starlight in some sector of a galaxy.[2] There should be many more civilizations we can see than civilizations we can contact.
- ^
I’ve seen it argued that the “Fermi Paradox” ought to be called simply the “Fermi Question” instead for reasons like this, and also that Fermi himself seems to have meant it as an honest question, not a paradox. However, it’s better known as the Paradox, and Fermi Question unfortunately collides with Fermi estimation.
- ^
It is technically possible that all interstellar civilizations don’t do anything visible to us—the Dark Forest theory is one variant of this—but that would contradict the “old civilization would contact and absorb ours” part of your reasoning.
Yes. Every other possible filter would gain equally, unless you think this implies that maybe we should discount other evolutionary steps more as well. But either way, that’s still bad on net because we lose probability mass on steps behind us.
Couple takeaways here. First, quoting the article:
By comparing the bird pallium to lizard and mouse palliums, they also found that the neocortex and DVR were built with similar circuitry — however, the neurons that composed those neural circuits were distinct.
“How we end up with similar circuitry was more flexible than I would have expected,” Zaremba said. “You can build the same circuits from different cell types.”
This is a pretty surprising level of convergence for two separate evolutionary pathways to intelligence. Apparently the neural circuits are so similar that when the original seminal paper on bird brains was written in 1969, it just assumed there had to be a common ancestor, and that thinking felt so logical it held for decades afterward.
Obviously, this implies strong convergent pressures for animal intelligence. It’s not obvious to me that artificial intelligence should converge in the same way, not being subject to same pressures all animals face, but we should maybe expect biological aliens to have intelligence more like ours than we’d previously expected.
Speaking of aliens, that’s my second takeaway: if decent-ish (birds like crows/ravens/parrots + mammals) intelligence has evolved twice on Earth, that drops the odds that the “evolve a tool-using animal with intelligence” filter is a strong Fermi Paradox filter. Thus, to explain the Fermi Paradox, we should posit increased odds that the Great Filter is in front of us. (However, my prior for the Great Filter being ahead of humanity is pretty low, we’re too close to AI and the stars—keep in mind that even a paperclipper has not been Filtered, a Great Filter prevents any intelligence from escaping Earth.)
Both the slowdown and race models predict that the future of Humanity is mostly in the hands of the United States—the baked-in disadvantage in chips from existing sanctions on China is crippling within short timelines, and no one else is contending.
So, if the CCP takes this model seriously, they should probably blockade Taiwan tomorrow? It’s the only fast way to equalize chip access over the next few years. They’d have to weigh the risks against the chance that timelines are long enough for their homegrown chip production to catch up, but there seems to be a compelling argument for a blockade now, especially considering the US has unusually tense relations with its allies at the moment.
China doesn’t need to perform a full invasion, just a blockade would be sufficient if you could somehow avoid escalation… though I’m not sure that you could, the US is already taking AI more seriously than China is. (It’s noteworthy that Daniel Kokotajlo’s 2021 prediction had US chip sanctions happening in 2024, when they really happened in 2022.)
Perhaps more AI Safety effort should be going into figuring out a practical method for international cooperation, I worry we’ll face war before we get AIs that can negotiate us out of it as described in the scenarios here.
I’m generally pretty receptive to “adjust the Overton window” arguments, which is why I think it’s good PauseAI exists, but I do think there’s a cost in political capital to saying “I want a Pause, but I am willing to negotiate”. It’s easy for your opponents to cite your public Pause support and then say, “look, they want to destroy America’s main technological advantage over its rivals” or “look, they want to bomb datacenters, they’re unserious”. (yes Pause as typically imagined requires international treaties, the attack lines would probably still work, there was tons of lying in the California SB 1047 fight and we lost in the end)
The political position AI safety has mostly taken instead on US regulation is “we just want some basic reporting and transparency” which is much harder to argue against, achievable, and still pretty valuable.
I can’t say I know for sure this is the right approach to public policy. There’s a reason politics is a dark art, there’s a lot of triangulating between “real” and “public” stances, and it’s not costless to compromise your dedication to the truth like that. But I think it’s part of why there isn’t as much support for PauseAI as you might expect. (the other main part being what 1a3orn says, that PauseAI is on the radical end of opinions in AI safety and it’s natural there’d be a gap between moderates and them)
Yeah it is confusing. You’d think there’s tons of available data on pixelated game screens. Maybe training on it somehow degrades performance on other images?