johnswentworth

• johnswentworth 6 Mar 2025 19:30 UTC

  5 points

  3

  in reply to: Thomas Kwa’s comment on: A Bear Case: My Pre­dic­tions Re­gard­ing AI Progress

  I think you should address Thane’s concrete example:

  For example: “fully-connected neural networks → transformers” definitely wasn’t continuous.

  That seems to me a pretty damn solid knock-down counterargument. There were no continuous language model scaling laws before the transformer architecture, and not for lack of people trying to make language nets.

• johnswentworth 6 Mar 2025 19:16 UTC

  5 points

  0

  in reply to: Donald Hobson’s comment on: The Case Against AI Con­trol Research

  If that AI produces slop, it should be pretty explicitly aware that it’s producing slop.

  This part seems false.

  As a concrete example, consider a very strong base LLM. By assumption, there exists some prompt such that the LLM will output basically the same alignment research you would. But with some other prompt, it produces slop, because it accurately predicts what lots of not-very-competent humans would produce. And when producing the sort of slop which not-very-competent humans produce, there’s no particular reason for it to explicitly think about what a more competent human would produce. There’s no particular reason for it to explicitly think “hmm, there probably exist more competent humans who would produce different text than this”. It’s just thinking about what token would come next, emulating the thinking of low-competence humans, without particularly thinking about more-competent humans at all.

  How many of these failure modes still happen when there is an AI at least as smart as you, that is aware of these failure modes and actively trying to prevent them?

  All of these failure modes apply when the AI is at least as smart as you and “aware of these failure modes” in some sense. It’s the “actively trying to prevent them” part which is key. Why would the AI actively try to prevent them? Would actively trying to prevent them give lower perplexity or higher reward or a more compressible policy? Answer: no, trying to prevent them would not give lower perplexity or higher reward or a more compressible policy.

• johnswentworth 5 Mar 2025 20:25 UTC

  9 points

  0

  in reply to: FluidThinkers’s comment on: What Is The Align­ment Prob­lem?

  The alignment problem does not assume AI needs to be kept in check, it is not focused on control, and adaptation and learning in synergy are entirely compatible with everything said in this post. At a meta level, I would recommend actually reading rather than dropping GPT2-level comments which clearly do not at all engage with what the post is talking about.

• johnswentworth 5 Mar 2025 18:21 UTC

  7 points

  0

  in reply to: faul_sname’s comment on: Statis­ti­cal Challenges with Mak­ing Su­per IQ babies

  I do agree that the end of the last glacial period was the obvious immediate trigger for agriculture. But the “humans are the stupidest thing which could take off model” still holds, because evolution largely operates on a slower timescale than the glacial cycle.

  Specifics: the last glacial period ran from roughly 115k years ago to 12k years ago. Whereas, if you look at a timeline of human evolution, most of the evolution from apes to humans happens on a timescale of 100k − 10M years. So it’s really only the very last little bit where an ice age was blocking takeoff. In particular, if human intelligence has been at evolutionary equilibrium for some time, then we should wonder why humanity didn’t take off 115k years ago, before the last ice age.

• johnswentworth 5 Mar 2025 17:15 UTC

  25 points

  8

  on: A Bear Case: My Pre­dic­tions Re­gard­ing AI Progress

  Noting for the sake of later evaluation: this rough picture matches my current median expectations. Not very high confidence; I’d give it roughly 60%.

• johnswentworth 5 Mar 2025 16:39 UTC

  7 points

  2

  in reply to: ChristianKl’s comment on: Statis­ti­cal Challenges with Mak­ing Su­per IQ babies

  The evidence I have mentally cached is brain size. The evolutionary trajectory of brain size is relatively easy to measure just by looking at skulls from archaeological sites, and IIRC it has increased steadily through human evolutionary history and does not seem to be in evolutionary equilibrium.

  (Also on priors, even before any evidence, we should strongly expect humans to not be in evolutionary equilibrium. As the saying goes, “humans are the stupidest thing which could take off, otherwise we would have taken off sooner”. I.e. since the timescale of our takeoff is much faster than evolution, the only way we could be at equilibrium is if a maximal-under-constraints intelligence level just happened to be exactly enough for humans to take off.)

  There’s probably other kinds of evidence as well; this isn’t a topic I’ve studied much.

• johnswentworth 4 Mar 2025 18:23 UTC

  13 points

  13

  in reply to: Jonas Hallgren’s comment on: Statis­ti­cal Challenges with Mak­ing Su­per IQ babies

  IIUC human intelligence is not in evolutionary equilibrium; it’s been increasing pretty rapidly (by the standards of biological evolution) over the course of humanity’s development, right up to “recent” evolutionary history. So difficulty-of-improving-on-a-system-already-optimized-by-evolution isn’t that big of a barrier here, and we should expect to see plenty of beneficial variants which have not yet reached fixation just by virtue of evolution not having had enough time yet.

  (Of course separate from that, there are also the usual loopholes to evolutionary optimality which you listed—e.g. mutation load or variants with tradeoffs in the ancestral environment. But on my current understanding those are a minority of the available gains from human genetic intelligence enhancement.)

• johnswentworth 4 Mar 2025 0:55 UTC

  10 points

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  Alternative model: French mathematicians don’t overperform in an objective sense. Rather, French mathematicians happened to end up disproportionately setting fashion trends in pure mathematics for a while, for reasons which are mostly just signalling games and academic politics rather than mathematical merit.

  The Bourbaki spring to mind here as a central example.

• johnswentworth 23 Feb 2025 5:39 UTC

  LW: 14 AF: 8

  5

  AF

  in reply to: joshc’s comment on: How might we safely pass the buck to AI?

  That’s a much more useful answer, actually. So let’s bring it back to Eliezer’s original question:

  Can you tl;dr how you go from “humans cannot tell which alignment arguments are good or bad” to “we justifiably trust the AI to report honest good alignment takes”? Like, not with a very large diagram full of complicated parts such that it’s hard to spot where you’ve messed up. Just whatever simple principle you think lets you bypass GIGO.

  [...]

  Broadly speaking, the standard ML paradigm lets you bootstrap somewhat from “I can verify whether this problem was solved” to “I can train a generator to solve this problem”.

  So to summarize your short, simple answer to Eliezer’s question: you want to “train AI agents that are [somewhat] smarter than ourselves with ground truth reward signals from synthetically generated tasks created from internet data + a bit of fine-tuning with scalable oversight at the end”. And then you hope/​expect/​(??have arguments or evidence??) that this allows us to (?justifiably?) trust the AI to report honest good alignment takes sufficient to put shortly-posthuman AIs inside the basin of attraction of a good eventual outcome, despite (as Eliezer puts it) humans being unable to tell which alignment takes are good or bad.

  Or, to compact the summary even further: you want to train the somewhat-smarter-than-human AI on easily-verifiable synthetically-generated tasks, and then hope/​expect that its good performance on those tasks generalizes to a problem which is not easily verifiable or synthetically generated, namely the problem of checking that a next generation of AI is in the basin of attraction of a good-to-humans outcome.

  (Note: I know you’ve avoided talking about the basin of attraction of a good-to-humans outcome, instead focused on just some short-term goal like e.g. not being killed by the very next generation of AI. Not focusing on the basin of attraction is a mistake, and we can go into why it’s a mistake if that turns out to be cruxy.)

  In Eliezer’s comment, he was imagining a training setup somewhat different from easily-verifiable synthetically-generated tasks:

  Assume that whenever OpenPhil tries to run an essay contest for saying what they’re getting wrong, their panel of judges ends up awarding the prize to somebody reassuringly saying that AI risk is an even smaller deal than OpenPhil thinks. How does OpenPhil bootstrap from that pattern of thumbs-up/​thumbs-down to an AI that actually has better-than-OpenPhil alignment takes?

  … but the analogue of the problem Eliezer was highlighting, in the context of training on easily-verifiable synthetically-generated tasks, is the question: how and why would we justifiably trust that an AI trained on easily-verifiable synthetic tasks generalizes to not-easily-verifiable real-world tasks?

• johnswentworth 22 Feb 2025 18:40 UTC

  LW: 23 AF: 12

  15

  AF

  in reply to: joshc’s comment on: How might we safely pass the buck to AI?

  The 3 month Eliezer sim might spin up many copies of other 3 month Eliezer sims, which together produce outputs that a 6-month Eliezer sim might produce.

  This seems very blatantly not viable-in-general, in both theory and practice.

  On the theory side: there are plenty of computations which cannot be significantly accelerated via parallelism with less-than-exponential resources. (If we do have exponential resources, then all binary circuits can be reduced to depth 2, but in the real world we do not and will not have exponential resources.) Serial computation requirements are, in fact, a thing. So you can’t just have a bunch of Eliezers do in 3 months what a single 6 month Eliezer could do, in general.

  Even if you allow the 3-month Eliezer sims to act one-after-another, rather than trying to get everything done in 3 months simulated time via pure parallelism, there’s still a tight communication bottleneck between each Eliezer sim. There are presumably plenty of circuits which cannot be implemented with tight information bottlenecks every n serial steps.

  … of course you could circumvent all that theory by e.g. just having each Eliezer emulate a single logic gate, or some comparably trivial thing, but at that point you run afoul of the non-compositionality of safety properties: putting together a bunch of “safe” things (or “interpretable” things, or “aligned” things, or “corrigible” things, …) does not in-general produce a safe thing.

  So that’s the theory side. What about the practice side?

  Well, Ought did that roughly that experiment years ago and it did not work at all. And that should not be surprising—as the link argues, we have extremely ample evidence from day-to-day life that such things do not work in general.

• johnswentworth 13 Feb 2025 23:04 UTC

  LW: 7 AF: 4

  0

  AF

  in reply to: kave’s comment on: Why Agent Foun­da­tions? An Overly Ab­stract Explanation

  I think that’s basically right, and good job explaining it clearly and compactly.

  I would also highlight that it’s not just about adversaries. One the main powers of proof-given-assumptions is that it allows to rule out large classes of unknown unknowns in one go. And, insofar as the things-proven-given-assumptions turn out to be false, it allows to detect previously-unknown unknowns.

• johnswentworth 9 Feb 2025 17:07 UTC

  14 points

  10

  in reply to: johnswentworth’s comment on: So You Want To Make Marginal Progress...

  Now to get a little more harsh...

  Without necessarily accusing Kaj specifically, this general type of argument feels motivated to me. It feels like willful ignorance, like sticking one’s head in the sand and ignoring the available information, because one wants to believe that All Research is Valuable or that one’s own research is valuable or some such, rather than facing the harsh truth that much research (possibly one’s own) is predictably-in-advance worthless.

• johnswentworth 9 Feb 2025 16:51 UTC

  7 points

  4

  in reply to: Kaj_Sotala’s comment on: So You Want To Make Marginal Progress...

  That sort of reasoning makes sense insofar as it’s hard to predict which small pieces will be useful. And while that is hard to some extent, it is not full we-just-have-no-idea-so-use-a-maxent-prior hard. There is plenty of work (including lots of research which people sink their lives into today) which will predictably-in-advance be worthless. And robust generalizability is the main test I know of for that purpose.

  Applying this to your own argument:

  Often when I’ve had a hypothesis about something that interests me, I’ve been happy that there has been *so much* scientific research done on various topics, many of them seemingly insignificant. While most of it is of little interest to me, the fact that there’s so much of it means that there’s often some prior work on topics that do interest me.

  It will predictably and systematically be the robustly generalizable things which are relevant to other people in unexpected ways.

• johnswentworth 8 Feb 2025 3:12 UTC

  15 points

  11

  in reply to: Ruby’s comment on: So You Want To Make Marginal Progress...

  Yup, if you actually have enough knowledge to narrow it down to e.g. a 65% chance of one particular major route, then you’re good. The challenging case is when you have no idea what the options even are for the major route, and the possibility space is huge.

• johnswentworth 8 Feb 2025 2:46 UTC

  6 points

  2

  in reply to: ryan_greenblatt’s comment on: So You Want To Make Marginal Progress...

  Yeah ok. Seems very unlikely to actually happen, and unsure whether it would even work in principle (as e.g. scaling might not take you there at all, or might become more resource intensive faster than the AIs can produce more resources). But I buy that someone could try to intentionally push today’s methods (both AI and alignment) to far superintelligence and simply turn down any opportunity to change paradigm.

• johnswentworth 8 Feb 2025 2:43 UTC

  13 points

  7

  in reply to: ryan_greenblatt’s comment on: So You Want To Make Marginal Progress...

  Aliens kill you due to slop, humans depend on the details.

  The basic issue here is that the problem of slop (i.e. outputs which look fine upon shallow review but aren’t fine) plus the problem of aligning a parent-AI in such a way that its more-powerful descendants will robustly remain aligned, is already the core of the superintelligence alignment problem. You need to handle those problems in order to safely do the handoff, and at that point the core hard problems are done anyway. Same still applies to aliens: in order to safely do the handoff, you need to handle the “slop/​nonslop is hard to verify” problem, and you need to handle the “make sure agents the aliens build will also be aligned, and their children, etc” problem.

• johnswentworth 8 Feb 2025 1:45 UTC

  12 points

  12

  in reply to: ryan_greenblatt’s comment on: So You Want To Make Marginal Progress...

  It’s not clear to me we’ll have (or will “need”) new paradigms before fully handing over all technical and strategic work to AIs which are capable enough to obsolete humans at all cognitive tasks.

  If you want to not die to slop, then “fully handing over all technical and strategic work to AIs which are capable enough to obsolete humans at all cognitive tasks” not a thing which happens at all until the full superintelligence alignment problem is solved. That is how you die to slop.

• johnswentworth 8 Feb 2025 1:43 UTC

  10 points

  0

  in reply to: ryan_greenblatt’s comment on: So You Want To Make Marginal Progress...

  If by superintelligence, you mean wildly superhuman AI, it remains non-obvious to me that new paradigms are needed (though I agree they will pretty likely arise prior to this point due to AIs doing vast quantity of research if nothing else). I think thoughtful and laborious implementation of current paradigm strategies (including substantial experimentation) could directly reduce risk from handing off to superintelligence down to perhaps 25% and I could imagine being argued considerably lower.

  I find it hard to imagine such a thing being at all plausible. Are you imagining that jupiter brains will be running neural nets? That their internal calculations will all be differentiable? That they’ll be using strings of human natural language internally? I’m having trouble coming up with any “alignment” technique of today which would plausibly generalize to far superintelligence. What are you picturing?

• johnswentworth 8 Feb 2025 1:04 UTC

  39 points

  14

  in reply to: ryan_greenblatt’s comment on: So You Want To Make Marginal Progress...

  This post seems to assume that research fields have big hard central problems that are solved with some specific technique or paradigm.

  This isn’t always true. [...]

  I would say it is basically-always true, but there are some fields (including deep learning today, for purposes of your comment) where the big hard central problems have already been solved, and therefore the many small pieces of progress on subproblems are all of what remains.

  And insofar as there remains some problem which is simply not solvable within a certain paradigm, that is a “big hard central problem”, and progress on the smaller subproblems of the current paradigm is unlikely by-default to generalize to whatever new paradigm solves that big hard central problem.

  I agree that paradigm shifts can invalidate large amounts or prior work, but it isn’t obvious whether this will occur in AI safety.

  I claim it is extremely obvious and very overdetermined that this will occur in AI safety sometime between now and superintelligence. The question which you’d probably find more cruxy is not whether, but when—in particular, does it come before or after AI takes over most of the research?

  … but (I claim) that shouldn’t be the cruxy question, because we should not be imagining completely handing off the entire alignment-of-superintelligence problem to early transformative AI; that’s a recipe for slop. We ourselves need to understand a lot about how things will generalize beyond the current paradigm, in order to recognize when that early transformative AI is itself producing research which will generalize beyond the current paradigm, in the process of figuring out how to align superintelligence. If an AI assistant produces alignment research which looks good to a human user, but won’t generalize across the paradigm shifts between here and superintelligence, then that’s a very plausible way for us to die.

So You Want To Make Marginal Progress...