Peter Johnson

Karma: 92

Peter Johnson Apr 29, 2025, 11:28 PM
1 point
0
in reply to: Daniel Kokotajlo’s comment on: AI 2027: What Superintelligence Looks Like
Apologies just saw this now since we were taking a break! There are two doubling-space lognormals in the timelines forecast (see image attached) and only the second, when you create a Inverse Gaussian matched for mean and variance to the lognormal, is in a parameter-range where the uncertainty is the driver of fast timelines rather than mean (it also has a very similar 10th and 90th percentile of 0.44 months and 18.7 months).
I do think speeding up to the second lognormal is not super well justified, but fine to ignore disagreements on parameter central tendencies (it’s kinda odd to say speeding-up because the mean actually gets slower while the median gets somewhat faster and the sub-median gets wildly faster (5x faster at the 10th percentile)).
I actually think adjusting this will make fast timelines significantly more appealing to people looking into the model because a big “what?” issue for me at least is how much mass implies we already have or are about to have SC in the timelines model, so adjustments that keep the median fairly close but sharply curtail how fast the 10th percentile are in the model would make me update to trust the model more (and thus believe a <2030 SC timeline more).

Peter Johnson Apr 23, 2025, 2:13 AM
3 points
1
in reply to: elifland’s comment on: AI 2027: What Superintelligence Looks Like
~~You can ignore for now since I need to work through whether this is still true depending on how we view the source of uncertainty in doubling time.~~ Edit: this explanation is correct afaict and worth looking into.
The parameters for the second log-normal (doubling time at RE-Bench saturation, 10th: 0.5 mo., 90th: 18 mo.) when made equivalent for an inverse gaussian by matching mean and variance (approx. InverseGaussian[7.97413, 1.315]) are implausible. The linked paper highlights that to be representing doubling processes reasonably, the ratio of first to second parameter ought to be << 2/ln(2) (or << (1/(2ln(2)^2))). The failure to match that indicates that the “size hypothesis” of any of the growth processes is violated, indicating that the distribution is no longer modeling uncertainty around such a process.
Ok, so that’s too many functions, what does it mean? In general, it means that our “uncertainty” is actually the main driver of fast timelines now rather than reflecting a lack of knowledge in any way. The distribution is so stretched that the mode and median are wildly smaller than the mean entirely due to the possibility that a random unknown event causes foom, unrelated to the estimated “growth rate” of the process. It’s like cranking up a noise term on a stock market model and being surprised that some companies are estimated to go to the moon tomorrow, then claiming it is due to estimating those stocks as potentially having huge upsides.
There is not a good solution that keeps the model intact (and the same basic issue is that the model is working in domains that are outcomes like time and frequency rather than inputs like time horizons, compute, or effective compute). If one were to use the same mean and scale up the second parameter, the left side of the pdf would collapse, and the mode and median would jump much higher resulting in a much later estimate of SC. That doesn’t mean that’s how to fix the model, but it does indicate fast timelines are incidental to and reflective of other issues in the model.

Peter Johnson Apr 22, 2025, 6:18 PM
3 points
0
on: AI 2027: What Superintelligence Looks Like
Edit: see subsequent response for more accurate formalizing
In the benchmark gaps timelines forecast there are two “doubling rate” parameters modeled with log-normal uncertainty. Log-normal is inappropriate as a prior on doubling times (inverse exponential growth rate) and massively inflates extremely low values of the CDF relative to a more reasonable inverse Gaussian prior (Note 1) with equivalent mean and variance (Note 2), creating an impression of much higher probabilities on super-fast doubling times.
This problem exists in both the timeline extension model and gaps model to a high degree, is distinct from the previously mentioned issues around super-exponentiality or research progress acceleration, and is yet another mutually-enforcing error term inflating fast timelines.
~~Note 1:~~ https://www.tandfonline.com/doi/pdf/10.1080/07362994.2015.1010124
~~Note 2: ratio of LogNormal/InverseGaussian CDFs analytically here for equivalent mean of e^1.5 and variance:~~ https://www.wolframalpha.com/input?i=plot+%281%2F2%29+*%281+%2B+erf%28%28-1+%2B+log%28x%29%29%2Fsqrt%282%29%29%29+%2F+%280.5+*%28erfc%28-%280.919989+%28-4.48169+%2B+x%29%29%2Fsqrt%28x%29%29+%2B+3.88584%C3%9710%5E6+erfc%28%280.919989+%284.48169+%2B+x%29%29%2Fsqrt%28x%29%29%29%29%2C+x+from+0+to+20

Peter Johnson Apr 22, 2025, 3:16 PM
4 points
1
in reply to: elifland’s comment on: AI 2027: What Superintelligence Looks Like
I bet that we will not see a model released in the future that equals or surpasses the general performance of Chinchilla while reducing the compute (in training FLOPs) required for such performance by an equivalent of 3.5x per year.
FWIW I think much of software progress comes from achieving better performance at a fixed or increased compute budget rather than making a fixed performance level more efficient, so I think this underestimates software progress.
The main justification for having compute efficiency be approximately equal to compute in terms of progress given in the timeline supplement and main dropdown is the Epoch AI measurements which are specifically about fixed-performance and lower compute. At the very least this concedes that the estimates are not based on trend-extrapolation and are conjecture.
I agree that it’s harder to quantify software improvements at the same or higher levels of compute in a way that can be easily compared against compute increases, but we can totally measure some part of it by looking at performance increasing given thes same compute budget (it’s quite hard to measure the metric of “how much compute would it have taken 2015 agorithms/data to reach 2025 performance” though, for obvious reasons).
Something being harder to measure is not an excuse for ignoring it.
Something being unfalsifiable forward-looking and unmeasurable backwards-looking is a justification for not treating it with high credence, so I think this is also a core disagreement.
To be clear, I agree that there will be some slowdown due to complementarity of software and hardware, and ideally this would be measured in the model. One can think that there will be multiple effects in different directions. I think that at the levels of research speedup observed in the timelines supplement, the magnitude is likely to be low enough to not change the overall takeaways from the model, but maybe you disagree. I might get around to adding this in as it would be nice.
Here are two charts demonstrating that small changes in estimates of current R&D contribution and changes in R&D speedup change the model massively in the absence of a singularity. I know we’re just going to go straight back to “well the real model is the even-more-unfalsifiable benchmarks and gaps model,” but I think that is unreasonable.
EDIT: THESE FIGURES OVERESTIMATE THE IMPACT OF REDUCING CURRENT ALGORITHMIC PROGRESS. THE SECOND IS WRONG, AND THE REAL IMPACT IS MORE CONTAINED.
~~Figure 1: R&D is 50% of current progress, with and without speedups, exponential only~~
~~Figure 2: R&D is 33% of current progress, with and without speedups, exponential only~~
~~I do not understand how “I think this variable doesn’t matter (without checking)” is a good defense about questionably implemented variables that~~ do ~~overdetermine the model, but “this variable doesn’t matter to outcomes” is not a valid critique w.r.t. things like “what are current capabilities/time horizon”~~
THIS SECOND ONE IS WRONG, MEDIAN HORIZON CHANGES BY CLOSER TO HALF A YEAR AT 33% (TO FEB 2029) THAN ALMOST 2 YEARS (TO APR 2031 AS INCORRECTLY SHOWN)

Peter Johnson Apr 22, 2025, 2:54 PM
2 points
1
in reply to: elifland’s comment on: AI 2027: What Superintelligence Looks Like
You’re right on the 143 being closer to 114! (I took March 1 2022 → July 1 2022 instead of March 22 2022 → June 1 2022 which is accurate).
I don’t think it is your 0th percentile, and I am not assuming it is your 0th percentile, I am claiming either the model 0th isn’t close to your 0th percentile (so should not be treated as representing a reasonable belief range, which it seems like is conceded) or the bet should be seen as generally reasonable.
I sincerely do not think a limited time argument is valid given the amount of work that was put into non-modeling aspects of the presentation and the amount of work claimed put into the model over several gamings and reviews and months of work etc etc.
If the burden of proof is on critics to do work you are not willing to do in order to show the model is flawed (for a bounty between 4-10% of the bounty you offer someone writing a supporting piece to advertise your position further), then the defense of limited time raises some hackles.

Peter Johnson Apr 22, 2025, 2:18 PM
6 points
−2
in reply to: elifland’s comment on: AI 2027: What Superintelligence Looks Like
I can’t argue against a handful different speedups all on the object level without reference to each other. The justifications generally lie on basically the same intuition which is that AI R&D is strongly enhanced by AI in a virtuous cycle. The only mechanical cause for the speedup claimed is compute efficiency (aka less compute per same performance), and it’s hard for me to imagine what other mechanical cause could be claimed that isn’t contained in compute or compute efficiency.
Finally if I understand the gaps model, it is not a trend exptrapolation model at all! It is purely guesses about calendar time put into a form they are hard to disentangle or validate.
To make effective bets we need a relatively high-probability, falsifiable, and quickly-resolving metric that is unlikely to be gamed. METR benchmarks (like every benchmark ever) are able to be gamed or reacted to (the gaming of which is the argument made about most of those handful of distinct speedups). However, if the model relies on a core assumption that is falsifiable, we should focus on that metric. If computational efficiency gains are not core to the model, I am confused on how it claims we will reach SC that is different from bare assertion that we reach SC soon with no reference to anything falsifiable!

Peter Johnson Apr 22, 2025, 2:03 PM
2 points
−1
in reply to: ryan_greenblatt’s comment on: AI 2027: What Superintelligence Looks Like
Your source specifically says it is far overtrained relative to optimal compute scaling laws?
“This is why it makes sense to train well past Chinchilla optimal for any model that will be deployed.”

Peter Johnson Apr 22, 2025, 12:29 AM
2 points
1
in reply to: Peter Johnson’s comment on: AI 2027: What Superintelligence Looks Like
If my belief was that we never cross that threshold, i would not be citing a paper that includes a figure explicitly showing that threshold being crossed repeatedly. My point is that counting it as a long term trend is indefensible.

Peter Johnson Apr 22, 2025, 12:10 AM
2 points
1
in reply to: ryan_greenblatt’s comment on: AI 2027: What Superintelligence Looks Like
Yes, and also GPT-4 is nowhere close to compute-efficient?
Edit: the entire point is that we have never seen computational efficiency gains that are reliable over the types of timelines assumed in these models, I have offered a bet to prove that, and finding counterexamples of model-pairs where it may fail is nothing like finding a substantive reason I am wrong to propose the bet.
Edit 2: with regard to your [?] I sincerely do not think the burden of proof is on me to demonstrate that a model is not compute efficient when I have already committed to monetary bets on models I believe are. If it is compute efficient according to even Kaplan or Chinchilla scaling laws, please demonstrate that for me. I did not bring it up as a compute-efficient model, you did!

Peter Johnson Apr 22, 2025, 12:10 AM
2 points
0
in reply to: Peter Johnson’s comment on: AI 2027: What Superintelligence Looks Like
For the record, here is the simple model without a super-exponential singularity, both with and without the R&D speedups:
I can hardly call this anything but extremely determinate of the results

Peter Johnson Apr 21, 2025, 11:47 PM
1 point
0
in reply to: ryan_greenblatt’s comment on: AI 2027: What Superintelligence Looks Like
There are more speedups hidden across parameters, e.g. “Doubling time at RE-Bench saturation toward our time horizon milestone, on a hypothetical task suite like HCAST but starting with only RE-Bench’s task distribution” which also just drops the doubling time.
Unless you are in the simpler model, in which case the singularity is hiding the importance of the R&D speedup.

Peter Johnson Apr 21, 2025, 11:41 PM
2 points
1
in reply to: ryan_greenblatt’s comment on: AI 2027: What Superintelligence Looks Like
R1 can’t possibly be below V3 cost because it is inclusive? If I’m not mistaken, R1 is not trained from scratch, but I could be wrong.
Second, GPT-4 is not a compute-efficient model, afaik, which is why Chinchilla was my choice, not a random big model. Furthermore, V3 does not come all that close to meeting the requirement for how many less FLOPs a model needs to hit the 3.5x per year reduction (it is <6x smaller when 3.5x per year over 1.75 years implies about 9x smaller), let alone the 4.6x per year reduction the forecaster models imply.
So even if you try to find a best case scenario for a jump in compute efficiency that breaks the general failure to hit a consistent trend over longer than 1.5 year timelines (which is the crux), it does not meet the standard. And it is commonly believed that V3 itself took advantage of undisclosed distillation, although I won’t press that.
So we have GPT-4, a non-effiency-frontier model vs. a questionably independent V3 that does not even hit the target claimed that also represents the biggest compute-efficiency upgrade in the last two years (or at least R1 might).
I don’t see how my bet as stated is likely to fall anytime soon.

Peter Johnson Apr 21, 2025, 11:23 PM
3 points
−2
in reply to: ryan_greenblatt’s comment on: AI 2027: What Superintelligence Looks Like
I will leave it to the author to confirm or deny your second point.
@elifland Here is evidence of my assertion that a generally engaged reader does not appreciate nearly how dominant AI R&D speedups are with regard to either model we talked about today.

Peter Johnson Apr 21, 2025, 11:18 PM

3 points

in reply to: ryan_greenblatt’s comment on: AI 2027: What Superintelligence Looks Like

From the code:

        while progress < base_time_in_months[i] and time < max_time:
            # Calculate progress fraction
            progress_fraction = progress / base_time_in_months[i]
            
            # Calculate algorithmic speedup based on intermediate speedup s(interpolate between present and SC rates)
            v_algorithmic = (1 + samples["present_prog_multiplier"][i]) * ((1 + samples["SC_prog_multiplier"][i])/(1 + samples["present_prog_multiplier"][i])) ** progress_fraction

            # adjust algorithmic rate if human alg progress has decreased, in betweene
            if time >= human_alg_progress_decrease_date:
                only_multiplier = v_algorithmic * 0.5
                only_additive = v_algorithmic - 0.5
                # geometric mean of only_multiplier and only_additive, aggregating between extremes of how AIs/humans could complement
                v_algorithmic = np.sqrt(only_multiplier * only_additive)
            
            # Get compute rate for current time (not affected by intermediate speedups)
            compute_rate = get_compute_rate(time, compute_decrease_date)
            # Total rate is mean of algorithmic and compute rates
            total_rate = (v_algorithmic + compute_rate) / 2

~~I said there was no compute~~ ~~acceleration~~ ~~not that there was no more compute scaling?~~ They fix constant compute velocity (compute_rate) and a equal and super-accelerated algorithmic or efficiency term (v_algorithmic) as shown in the code here.

Peter Johnson Apr 21, 2025, 11:15 PM
2 points
0
in reply to: ryan_greenblatt’s comment on: AI 2027: What Superintelligence Looks Like
Do you have a source because everything I can find implies approximately equal FLOPs in e2e training costs for GPT-4 and R1 or Deepseek-V3?

Peter Johnson Apr 21, 2025, 11:12 PM
3 points
0
in reply to: ryan_greenblatt’s comment on: AI 2027: What Superintelligence Looks Like
I’m not sure what you mean? In all timelines models presented there is acknowledgement that compute does not accelerate. However, “research progress”, and necessarily compute-efficiency, is the main driver of capabilities acceleration in the more detailed gaps model, and no other mechanism for why capabilities enhance faster than linear trend is proposed (in the simple model the acceleration factor is absurd enough to not engage with as modeling compute-efficiency or compute, so I don’t challenge them to defend that model). I am not saying compute has zero impact, I am quantifying their own estimated impact of compute vs. efficiency and offering a huge discount against the lowest possible interpretations of how efficiency moves capabilities in their model.

Peter Johnson Apr 21, 2025, 9:42 PM
5 points
−2
on: AI 2027: What Superintelligence Looks Like
I think I have a sense now of what one crux of disagreement is between the assumptions in the timelines models and my beliefs, so I’d like to propose a bet.
Crux:
I believe compute-optimality has not scaled at the level the forecasts claim and will not scale super-exponentially in the near future as the models near-universally predict.
The modes assume “improved algorithms...makes up about half of current AI progress”. This is modeled as research progress equaling the last few years of compute increases (aka ~4.6x per year per Epoch AI) in terms of how much better models are vis-a-vis effective compute. This is both projected forward and treated with either 1 (in 45% of cases) or 2 (in all cases) super-exponential terms that make it significantly faster than an inferred 4.6x per year.
I am willing to make a bet that implies that compute-optimality has not been increasing even 3.5x per year since Chinchilla’s release in March 2022 and that this failure to reach a much lower bar will continue. This is well before ChatGPT became popular knowledge and since when, the authors claim strong evidence of already-existing super-exponentiality with regards to compute-efficiency progress exceeding the baseline 4.6x assumed. Furthermore, I will accept a negative return for the first 10 months we continue to fail to meet this much lower standard.
For reference, the 0th percentile assumed increase in computational efficiency by the authors of the forecasts is about ~~143x~~ 114x since the Chinchilla release while I am accepting values of just 60x as an immediate loss of my entire principal. By the time the bet turns to a positive return for me (around April 2026), their 0th percentile model assumes increases in computational efficiency of nearly 500x while I accept even a 150x demonstrated increase as a loss.
Bet:
I bet that we will not see a model released in the future that equals or surpasses the general performance of Chinchilla while reducing the compute (in training FLOPs) required for such performance by an equivalent of 3.5x per year.
I am willing to put up $5k with the condition that starting June 1, 2025 the taker of the bet pays $500 each month such a model fails to appear, and does so for 60 months. Because the bet starts below the 0th percentile implied computational efficiency improvements modeled and only gets further away from such a level, this should be an obvious win for the forecasters.
I am willing to accept an arbiter of the taker’s choice (reasonably assured to be fair by others) to evaluate that the benchmark scores of the model do represent broadly equivalent general performance to Chinchilla on tasks including at least zero-shot WikiText103 perplexity and 5-shot MMLU, that the model is not a distill or otherwise specifically optimized for such benchmarks beyond reason, and that the tasks are not specific to domains unused as benchmarks in the time Chinchilla was being trained and released. This obviously has a huge disadvantage for me in that test leakage has increased massively since that time.
I am willing to consider other bets that encode the same general form that there is not a generalized increase in computational efficiency anywhere close to that the authors claim, and that a massive increase in such efficiency due to research progress speedups will not occur in the foreseeable future.
I claim that a response that “increases in computational efficiency only accrue to frontier models” or something similar does two things:
1. Is unfalsifiable, thus should not be taken seriously unless an alternative metric is proposed that concretely demonstrates computational efficiency gains having been achieved near the levels claimed.
2. Largely concedes the main point I want to make that as we reach a slowdown in compute in the next 3 years, there will not be a rapid increase in rate of computational efficiency gains, but rather a significant decrease as the positive mutual effect of compute increases on algorithmic efficiency gains decline.
If there cannot be any demonstration that current levels of computational efficiency improvements they assume are even plausible, I think that should be considered as substantively eliminating support for fast timelines in the range claimed here (as removing such assumptions from the models will demonstrate).

Peter Johnson Apr 20, 2025, 11:48 PM
3 points
0
in reply to: Daniel Kokotajlo’s comment on: AI 2027: What Superintelligence Looks Like
Ah, I was trying to avoid implying a lack of integrity in the forecasting effort, and as a result ended up implying knowing other things about your mental state.
To restate: I do not think a forecaster not previously committed to achieving a result with a median around that point would have, after doing a perturbation analysis that would display how dominant the two superexponential terms are in predetermining that specific outcome, presented those conclusions without hammering the point that those two superexponential terms are completely determining the topline result and that no other parameters make a meaningful contribution, whether forecasts of compute availability, current capabilities, capabilities required for SC, or otherwise.
Sorry if my previous comment implied something else!

Peter Johnson Apr 20, 2025, 6:29 PM
6 points
−1
in reply to: elifland’s comment on: AI 2027: What Superintelligence Looks Like
Looking forward to getting clarity!
“we would have ended up with similar forecasts in terms of their qualitative conclusions about how plausible AGI in/by 2027 is” if this is with regard to perturbation studies of the two super-exponential terms, I believe based on the work I’ve shared in part here it is false, but happy to agree to disagree on the rest :)

Peter Johnson Apr 20, 2025, 5:57 PM
8 points
1
in reply to: elifland’s comment on: AI 2027: What Superintelligence Looks Like
I honestly think that’s pretty irrelevant since the data are purely produced through non-reproducible intuited or verbally justified parameter estimates, so intentionality is completely opaque to readers, which is really the bigger core problem with the exercise.
Okay, we have a more fundamental disagreement than I previously realized. Can you tell me which step of the following argument you disagree with:
1. It’s important to forecast AGI timelines.
2. In order to forecast AGI timelines while taking into account all significantly relevant factors, one needs to use intuitive estimations for several important parameters.
3. Despite (2), it’s important to try anyways and make the results public to (a) advance the state of public understanding of AGI timelines and (b) make our reasoning for our timelines more transparent than if we gave a vague justification with no model.
4. Therefore, it’s a good and important thing to make and publish models like ours
I have nothing against publishing models like this! I think the focus should be on modeling assumptions like super-exponentiality and treated as modeling exercises first-and-foremost. Assumptions that highly constrain the model should be first and foremost rather than absent from publicly facing write-ups and only in appendices. Perturbation tests should be conducted and publicly shown. It is true that I don’t think this represents high-quality quantitative work, but that doesn’t mean I don’t think any of it is relevant, even if I think the presentation does not demonstrate a good sense of responsibility. The hype and advertising-ness of the presentation is consistent, and while I have been trying to avoid particular comment thereon, it is a constant weight hanging over the discussion. Publication of a model and breathless speculation are not the same, and I really do not want to litigate the speculative aspects but they are the main thing everyone is talking about including in the New York Times.
I think this is not true, but I don’t think it is worth litigating separately from the research progress super-exponential factor which is also modeled as already happening. so it is easeir (though I don’t think easy) to defend both individually based on small potatoes evidence of a speedup, but I can’t imagine how both could be defended at once, so that has to come first so we don’t double count evidence.
I don’t understand why you think we shouldn’t account for research progress speedups due to AI systems on the way to superhuman coder. And this is clearly a separate dynamic from the trend being possibly superexopnential without these speedups taken into account (which, again, we already see some empirical evidence for this possibility). I’d appreciate if you made object-level arguments against one or both of these factors being included.
I think this has a relatively simple crux
1. Research progress has empirically slowed according to your own source of EpochAI (Edit: as a percentage of all progress which is the term used in the model. It is constant in absolute terms)
2. You assume the opposite which would explain away the METR speed-up on its own
3. So justifying the additional super-exponential term with the METR speed-up is double counting evidence
Then, split out the calculate_base_time function from the calculate_sc_arrival_year function and plot those results (here is my snippet to adjust the base time to decimal year and limit to the super-exponential group: base_time_in_months[se_mask] / 12 + 2025 ). You’ll have to do some simple wrangling to expose the se_mask and get the plotting function to work but nothing substantive.
Could you share the code on a fork or PR?
I can drop my versions of the files in a drive or something, I’m just editing on the fly, so they aren’t intended to be snapshotted, but here you go!
https://drive.google.com/drive/folders/1_e-hnmSy2FoMSD4UiWKWKNKnsapZDm1M?usp=drive_link
Edit:
Imagine if climate researchers had made a model where carbon impact on the environment gets twice-to-infinity times larger in 2028 based on “that’s what we think might be happening in Jupiter.”
A few thoughts:
1. Climate change forecasting is fundamentally more amenable to grounded quantitative modeling than AGI forecasting, and even there my impression is that there’s substantial disagreement based on qualitative arguments regarding various parameter settings (though I’m very far from an expert on this).
2. Forecasts which include intuitive estimations are commonplace and often useful (see e.g. intelligence analysis, Superforecasting, prediction markets, etc.).
3. I’d be curious if you have the same criticism of previous timelines forecasts like Bio Anchors.
4. I don’t understand your analogy re: Jupiter. In the timelines model, we are trying to predict what will happen in the real world on Earth.
4. The analogy is that the justification uses analogy to humans which are notably not LLMs. If it is purely based on the METR “speed-up” that should be made more clear as it is a (self-admittedly) weak argument.