That’s right. Being financially motivated to accurately predict timelines is a whole different thing than having the relevant expertise to predict timelines.
The part of Ajeya’s comment that stood out to me was this:
On a meta level I now defer heavily to Ryan and people in his reference class (METR and Redwood engineers) on AI timelines, because they have a similarly deep understanding of the conceptual arguments I consider most important while having much more hands-on experience with the frontier of useful AI capabilities (I still don’t use AI systems regularly in my work).
Because accurate prediction in a specialized domain requires expertise more than motivation. Forecasting is one relevant skill but knowledge of both current AI and knowledge of theoretical paths to AGI are also highly relevant.
Superforecasters can beat domain experts, as shown in Phil Tetlock’s work comparing superforecasters to intelligence analysts.
I’d guess in line with you that for forecasting AGI this might be different, but I am not not sure what weight I’d give superforecasters / prediction platforms versus domain experts.
Right. I think this is different in AGI timelines because standard human expertise/intuition doesn’t apply nearly as well as in the intelligence analyst predictions.
But outside of that speculation, what you really want is predictions from people who have both prediction expertise and deep domain expertise. Averaging in a lot of opinions is probably not helping in a domain so far outside of standard human intuitions.
I think predictions in this domain usually don’t have a specific end-point in mind. They define AGI by capabilities. But the path through mind-space is not at all linear; it probably has strong nonlinearities in both directions. The prediction end-point is in a totally different space than the one in which progress occurs.
Standard intuitions like “projects take a lot longer than their creators and advocates think they will” are useful. But in this case, most people doing the prediction have no gears-level model of the path to AGI, because they have no, or at most a limited, gears-level model of how AGI would work. That is a sharp contrast to thinking about political predictions and almost every other arena of prediction.
So I’d prefer a single prediction from an expert with some gears-level models of AGI for different paths, over all of the prediction experts in the world who lack that crucial cognitive tool.
I wouldn’t update too much from Manifold or Metaculus.
Instead, I would look at how people who have a track record in thinking about AGI-related forecasting are updating.
See for instance this comment (which was posted post-o3, but unclear how much o3 caused the update): https://www.lesswrong.com/posts/K2D45BNxnZjdpSX2j/ai-timelines?commentId=hnrfbFCP7Hu6N6Lsp
Or going from this prediction before o3: https://x.com/ajeya_cotra/status/1867813307073409333
To this one: https://x.com/ajeya_cotra/status/1870191478141792626
Ryan Greenblatt made similar posts / updates.
That’s right. Being financially motivated to accurately predict timelines is a whole different thing than having the relevant expertise to predict timelines.
The part of Ajeya’s comment that stood out to me was this:
I’d also look at Eli Lifland’s forecasts as well:
Gwern and Daniel Kokotajlo have a pretty notable track records at predicting AI scaling too, and they have comments in this thread.
Why not?
Because accurate prediction in a specialized domain requires expertise more than motivation. Forecasting is one relevant skill but knowledge of both current AI and knowledge of theoretical paths to AGI are also highly relevant.
Superforecasters can beat domain experts, as shown in Phil Tetlock’s work comparing superforecasters to intelligence analysts.
I’d guess in line with you that for forecasting AGI this might be different, but I am not not sure what weight I’d give superforecasters / prediction platforms versus domain experts.
This isn’t accurate, see this post: especially (3a), (3b), and https://docs.google.com/document/d/1ZEEaVP_HVSwyz8VApYJij5RjEiw3mI7d-j6vWAKaGQ8/edit?tab=t.0#heading=h.mma60cenrfmh Goldstein et al (2015)
Right. I think this is different in AGI timelines because standard human expertise/intuition doesn’t apply nearly as well as in the intelligence analyst predictions.
But outside of that speculation, what you really want is predictions from people who have both prediction expertise and deep domain expertise. Averaging in a lot of opinions is probably not helping in a domain so far outside of standard human intuitions.
I think predictions in this domain usually don’t have a specific end-point in mind. They define AGI by capabilities. But the path through mind-space is not at all linear; it probably has strong nonlinearities in both directions. The prediction end-point is in a totally different space than the one in which progress occurs.
Standard intuitions like “projects take a lot longer than their creators and advocates think they will” are useful. But in this case, most people doing the prediction have no gears-level model of the path to AGI, because they have no, or at most a limited, gears-level model of how AGI would work. That is a sharp contrast to thinking about political predictions and almost every other arena of prediction.
So I’d prefer a single prediction from an expert with some gears-level models of AGI for different paths, over all of the prediction experts in the world who lack that crucial cognitive tool.