Cole Wyeth

Karma: 1,819

I am a PhD student in computer science at the University of Waterloo, supervised by Professor Ming Li and advised by Professor Marcus Hutter.

My current research is related to applications of algorithmic probability to sequential decision theory (universal artificial intelligence). Recently I have been trying to start a dialogue between the computational cognitive science and UAI communities. Sometimes I build robots, professionally or otherwise. Another hobby (and a personal favorite of my posts here) is the Sherlockian abduction master list, which is a crowdsourced project seeking to make “Sherlock Holmes” style inference feasible by compiling observational cues. Give it a read and see if you can contribute!

See my personal website colewyeth.com for an overview of my interests and work.

I do ~two types of writing, academic publications and (lesswrong) posts. With the former I try to be careful enough that I can stand by ~all (strong/central) claims in 10 years, usually by presenting a combination of theorems with rigorous proofs and only more conservative intuitive speculation. With the later, I try to learn enough by writing that I have changed my mind by the time I’m finished—and though I usually include an “epistemic status” to suggest my (final) degree of confidence before posting, the ensuing discussion often changes my mind again.

Cole Wyeth Apr 14, 2025, 6:21 PM
2 points
0
on: Unbendable Arm as Test Case for Religious Belief
I’m pretty sure I learned to do this based on a simple mechanistic description (without detailed physics) and practice.

Cole Wyeth Apr 13, 2025, 8:15 PM
5 points
0
in reply to: Vladimir_Nesov’s comment on: Cole Wyeth’s Shortform
Yeah, I was thinking greater effort is actually necessary in this case. For context, my lower effort posts are usually more popular. Also the ones that focus on LLMs which is really not my area of expertise.

Cole Wyeth Apr 13, 2025, 7:10 PM
10 points
15
on: Cole Wyeth’s Shortform
Particularly after my last post, I think my lesswrong writing has had bit too high of a confidence / effort ratio. Possibly I just know the norms of this site well enough lately that I don’t feel as much pressure to write carefully. I think I’ll limit my posting rate a bit while I figure this out.

Cole Wyeth Apr 13, 2025, 6:23 PM
2 points
0
in reply to: lc’s comment on: Reactions to METR task length paper are insane
I am saying so! The market is definitely not pricing in AGI; doesn’t matter if it comes in 2028, or 2035, or 2040. Though interest rates are a pretty bad way to arb this; I would just buy call options on the Nasdaq.
Hmm well at least you’re consistent.
They’re not that useful yet.
Certainly I can see why you expect them to become more useful, I still feel like there’s some circularity here. Do you expect the current paradigm to continue advancing because LLM agents are somewhat useful now (as you said, for things like coding)? Unless that effect is currently negligible (and will undergo a sharp transition at some point) it seems we should expect it to already be reflected in the exponential growth rate claimed by METR.

Cole Wyeth Apr 13, 2025, 6:20 PM
2 points
0
in reply to: AnthonyC’s comment on: Reactions to METR task length paper are insane
You’re right, but creating unexpected new knowledge is not a PhD requirement. I expect it’s pretty rare that a PhD students achieves that level of research.
I do weakly expect it to be necessary to reach AGI though. Also, I personally wouldn’t want to do a PhD that didn’t achieve this!
It wasn’t a great explanation, sorry, and there are definitely some leaps, digressions, and hand-wavy bits. But basically: Even if current AI research were all blind mutation and selection, we already know that that can yield general intelligence from animal-level-intelligence because evolution did it. And we already have various examples of how human research can apply much greater random and non-random mutation, larger individual changes, higher selection pressure in a preferred direction, and more horizontal transfer of traits than evolution can, enabling (very roughly estimated) ~3-5 OOMs greater progress per generation with fewer individuals and shorter generation times.
Okay, then I understand the intuition but I think it needs a more rigorous analysis to even make an educated guess either way.
Saw your edit above, thanks.
No, thank you!

Cole Wyeth Apr 13, 2025, 3:22 PM
2 points
0
in reply to: lc’s comment on: Reactions to METR task length paper are insane
Thanks for engaging in detail with my post. It seems there were a few failures of communication that are worth clarifying.
It’s (the outside view) not really a well-defined thing, which is why the standard on this site is to taboo those words and just explain what your lines of evidence are, or the motivation for any special priors if you have them.
I thought it was clear that I’m not confident in any outside view prediction of AGI timelines, from various statements/phrasings here (including the sentence you’re quoting, which questions the well-definedness of “the outside view”) and the fact that the central focus of the post is disputing an outside view argument. Apparently I did not communicate this clearly, because many commenters have objected to my vague references to possible outside views as if I were treating them as solid evidence, when in fact they aren’t really a load bearing part of my argument here. Possibly the problem is that I don’t think anyone has a good inside view either! But in fact I am just “radically” uncertain about AGI timelines—my uncertainty is ~in the exponent.
Still, I find your response a little ironic since this site is practically the only place I’ve seen the term “outside view” used. It does seem to be less common over the last year or two, since this post which you’re probably referring to.
So, your claim is that interest rates would be very high if AGI were imminent, and they’re not so it’s not. The last time someone said this, if the people arguing in the comment section had simply made a bet on interest rates changing, they would have made a lot of money! Ditto for buying up AI-related stocks or call options on those stocks.
Interesting, but non-sequitur. That is, either you believe that interest rates will predictably increase and there’s free money on the table, and you should just say so, or not, and this anecdote doesn’t seem to be relevant (similarly, I made money buying NVDA around that time, but I don’t think that proves anything).
You could say that the “inventing important new ideas” part is going to be such a heavy bottleneck, that this speedup won’t amount to much. But I think that’s mostly wrong, and that if you asked ML researchers at OpenAI, a drop in remote worker that could “only” be directed to do things that otherwise took 12 hours would speed up their work by a lot.
Perhaps, but shouldn’t LLMs already be speeding up AI progress? And if so, shouldn’t that already be reflected in METR’s plot? Are you predicting superexponential growth here?
It’s actually not circular at all. “Current AI research” has taken us from machines that can’t talk to machines that can talk, write computer programs, give advice, etc. in about five years. That’s the empirical evidence that you can make research progress doing “random” stuff. In the absence of further evidence, people are just expecting the thing that has happened over the last five years to continue. You can reject that claim, but at this point I think the burden of proof is on the people that do.
It seems to me that progress has been slowing for the last couple of years. If this trend continues, progress will stall.

Cole Wyeth Apr 13, 2025, 2:59 PM
2 points
−2
in reply to: AnthonyC’s comment on: Reactions to METR task length paper are insane
I imagine you could reliably complete a PhD in many fields with a week-long time horizon, as long as you get good enough weekly feedback from a competent advisor. 1: Talk to advisor about what it takes to get a PhD. 2: Divide into a list of <1 week-long tasks. 3) Complete task 1, get feedback, revise list. 4) Either repeat the current task or move on to the new next task, depending on feedback. 5) Loop until complete. 5a) Every ten or so loops, check overall progress to date against the original requirements. Evaluate whether overall pace of progress is acceptable. If not, come up with possible new plans and get advisor feedback
I think it’s nearly impossible to create unexpected new knowledge this way.
As far as not believing the current paradigm could reach AGI, which paradigm do you mean? I don’t think “random variation and rapid iteration” is a fair assessment of the current research process. But even if it were, what should I do with that information? Well, luckily we have a convenient example of what it takes for blind mutations with selection pressure to raise intelligence to human levels: us! I am pretty confident saying that current LLMs would outperform, say, Australopithecus, on any intellectual ability, but not Home sapiens. So that happens in a few million years, let’s say 200k generations of 10-100k individuals each, in which intelligence was one of many, many factors weakly driving selection pressure with at most a small number of variations per generation. I can’t really quantify how much human intelligence and directed effort speed up progress compared to blind chance, but consider that 1) a current biology grad student can do things with genetics in an afternoon that evolution needs thousands of generations and millions of individuals or more to do, and 2) the modern economic growth rate, essentially a sum of the impacts of human insight on human activity, is around 15000x faster than it was in the paleolithic. Naively extrapolated, this outside view would tell me that science and engineering can take us from Australopithecus-level to human-level in about 13 generations (unclear which generation we’re on now). The number of individuals needed per generation is dependent on how much we vary each individual, but plausibly in the single or double digits.
I can’t parse this.
My disagreement with your conclusion from your third objection is that scaling inference time compute increases performance within a generation, but that’s not how the iteration goes between generations. We use reasoning models with more inference time compute to generate better data to train better base models to more efficiently reproduce similar capability levels with less compute to build better reasoning models. So if you build the first superhuman coder and find it’s expensive to run, what’s the most obvious next step in the chain? Follow the same process as we’ve been following for reasoning models and if straight lines on graphs hold, then six months later we’ll plausibly have one that’s a tenth the cost to run. Repeat again for the next six months after that.
You’re probably right about distilling CoT.

Cole Wyeth Apr 13, 2025, 1:29 PM
8 points
8
in reply to: Chris_Leong’s comment on: Chris_Leong’s Shortform
Upskilling bright young people “to do alignment” is tricky to do in a systematic way, since bright young people want / need to do whatever they’re curious about.

Cole Wyeth Apr 11, 2025, 7:25 PM
9 points
0
in reply to: Daniel Kokotajlo’s comment on: AI 2027: What Superintelligence Looks Like
Similarly, if the METR trend continues I will become very worried that AGI is near.

Cole Wyeth Apr 11, 2025, 7:08 PM
4 points
0
in reply to: Daniel Kokotajlo’s comment on: AI 2027: What Superintelligence Looks Like
Yes. Though, I find it a bit hard to visualize a 4 hour software engineering task that can’t be done in 1 hour, so I’m more clear on there not being a 16x or so improvement in 2 years

Cole Wyeth Apr 11, 2025, 6:53 PM
2 points
0
in reply to: tangerine’s comment on: Reactions to METR task length paper are insane
I don’t think I have anything better than a guess about any of those questions.

Cole Wyeth Apr 11, 2025, 6:52 PM
2 points
0
in reply to: AnthonyC’s comment on: Reactions to METR task length paper are insane
I think this is at least superficially a reasonable interpretation, and if the new linear relationship continues then I’d be convinced it’s right, but I wish you had engaged more with the arguments I made in the post or could be a bit more explicit about which you don’t follow?
Basically, I just have very low confidence in putting a line through these points because I don’t see a principled reason to expect a linear relationship to hold, and I see some reasons to expect that it should not.

Cole Wyeth Apr 11, 2025, 6:09 PM
4 points
0
in reply to: Daniel Kokotajlo’s comment on: AI 2027: What Superintelligence Looks Like
Sure, I suppose that now I’ve started recklessly speculating about the future I might as well follow through.

I expect the departure to be pretty clear though, because we won’t see superhuman ai engineers before 2030. Even that prediction needs to be operationalized a bit of course.

Cole Wyeth Apr 11, 2025, 5:59 PM
4 points
1
in reply to: Steven Byrnes’s comment on: Reactions to METR task length paper are insane
Thanks for the links, I’ll look into it.
I agree that the S&P is pretty much reading tea leaves, the author of the interest rates post @basil.halperin has separately argued it is not reliable.

Cole Wyeth Apr 11, 2025, 5:47 PM
6 points
0
in reply to: Daniel Kokotajlo’s comment on: Reactions to METR task length paper are insane
Does the METR report then update you towards later timelines?
(I am not saying it necessarily should, since obviously I have some doubts about the strength of its evidence)

Cole Wyeth Apr 11, 2025, 5:46 PM
2 points
0
in reply to: Lucas Teixeira’s comment on: Reactions to METR task length paper are insane
I agree, but my experience of doing a PhD actually feels more like an integrated project of discovery and creation.

Cole Wyeth Apr 11, 2025, 5:28 PM
8 points
0
in reply to: tangerine’s comment on: Reactions to METR task length paper are insane
Unfortunately, those are two importantly different questions. While it is certainly possible that AGI will arrive before 2030, I am not sure that it would be predictable in advance without a “textbook of the (far) future” on deep learning. Of course, it is much more likely to be predictable a couple of days in advance than a year in advance, but even the former may conceivably be quite challenging depending on situational awareness of near-human-level models in training.
What would you expect the world to look like if AGI < 2030?
In many respects, similar to how it does look (e.g. AI passes the Turing test pretty easily / convincingly, is competent at some coding tasks, etc. etc.).
It’s more “interesting” to discuss the departures:
- Claude would probably be better at Pokemon
- Maybe self-driving vehicles are more common / robust
- I think the METR plot would look different. I don’t think task length should be the bottleneck. I think that indicates that something isn’t scalable. But I am not sure.
- There would perhaps be a few novel insights or discoveries from LLMs in a few domains where they have an advantage over us. I am not convinced this is true and in fact absence of evidence is evidence of absence: https://www.lesswrong.com/posts/GADJFwHzNZKg2Ndti/have-llms-generated-novel-insights
- I would weakly expect to find AI a little more useful, but more strongly, I would expect to be finding it increasingly useful over the last couple of years, and I don’t.
- “Hallucinations” (I think @Abram Demski has a better term I can’t remember) would be easier to get rid of.
Each point is capable of shifting my opinion to a greater or lesser degree. Mainly seeing novel insights would change my mind. If METR’s results hold up that will also change my mind.

Cole Wyeth Apr 11, 2025, 2:33 PM
2 points
0
in reply to: Jonas Hallgren’s comment on: Reactions to METR task length paper are insane
Yeah.

Cole Wyeth Apr 11, 2025, 1:53 PM
7 points
2
in reply to: Rafael Harth’s comment on: Reactions to METR task length paper are insane
I remember enjoying that post (perhaps I even linked it somewhere?) and I think it’s probably the case that the inefficiency in task length scaling has to do with LLMs having only a subset of cognitive abilities available. I’m not really committed to a view on that here though.
The links don’t seem to prove that the points are “inaccurate.”

Cole Wyeth Apr 11, 2025, 1:38 PM
4 points
2
in reply to: elifland’s comment on: Reactions to METR task length paper are insane
I think it was a mistake to focus particularly on AI 2027 when the METR report does not seem to play a prominent role there.