I am not convinced, at least for my own purposes, although obviously most people will be unable to come up with valuable insights here. I think salience of ideas is a big deal, people don’t do things, and yes often I get ideas that seem like they might not get discovered forever otherwise
My model is that “people don’t do things” is the bigger bottleneck on capabilities progress than “no-one’s thought of that yet”.
I’m sure there is a person in each AGI lab who has had, at some point, an idea for capability-improvement isomorphic to almost any idea an alignment researcher had (perhaps with some exceptions). But the real blockers are, “Was this person one of the people deciding the direction of company research?”, and, “If yes, do they believe in this idea enough to choose to allocate some of the limited research budget to it?”.
And the research budget appears very limited. o1 seems to be incredibly simple, so simple all the core ideas were floating around back in 2022. Yet it took perhaps a year (until the Q* rumors in November 2023) to build a proof-of-concept prototype, and two years to ship it. Making even something as straightforward-seeming as that was overwhelmingly fiddly. (Arguably it was also delayed by OpenAI researchers having to star in a cyberpunk soap opera, except what was everyone else doing?)
So making a bad call regarding what bright idea to pursue is highly costly, and there are only so many ideas you can pursue in parallel. This goes tenfold for any ideas that might only work at sufficiently big scale – imagine messing up a GPT-5-level training run because you decided to try out something daring.
But: this still does not mean you can freely share capability insights. Yes, “did an AI capability researcher somewhere ever hear of this idea?” doesn’t matter as much as you’d think. What does matter is, “is this idea being discussed widely enough to be fresh on the leading capability researchers’ minds?”. If yes, then:
They may be convinced by one of the justifications regarding why this is a good idea.
This idea may make it to the top of a leading researcher’s mind, such that they would be idly musing on it 24⁄7 until finding a variant of it/an implementation of it that they’d be willing to try.
If the idea is the talk of the town, they may not face as much reputational damage if they order R&D departments to focus on it and then it fails. (A smaller factor, but likely still in play.)
So I think avoiding discussion of potential capability insights is ever a good policy.
Edit: I. e., don’t give capability insights steam.
My model is that “people don’t do things” is the bigger bottleneck on capabilities progress than “no-one’s thought of that yet”.
I’m sure there is a person in each AGI lab who has had, at some point, an idea for capability-improvement isomorphic to almost any idea an alignment researcher had (perhaps with some exceptions). But the real blockers are, “Was this person one of the people deciding the direction of company research?”, and, “If yes, do they believe in this idea enough to choose to allocate some of the limited research budget to it?”.
And the research budget appears very limited. o1 seems to be incredibly simple, so simple all the core ideas were floating around back in 2022. Yet it took perhaps a year (until the Q* rumors in November 2023) to build a proof-of-concept prototype, and two years to ship it. Making even something as straightforward-seeming as that was overwhelmingly fiddly. (Arguably it was also delayed by OpenAI researchers having to star in a cyberpunk soap opera, except what was everyone else doing?)
So making a bad call regarding what bright idea to pursue is highly costly, and there are only so many ideas you can pursue in parallel. This goes tenfold for any ideas that might only work at sufficiently big scale – imagine messing up a GPT-5-level training run because you decided to try out something daring.
But: this still does not mean you can freely share capability insights. Yes, “did an AI capability researcher somewhere ever hear of this idea?” doesn’t matter as much as you’d think. What does matter is, “is this idea being discussed widely enough to be fresh on the leading capability researchers’ minds?”. If yes, then:
They may be convinced by one of the justifications regarding why this is a good idea.
This idea may make it to the top of a leading researcher’s mind, such that they would be idly musing on it 24⁄7 until finding a variant of it/an implementation of it that they’d be willing to try.
If the idea is the talk of the town, they may not face as much reputational damage if they order R&D departments to focus on it and then it fails. (A smaller factor, but likely still in play.)
So I think avoiding discussion of potential capability insights is ever a good policy.
Edit: I. e., don’t give capability insights steam.