I am amused that the footnotes are as long as the actual post.
Footnote 3 includes a rather salient point:
However, if you instead think that something like the typical amount of computing power available to talented researchers is what’s most important — or if you simply think that looking at the amount of computing power available to various groups can’t tell us much at all — then the OpenAI data seems to imply relatively little about future progress.
Especially in the light of this news item from Import AI #126:
The paper obtained state-of-the-art scores on lipreading, significantly exceeding prior SOTAs. It achieved this via a lot of large-scale infrastructure, combined with some elegant algorithmic tricks. But ultimately it was rejected from ICLR, with a comment from a meta-reviewer saying ‘Excellent engineering work, but it’s hard to see how others can build on it’, among other things.
It’s possible that we will see more divergence between ‘big compute’ and ‘small compute’ worlds in a way that one might expect will slow down progress (because the two worlds aren’t getting the same gains from trade that they used to).
I’m confused. Do you mean “worlds” as in “future trajectories of the world” or as in “subcommunities of AI researchers”? And what’s a concrete example of gains from trade between worlds?
Subcommunities of AI researchers. A simple concrete example of gains from trade is when everyone uses the same library or conceptual methodology, and someone finds a bug. The primary ones of interest are algorithmic gains; the new thing used to do better lipreading can also be used by other researchers to do better on other tasks (or to further enhance this approach and push it further for lipreading).
I am amused that the footnotes are as long as the actual post.
Footnote 3 includes a rather salient point:
Especially in the light of this news item from Import AI #126:
It’s possible that we will see more divergence between ‘big compute’ and ‘small compute’ worlds in a way that one might expect will slow down progress (because the two worlds aren’t getting the same gains from trade that they used to).
(For posterity: the above link is the homepage, and this is the article Vaniver referred to.)
Link fixed, thanks!
I’m confused. Do you mean “worlds” as in “future trajectories of the world” or as in “subcommunities of AI researchers”? And what’s a concrete example of gains from trade between worlds?
Subcommunities of AI researchers. A simple concrete example of gains from trade is when everyone uses the same library or conceptual methodology, and someone finds a bug. The primary ones of interest are algorithmic gains; the new thing used to do better lipreading can also be used by other researchers to do better on other tasks (or to further enhance this approach and push it further for lipreading).