while this paradigm of ‘training a model that’s an agi, and then running it at inference’ is one way we get to transformative agi, i find myself thinking that probably WON’T be the first transformative AI, because my guess is that there are lots of tricks using lots of compute at inference to get not quite transformative ai to transformative ai.
my guess is that getting to that transformative level is gonna require ALL the tricks and compute, and will therefore eek out being transformative BY utilizing all those resources.
one of those tricks may be running millions of copies of the thing in an agentic swarm, but i would expect that to be merely a form of inference time scaling, and therefore wouldn’t expect ONE of those things to be transformative AGI on it’s own.
and i doubt that these tricks can funge against train time compute, as you seem to be assuming in your analysis. my guess is that you hit diminishing returns for various types of train compute, then diminishing returns for various types of inference compute, and that we’ll get to a point where we need to push both of them to that point to get tranformative ai
Okay, so I am inclined to agree with Matt that the scenario of “crazy inefficient hacks burning absurd amounts of inference compute” would likely be a good description of the very first ever instance of an AGI.
However!
How long would that situation last? I expect, not long enough to be strategically relevant enough to include in a forecast like this one. If such inefficiencies in inference compute are in place, and the system was trained on and is running on many orders of magnitude more compute than the human brain runs on… Surely there’s a huge amount of low-hanging fruit which the system itself will be able to identify to render itself more efficient. Thus, in just a few hours or days you should expect a rapid drop in this inefficiency, until the low-hanging fruit is picked and you end up closer to the estimates in the post.
If this is correct, then the high-inefficiency-initial-run is mainly relevant for informing the search space of the frontier labs for scaffolding experiments.
Why do you imagine this? I imagine we’d get something like one Einstein from such a regime, which would maybe increase the timelines over existing AI labs by 1.2x or something? Eventually this gain compounds but I imagine that could tbe relatively slow and smooth , with the occasional discontinuous jump when something truly groundbreaking is discovered
I’m not sure how to answer this in a succinct way. I have rather a lot of ideas on the subject, including predictions about several likely ways components x/y/z may materialize. I think one key piece I’d highlight is that there’s a difference between:
coming up with a fundamental algorithmic insight that then needs not only experiments to confirm but also a complete retraining of the base model to take advantage of
coming up with other sorts of insights that offer improvements to the inference scaffolding or adaptability of the base model, which can be rapidly and cheaply experimented on without needing to retrain the base model.
It sounds to me that the idea of scraping together a system roughly equivalent to an Albert Einstein (or Ilya Sutskever or Geoffrey Hinton or John von Neumann) would put us in a place where there were improvements that the system itself could seek in type 1 or type 2. The trajectory you describe around gradually compounding gains sounds like what I imagine type 1 to look like in a median case. I think there’s also some small chance for getting a lucky insight and having a larger type 1 jump forwards. More importantly for expected trajectories is that I expect type 2 insights to have a very rapid feedback cycle, and thus even while having a relatively smooth incremental improvement curve the timeline for substantial improvements would be better measured in days than in years.
Does that make sense? Am I interpreting you correctly?
I still don’t quite get it. We already have an Ilya Sutskever who can make type 1 and type 2 improvements, and don’t see the sort of jump’s in days your talking about (I mean, maybe we do, and they just look discontinuous because of the release cycles?)
while this paradigm of ‘training a model that’s an agi, and then running it at inference’ is one way we get to transformative agi, i find myself thinking that probably WON’T be the first transformative AI, because my guess is that there are lots of tricks using lots of compute at inference to get not quite transformative ai to transformative ai.
Agreed that this is far from the only possibility, and we have some discussion of increasing inference time to make the final push up to generality in the bit beginning “If general intelligence is achievable by properly inferencing a model with a baseline of capability that is lower than human-level...” We did a bit more thinking around this topic which we didn’t think was quite core to the post, so Connor has written it up on his blog here: https://arcaderhetoric.substack.com/p/moravecs-sea
and i doubt that these tricks can funge against train time compute, as you seem to be assuming in your analysis.
Our method 5 is intended for this case—we’d use an appropriate ‘capabilities per token’ multiplier to account for needing extra inference time to reach human level.
while this paradigm of ‘training a model that’s an agi, and then running it at inference’ is one way we get to transformative agi, i find myself thinking that probably WON’T be the first transformative AI, because my guess is that there are lots of tricks using lots of compute at inference to get not quite transformative ai to transformative ai.
my guess is that getting to that transformative level is gonna require ALL the tricks and compute, and will therefore eek out being transformative BY utilizing all those resources.
one of those tricks may be running millions of copies of the thing in an agentic swarm, but i would expect that to be merely a form of inference time scaling, and therefore wouldn’t expect ONE of those things to be transformative AGI on it’s own.
and i doubt that these tricks can funge against train time compute, as you seem to be assuming in your analysis. my guess is that you hit diminishing returns for various types of train compute, then diminishing returns for various types of inference compute, and that we’ll get to a point where we need to push both of them to that point to get tranformative ai
Okay, so I am inclined to agree with Matt that the scenario of “crazy inefficient hacks burning absurd amounts of inference compute” would likely be a good description of the very first ever instance of an AGI.
However!
How long would that situation last? I expect, not long enough to be strategically relevant enough to include in a forecast like this one. If such inefficiencies in inference compute are in place, and the system was trained on and is running on many orders of magnitude more compute than the human brain runs on… Surely there’s a huge amount of low-hanging fruit which the system itself will be able to identify to render itself more efficient. Thus, in just a few hours or days you should expect a rapid drop in this inefficiency, until the low-hanging fruit is picked and you end up closer to the estimates in the post.
If this is correct, then the high-inefficiency-initial-run is mainly relevant for informing the search space of the frontier labs for scaffolding experiments.
Why do you imagine this? I imagine we’d get something like one Einstein from such a regime, which would maybe increase the timelines over existing AI labs by 1.2x or something? Eventually this gain compounds but I imagine that could tbe relatively slow and smooth , with the occasional discontinuous jump when something truly groundbreaking is discovered
I’m not sure how to answer this in a succinct way. I have rather a lot of ideas on the subject, including predictions about several likely ways components x/y/z may materialize. I think one key piece I’d highlight is that there’s a difference between:
coming up with a fundamental algorithmic insight that then needs not only experiments to confirm but also a complete retraining of the base model to take advantage of
coming up with other sorts of insights that offer improvements to the inference scaffolding or adaptability of the base model, which can be rapidly and cheaply experimented on without needing to retrain the base model.
It sounds to me that the idea of scraping together a system roughly equivalent to an Albert Einstein (or Ilya Sutskever or Geoffrey Hinton or John von Neumann) would put us in a place where there were improvements that the system itself could seek in type 1 or type 2. The trajectory you describe around gradually compounding gains sounds like what I imagine type 1 to look like in a median case. I think there’s also some small chance for getting a lucky insight and having a larger type 1 jump forwards. More importantly for expected trajectories is that I expect type 2 insights to have a very rapid feedback cycle, and thus even while having a relatively smooth incremental improvement curve the timeline for substantial improvements would be better measured in days than in years.
Does that make sense? Am I interpreting you correctly?
I still don’t quite get it. We already have an Ilya Sutskever who can make type 1 and type 2 improvements, and don’t see the sort of jump’s in days your talking about (I mean, maybe we do, and they just look discontinuous because of the release cycles?)
Agreed that this is far from the only possibility, and we have some discussion of increasing inference time to make the final push up to generality in the bit beginning “If general intelligence is achievable by properly inferencing a model with a baseline of capability that is lower than human-level...” We did a bit more thinking around this topic which we didn’t think was quite core to the post, so Connor has written it up on his blog here: https://arcaderhetoric.substack.com/p/moravecs-sea
Our method 5 is intended for this case—we’d use an appropriate ‘capabilities per token’ multiplier to account for needing extra inference time to reach human level.