I’m still a bit puzzled by the link between human level on text prediction and ‘human level’ unconditionally—if I recall our near-bet during the forecasting tournament, our major disagreement was on whether direct scaling of GPT like systems takes us near to AGI. I often think that (because we don’t have direct experience with any verbal intelligences in capability between GPT-3 and human brains) we’re often impoverished when trying to think about such intelligences. I imagine that a GPT-6 that is almost ‘human level on text prediction’ could still be extremely deficient in other areas—it would be very weird to converse with, maybe like an amnesiac or confabulator that’s very articulate and with good short-term memory.
If language models scale to near-human performance but the other milestones don’t fall in the process, and my initial claim is right, that gives us very transformative AI but not AGI. I think that the situation would look something like this:
So there would be 2 (maybe 3?) breakthroughs remaining. It seems like you think just scaling up a GPT will also resolve those other milestones, rather than just giving us human-like language comprehension. Whereas if I’m right and also those curves do extrapolate, what we would get at the end would be an excellent text generator, but it wouldn’t be an agent, wouldn’t be capable of long-term planning and couldn’t be accurately described as having a utility function over the states of the external world, and I don’t see any reason why trivial extensions of GPT would be able to do that either since those seem like problems that are just as hard as human-like language comprehension. GPT seems like it’s also making some progress on cumulative learning, though it might need some RL-based help with that, but none at all on managing mental activity for longterm planning or discovering new action sets.
I agree with you that human level on text prediction is not the same as human level generally, and that we might get the former long before we get the latter. Nevertheless, I think it’s possible that the two will come together—or, more cautiously, that something human-level at task prediction would also be transformative. (Maybe the text predictor by itself wouldn’t be an agent, but the text predictor could be re-trained as an agent fairly easily, or combined into a larger system that uses tree search or something and thus is an agent. Maybe it would be good enough at enough things that it would make a ridiculous amount of money automating away many jobs and inspire a huge boost in investment, which would then lead to human-level AGI. Or maybe it would be really good at persuasion… I’m going to write a post about persuasion tools someday soon.)
I think that it could plausibly be quite transformative in a TAI sense and occur over the next ten years, so perhaps we don’t have all that much of a disagreement on that point. I also think (just because we don’t have an especially clear idea of how modular intelligence is) that it could be quite uniform and a text predictor could surprise us with humanlike planning.
Maybe the text predictor by itself wouldn’t be an agent, but the text predictor could be re-trained as an agent fairly easily, or combined into a larger system that uses tree search or something and thus is an agent.
This maybe reflects a difference in intuition about how difficult agentive behaviour is to reach rather than language understanding. I would expect a simple tree search algorithm powered by GPT-6 to be… a model with humanlike language comprehension and incredibly dumb agentive behaviour, and that it wouldn’t be able to leverage the ‘intelligence’ of the language model in any significant way, because I see that as a seperate problem requiring seperate, difficult work. But I could be wrong.
I think there is a potential bias in that human-like language understanding and agentive behaviour have always gone together in human beings—we have no idea what a human-level language model that wasn’t human-level intelligent would be like. Since we can’t imagine it, we tend to default to imagining a human-in-a-box. I’m trying to correct for this bias by imagining that it might be quite different.
Nice, that’s probably a crux for us. I would expect tree search powered by GPT-6 to be probably pretty agentic. Isn’t that how AlphaZero works? Tree search + a win probability predictor?
It may well be a crux—an efficient ‘tree search’ or a similar goal-directed wrapper around a GPT-based system, that can play a role in real-world open-ended planning (presumably planning for an agent to be effecting outcomes in the real world via its text generation), would have to cover continuous action spaces and possible states containing unknown and shifting sets of possible actions (unlike the discrete and small, relative to the real universe, action space of Go which is perfect for a tree search), running (or approximating running) millions of primitive steps (individual text generations and exchanges) into the future (for long-term planning towards e.g. a multi-decade goal like humans are capable of).
That sounds like a problem that’s at least as hard as a language-model ‘success probability predictor’ GPT-N (probably with reward-modelling help, so it can optimize for a specific goal with its text generation). Though such a system would still be highly transformative, if it was human-level at prediction.
To clarify, this is Transformative not ‘Radically Transformative’ - transformative like Nuclear Power/Weapons, not like a new Industrial Revolution or an intelligence explosion.
I would expect tree search powered by GPT-6 to be probably pretty agentic.
I could imagine (if you found a domain with a fairly constrained set of actions and states, but involved text prediction somehow) that you could get agentic behaviour out of a tree search like the ones we currently have + GPT-N + an RL wrapper around the GPT-N. That might well be quite transformative—could imagine it being very good for persuasion, for example.
I’m still a bit puzzled by the link between human level on text prediction and ‘human level’ unconditionally—if I recall our near-bet during the forecasting tournament, our major disagreement was on whether direct scaling of GPT like systems takes us near to AGI. I often think that (because we don’t have direct experience with any verbal intelligences in capability between GPT-3 and human brains) we’re often impoverished when trying to think about such intelligences. I imagine that a GPT-6 that is almost ‘human level on text prediction’ could still be extremely deficient in other areas—it would be very weird to converse with, maybe like an amnesiac or confabulator that’s very articulate and with good short-term memory.
If language models scale to near-human performance but the other milestones don’t fall in the process, and my initial claim is right, that gives us very transformative AI but not AGI. I think that the situation would look something like this:
If GPT-N reaches par-human:
So there would be 2 (maybe 3?) breakthroughs remaining. It seems like you think just scaling up a GPT will also resolve those other milestones, rather than just giving us human-like language comprehension. Whereas if I’m right and also those curves do extrapolate, what we would get at the end would be an excellent text generator, but it wouldn’t be an agent, wouldn’t be capable of long-term planning and couldn’t be accurately described as having a utility function over the states of the external world, and I don’t see any reason why trivial extensions of GPT would be able to do that either since those seem like problems that are just as hard as human-like language comprehension. GPT seems like it’s also making some progress on cumulative learning, though it might need some RL-based help with that, but none at all on managing mental activity for longterm planning or discovering new action sets.
I agree with you that human level on text prediction is not the same as human level generally, and that we might get the former long before we get the latter. Nevertheless, I think it’s possible that the two will come together—or, more cautiously, that something human-level at task prediction would also be transformative. (Maybe the text predictor by itself wouldn’t be an agent, but the text predictor could be re-trained as an agent fairly easily, or combined into a larger system that uses tree search or something and thus is an agent. Maybe it would be good enough at enough things that it would make a ridiculous amount of money automating away many jobs and inspire a huge boost in investment, which would then lead to human-level AGI. Or maybe it would be really good at persuasion… I’m going to write a post about persuasion tools someday soon.)
I think that it could plausibly be quite transformative in a TAI sense and occur over the next ten years, so perhaps we don’t have all that much of a disagreement on that point. I also think (just because we don’t have an especially clear idea of how modular intelligence is) that it could be quite uniform and a text predictor could surprise us with humanlike planning.
This maybe reflects a difference in intuition about how difficult agentive behaviour is to reach rather than language understanding. I would expect a simple tree search algorithm powered by GPT-6 to be… a model with humanlike language comprehension and incredibly dumb agentive behaviour, and that it wouldn’t be able to leverage the ‘intelligence’ of the language model in any significant way, because I see that as a seperate problem requiring seperate, difficult work. But I could be wrong.
I think there is a potential bias in that human-like language understanding and agentive behaviour have always gone together in human beings—we have no idea what a human-level language model that wasn’t human-level intelligent would be like. Since we can’t imagine it, we tend to default to imagining a human-in-a-box. I’m trying to correct for this bias by imagining that it might be quite different.
Nice, that’s probably a crux for us. I would expect tree search powered by GPT-6 to be probably pretty agentic. Isn’t that how AlphaZero works? Tree search + a win probability predictor?
It may well be a crux—an efficient ‘tree search’ or a similar goal-directed wrapper around a GPT-based system, that can play a role in real-world open-ended planning (presumably planning for an agent to be effecting outcomes in the real world via its text generation), would have to cover continuous action spaces and possible states containing unknown and shifting sets of possible actions (unlike the discrete and small, relative to the real universe, action space of Go which is perfect for a tree search), running (or approximating running) millions of primitive steps (individual text generations and exchanges) into the future (for long-term planning towards e.g. a multi-decade goal like humans are capable of).
That sounds like a problem that’s at least as hard as a language-model ‘success probability predictor’ GPT-N (probably with reward-modelling help, so it can optimize for a specific goal with its text generation). Though such a system would still be highly transformative, if it was human-level at prediction.
To clarify, this is Transformative not ‘Radically Transformative’ - transformative like Nuclear Power/Weapons, not like a new Industrial Revolution or an intelligence explosion.
I could imagine (if you found a domain with a fairly constrained set of actions and states, but involved text prediction somehow) that you could get agentic behaviour out of a tree search like the ones we currently have + GPT-N + an RL wrapper around the GPT-N. That might well be quite transformative—could imagine it being very good for persuasion, for example.