In particular, existing AI training strategies don’t need to handle a “drastic” distribution shift from low levels of intelligence to high levels of intelligence. There’s nothing in the foreseeable ways of building AI that would call for a big transfer like this, rather than continuously training as intelligence gradually increases.
An obvious possible regime change is the shift to training (some) agents that do lifetime learning rather than only incorporating capability from SGD.
That’s one thing simple thing that seems likely to generate a sharp left turn.
I wouldn’t call that a simple thing—”lifetime learning” is a description of a goal, not an algorithm. How are these agents trained? It’s hard produce sophisticated long-horizon learning mechanisms by gradient descent using existing techniques (because you don’t many lifetimes over which to adjust such mechanisms by gradient descent). So most of the time the mechanism is something built by humans or that transfers from short tasks, and then we need to talk details.
Perhaps more importantly, why does lifetime learning go quickly from “doesn’t help meaningfully” to “and now the agents are radically superhuman”? I think at a basic level I don’t understand the mechanism of the potential sharp left turn (though I may also not fully understand what that term refers to).
An obvious possible regime change is the shift to training (some) agents that do lifetime learning rather than only incorporating capability from SGD.
That’s one thing simple thing that seems likely to generate a sharp left turn.
I wouldn’t call that a simple thing—”lifetime learning” is a description of a goal, not an algorithm. How are these agents trained? It’s hard produce sophisticated long-horizon learning mechanisms by gradient descent using existing techniques (because you don’t many lifetimes over which to adjust such mechanisms by gradient descent). So most of the time the mechanism is something built by humans or that transfers from short tasks, and then we need to talk details.
Perhaps more importantly, why does lifetime learning go quickly from “doesn’t help meaningfully” to “and now the agents are radically superhuman”? I think at a basic level I don’t understand the mechanism of the potential sharp left turn (though I may also not fully understand what that term refers to).