Agreed that the current paradigm is somewhat hard to self-improve. But note that this is a one-time cost rather than a permanent slowdown.
If an AI is better than humans at AI design, and can get the resources to experiment and train successors, it’s going to have incentives to design successors that are better at self-improvement than it is. At which point FOOM resumes apace.
Also, in the current paradigm, overhang in one part can allow for sudden progress in other parts. For example, if you have an agent with a big complicated predictive model of the world wrapped in some infrastructure that dictates how it makes plans and takes actions, then if the big complicated predictive model is powerful but the wrapping infrastructure is suboptimal, there can be sudden capability gains by optimizing the surrounding infrastructure.
It’s not clear to me that it’s necessarily possible to get to a point where a model can achieve rapid self-improvement without expensive training or experimenting. Evolution hasn’t figured out a way to substantially reduce the time and resources required for any one human’s cognitive development.
I agree that even in the current paradigm there are many paths towards sudden capability gains, like the suboptimal infrastructure scenario you pointed to. I just don’t know if I would consider that FOOM, which in my understanding implies rapid recursive self-improvement.
Maybe this is just a technicality. I expect things to advance pretty rapidly from now on with no end in sight. But before we had these huge models, FOOM with very fast recursive self-improvement seemed almost inevitable to me. Now I think that it’s possible that model size and training compute put at least some cap on the rate of self-improvement (maybe weeks instead of minutes).
Agreed that the current paradigm is somewhat hard to self-improve. But note that this is a one-time cost rather than a permanent slowdown.
If an AI is better than humans at AI design, and can get the resources to experiment and train successors, it’s going to have incentives to design successors that are better at self-improvement than it is. At which point FOOM resumes apace.
Also, in the current paradigm, overhang in one part can allow for sudden progress in other parts. For example, if you have an agent with a big complicated predictive model of the world wrapped in some infrastructure that dictates how it makes plans and takes actions, then if the big complicated predictive model is powerful but the wrapping infrastructure is suboptimal, there can be sudden capability gains by optimizing the surrounding infrastructure.
It’s not clear to me that it’s necessarily possible to get to a point where a model can achieve rapid self-improvement without expensive training or experimenting. Evolution hasn’t figured out a way to substantially reduce the time and resources required for any one human’s cognitive development.
I agree that even in the current paradigm there are many paths towards sudden capability gains, like the suboptimal infrastructure scenario you pointed to. I just don’t know if I would consider that FOOM, which in my understanding implies rapid recursive self-improvement.
Maybe this is just a technicality. I expect things to advance pretty rapidly from now on with no end in sight. But before we had these huge models, FOOM with very fast recursive self-improvement seemed almost inevitable to me. Now I think that it’s possible that model size and training compute put at least some cap on the rate of self-improvement (maybe weeks instead of minutes).