The optimal strategy will likely substantially change depending on whether takeoffs happen over years or decades so it might make sense to conceptually separate these time scales.
when in practice there will be an intermediate regime where the system {the humans building the AI + the AI}
It seems that we are already in this regime, of {H,AI} and probably have been since as the system {H,AI} has existed? (although with different dynamics of growth and with a growing influence of AI helping humans )
The system {H,AI} is currently already self-improving with human using AI to make progress in AI, but one question is for example whether the system {H,AI} will keep improving at an accelerating rate until AI can foom autonomously, or will we see diminishing returns and then some re-acceleration?
More specifically, interaction between AI labor (researchers+engineers) and AI capability tech (compute, models, datasets, environments) to grow AI capability tech are the sort of models that could also be useful to make more crisp at more and check empirically.
I agree that we are already in this regime. In the section “AI Helping Humans with AI” I tried to make it more precise at what threshold we would see substantial change in how humans interact with AI to build more advanced AI systems. Essentially, it will be when most people would use those tools most of their time (like on a daily basis) and they would observe some substantial gains of productivity (like using some oracle to make a lot of progress on a problem they are stuck on, or Copilot auto-completing a lot of their lines of code without having to manually edit.) The intuition for a threshold is “most people would need to use”.
Re diminishing returns: see my other comment. In summary, if you just consider one team building AIHHAI, they would get more data and research as input from the outside world, and they would get increases in productivity from using more capable AIHHAIs. Diminishing returns could happen if: 1) scaling laws for coding AI do not hold anymore 2) we are not able to gather coding data (or do other tricks like data augmentation) at a pace high enough 3) investments for some reasons do not follow 4) there are some hardware bottlenecks in building larger and larger infrastructures. For now I have only seen evidence for 2) and this seems something that can be solved via transfer learning or new ML research.
Better modeling of those different interactions between AI labor and AI capability tech are definitely needed. For some high-level picture that mostly thinks about substitutability between capital and labor, applying to AI, I would recommend this paper (or video and slides). The equation that is the closest to self-improving {H,AI} would be this one.
The optimal strategy will likely substantially change depending on whether takeoffs happen over years or decades so it might make sense to conceptually separate these time scales.
It seems that we are already in this regime, of {H,AI} and probably have been since as the system {H,AI} has existed? (although with different dynamics of growth and with a growing influence of AI helping humans )
The system {H,AI} is currently already self-improving with human using AI to make progress in AI, but one question is for example whether the system {H,AI} will keep improving at an accelerating rate until AI can foom autonomously, or will we see diminishing returns and then some re-acceleration?
More specifically, interaction between AI labor (researchers+engineers) and AI capability tech (compute, models, datasets, environments) to grow AI capability tech are the sort of models that could also be useful to make more crisp at more and check empirically.
I agree that we are already in this regime. In the section “AI Helping Humans with AI” I tried to make it more precise at what threshold we would see substantial change in how humans interact with AI to build more advanced AI systems. Essentially, it will be when most people would use those tools most of their time (like on a daily basis) and they would observe some substantial gains of productivity (like using some oracle to make a lot of progress on a problem they are stuck on, or Copilot auto-completing a lot of their lines of code without having to manually edit.) The intuition for a threshold is “most people would need to use”.
Re diminishing returns: see my other comment. In summary, if you just consider one team building AIHHAI, they would get more data and research as input from the outside world, and they would get increases in productivity from using more capable AIHHAIs. Diminishing returns could happen if: 1) scaling laws for coding AI do not hold anymore 2) we are not able to gather coding data (or do other tricks like data augmentation) at a pace high enough 3) investments for some reasons do not follow 4) there are some hardware bottlenecks in building larger and larger infrastructures. For now I have only seen evidence for 2) and this seems something that can be solved via transfer learning or new ML research.
Better modeling of those different interactions between AI labor and AI capability tech are definitely needed. For some high-level picture that mostly thinks about substitutability between capital and labor, applying to AI, I would recommend this paper (or video and slides). The equation that is the closest to self-improving {H,AI} would be this one.