Generally the way that people solve hard problems is to solve related easy problems first, and this is true even if the technology in question gets much more powerful. Imagine if we had to land rockets on barges before anyone had invented PID controllers and observed their failure modes.
This raises questions about the reference class.
Does controlling a self-learning (and evolving) system fit in the same reference class as the problems that engineers have “generally” been able to solve (such as moving rockets)?
Is the notion of “powerful” technologies in the sense of eg. rockets being powerful the same notion as “powerful” in the sense of fully autonomous learning being powerful?
Based on this, can we rely on the reference class of past “powerful” technologies as an indicator of being able to make incremental progress on making and keeping “AGI” safe?
This raises questions about the reference class.
Does controlling a self-learning (and evolving) system fit in the same reference class as the problems that engineers have “generally” been able to solve (such as moving rockets)?
Is the notion of “powerful” technologies in the sense of eg. rockets being powerful the same notion as “powerful” in the sense of fully autonomous learning being powerful?
Based on this, can we rely on the reference class of past “powerful” technologies as an indicator of being able to make incremental progress on making and keeping “AGI” safe?