A humanity that just finished coughing up a superintelligence has the potential to cough up another superintelligence, if left unchecked. Humanity alone might not stand a chance against a superintelligence, but the next superintelligence humanity builds could in principle be a problem.
That’s doubtful. A superintelligence is a much stronger, more capable builder of the next generation of superintelligences than humanity (that’s the whole idea behind foom). So what the superintelligence needs to worry about in this sense is whether the next generations of superintelligences it itself produces are compatible with its values and goals (“self-alignment”).
It does not seem likely that humanity on its own (without tight alliances with already existing superintelligent AIs) will be competitive in this sense.
But this example shows that we should separate the problems of Friendliness of strongly superintelligent AIs and the problems of the period of transition to superintelligence (when things are more uncertain).
The first part of the post is relevant to the period of strongly superintelligent AIs, but this example can only be relevant to the period of transition and incomplete dominance of AIs.
The more I think about all this, the more it seems to me that problems of having positive rather than sharply negative period of strong superintelligence and problems of safely navigating the transition period are very different and we should not conflate them.
Why does the fact that a superintelligence needs to solve the alignment problem for its own sake (to safely build its own successors) mean that humans building other superintelligences wouldn’t be a problem for it? It’s possible to have more than one problem at a time.
It’s possible, but I think it would require a modified version of the “low ceiling conjecture” to be true.
The standard “low ceiling conjecture” says that human-level intelligence is the hard (or soft) limit, and therefore it will be impossible (or would take a very long period of time) to move from human-level AI to superintelligence. I think most of us tend not to believe that.
A modified version would keep the hard (or soft) limit, but would raise it slightly, so that rapid transition to superintelligence is possible, but the resulting superintelligence can’t run away fast in terms of capabilities (no near-term “intelligence explosion”). If one believes this modified version of the “low ceiling conjecture”, then subsequent AIs produced by humanity might indeed be relevant.
That’s doubtful. A superintelligence is a much stronger, more capable builder of the next generation of superintelligences than humanity (that’s the whole idea behind foom). So what the superintelligence needs to worry about in this sense is whether the next generations of superintelligences it itself produces are compatible with its values and goals (“self-alignment”).
It does not seem likely that humanity on its own (without tight alliances with already existing superintelligent AIs) will be competitive in this sense.
But this example shows that we should separate the problems of Friendliness of strongly superintelligent AIs and the problems of the period of transition to superintelligence (when things are more uncertain).
The first part of the post is relevant to the period of strongly superintelligent AIs, but this example can only be relevant to the period of transition and incomplete dominance of AIs.
The more I think about all this, the more it seems to me that problems of having positive rather than sharply negative period of strong superintelligence and problems of safely navigating the transition period are very different and we should not conflate them.
Why does the fact that a superintelligence needs to solve the alignment problem for its own sake (to safely build its own successors) mean that humans building other superintelligences wouldn’t be a problem for it? It’s possible to have more than one problem at a time.
It’s possible, but I think it would require a modified version of the “low ceiling conjecture” to be true.
The standard “low ceiling conjecture” says that human-level intelligence is the hard (or soft) limit, and therefore it will be impossible (or would take a very long period of time) to move from human-level AI to superintelligence. I think most of us tend not to believe that.
A modified version would keep the hard (or soft) limit, but would raise it slightly, so that rapid transition to superintelligence is possible, but the resulting superintelligence can’t run away fast in terms of capabilities (no near-term “intelligence explosion”). If one believes this modified version of the “low ceiling conjecture”, then subsequent AIs produced by humanity might indeed be relevant.