Some central reasons the terrain looks thresholdy to me:
Science often comes with “click” moments, where many things slide into place and start making sense.
As we enter the ‘AI can do true science’ regime, it becomes important that AI can unlock new technologies (both cognitive/AI technologies, and other impactful technologies), new scientific disciplines and subdisciplines, new methodologies and ways of doing intellectual inquiry, etc.
‘The ability to invent new technologies’ and ‘the ability to launch into new scientific fields/subfields’, including ones that may not even be on our radar today (whether or not they’re ‘hard’ in an absolute sense — sometimes AI will just think differently from us), is inherently thresholdy, because ‘starting or creating an entirely new thing’ is a 0-to-1 change, more so than ‘incrementally improving on existing technologies and subdisciplines’ tends to be.
Many of these can also use one discovery/innovation to reach other discoveries/innovations, increasing the thresholdiness. (An obvious example of this is RSI, but AI can also just unlock a scientific subdiscipline that chains into a bunch of new discoveries, leads to more new subdisciplines, etc.)
Empirically, humans did not need to evolve separate specialized-to-the-field modules in order to be able to do biotechnology as well as astrophysics as well as materials science as well as economics as well as topology. Some combination of ‘human-specific machinery’ and ‘machinery that precedes humans’ sufficed to do all the sciences (that we know of), even though those fields didn’t exist in the environment our brain was being built in. Thus, general intelligence is a thing; you can figure out how to do AI in such a way that once you can do one science, you have the machinery in hand to do all the other sciences.
Empirically, all of these fields sprang into existence almost simultaneously for humans, within the space of a few decades or centuries. So in addition to the general points above about “clicks are a thing” and “starting new fields and inventing new technologies is threshold-y”, it’s also the case that AGI is likely to unlock all of the sciences simultaneously in much the same way humans did.
That one big “click” moment, that unlocks all the other click moments and new sciences/technologies and sciences-and-technologies-that-chain-off-of-those-sciences-and-technologies, implies that many different thresholds are likely to get reached at the same time.
Which increases the probability that even if one specific threshold wouldn’t have been crazily high-impact on its own, the aggregate effect of many of those thresholds at once does end up crazily high-impact.
you can figure out how to do AI in such a way that once you can do one science, you have the machinery in hand to do all the other sciences
And indeed, I would be extremely surprised if we find a way to do AI that only lets you build general-purpose par-human astrophysics AI, but doesn’t also let you build general-purpose par-human biochemistry AI.
(There may be an AI technique like that in principle, but I expect it to be a very weird technique you’d have to steer toward on purpose; general techniques are a much easier way to build science AI. So I don’t think that the first general-purpose astrophysics AI system we build will be like that, in the worlds where we build general-purpose astrophysics AI systems.)
Some central reasons the terrain looks thresholdy to me:
Science often comes with “click” moments, where many things slide into place and start making sense.
As we enter the ‘AI can do true science’ regime, it becomes important that AI can unlock new technologies (both cognitive/AI technologies, and other impactful technologies), new scientific disciplines and subdisciplines, new methodologies and ways of doing intellectual inquiry, etc.
‘The ability to invent new technologies’ and ‘the ability to launch into new scientific fields/subfields’, including ones that may not even be on our radar today (whether or not they’re ‘hard’ in an absolute sense — sometimes AI will just think differently from us), is inherently thresholdy, because ‘starting or creating an entirely new thing’ is a 0-to-1 change, more so than ‘incrementally improving on existing technologies and subdisciplines’ tends to be.
Many of these can also use one discovery/innovation to reach other discoveries/innovations, increasing the thresholdiness. (An obvious example of this is RSI, but AI can also just unlock a scientific subdiscipline that chains into a bunch of new discoveries, leads to more new subdisciplines, etc.)
Empirically, humans did not need to evolve separate specialized-to-the-field modules in order to be able to do biotechnology as well as astrophysics as well as materials science as well as economics as well as topology. Some combination of ‘human-specific machinery’ and ‘machinery that precedes humans’ sufficed to do all the sciences (that we know of), even though those fields didn’t exist in the environment our brain was being built in. Thus, general intelligence is a thing; you can figure out how to do AI in such a way that once you can do one science, you have the machinery in hand to do all the other sciences.
Empirically, all of these fields sprang into existence almost simultaneously for humans, within the space of a few decades or centuries. So in addition to the general points above about “clicks are a thing” and “starting new fields and inventing new technologies is threshold-y”, it’s also the case that AGI is likely to unlock all of the sciences simultaneously in much the same way humans did.
That one big “click” moment, that unlocks all the other click moments and new sciences/technologies and sciences-and-technologies-that-chain-off-of-those-sciences-and-technologies, implies that many different thresholds are likely to get reached at the same time.
Which increases the probability that even if one specific threshold wouldn’t have been crazily high-impact on its own, the aggregate effect of many of those thresholds at once does end up crazily high-impact.
And indeed, I would be extremely surprised if we find a way to do AI that only lets you build general-purpose par-human astrophysics AI, but doesn’t also let you build general-purpose par-human biochemistry AI.
(There may be an AI technique like that in principle, but I expect it to be a very weird technique you’d have to steer toward on purpose; general techniques are a much easier way to build science AI. So I don’t think that the first general-purpose astrophysics AI system we build will be like that, in the worlds where we build general-purpose astrophysics AI systems.)