Yeah—this is a case where how exactly the transition goes seems to make a very big difference. If it’s a fast transition to a singleton, altering the goals of the initial AI is going to be super influential. But if it’s that there are many generations of AIs that over time become the larger majority of the economy, then just control everything—predictably altering how that goes seems a lot harder at least.
Comparing the entirety of the Bostrom/Yudkowsky singleton intelligence explosion scenario to the slower more spread out scenario, it’s not clear that it’s easier to predictably alter the course of the future in the first compared to the second.
In the first, assuming you successfully set the goals of the singleton, the hard part is over and the future can be steered easily because there are, by definition, no more coordination problems to deal with. But in the first, a superintelligent AGI could explode on us out of nowhere with little warning and a ‘randomly rolled utility function’, so the amount of coordination we’d need pre-intelligence explosion might be very large.
In the second slower scenario, there are still ways to influence the development of AI—aside from massive global coordination and legislation, there may well be decision points where two developmental paths are comparable in terms of short-term usefulness but one is much better than the other in terms of alignment or the value of the long-term future.
Stuart Russell’s claim that we need to replace ‘the standard model’ of AI development is one such example—if he’s right, a concerted push now by a few researchers could alter how nearly all future AI systems are developed for the better. So different conditions have to be met for it to be possible to predictably alter the future long in advance on the slow transition model (multiple plausible AI development paths that could be universally adopted and have ethically different outcomes) compared to the fast transition model (the ability to anticipate when and where the intelligence explosion will arrive and do all the necessary alignment work in time), but its not obvious to me one is easier to meet than the other.
For this reason, I think it’s unlikely there will be a very clearly distinct “takeoff period” that warrants special attention compared to surrounding periods.
I think the period AI systems can, at least in aggregate, finally do all the stuff that people can do might be relatively distinct and critical—but, if progress in different cognitive domains is sufficiently lumpy, this point could be reached well after the point where we intuitively regard lots of AI systems as on the whole “superintelligent.”
This might be another case (like ‘the AIs utility function’) where we should just retire the term as meaningless, but I think that ‘takeoff’ isn’t always a strictly defined interval, especially if we’re towards the medium-slow end. The start of the takeoff has a precise meaning only if you believe that RSI is an all-or-nothing property. In this graph from a post of mine, the light blue curve has an obvious start to the takeoff where the gradient discontinuously changes, but what about the yellow line? There clearly is a takeoff in that progress becomes very rapid, but there’s no obvious start point, but there is still a period very different from our current period that is reached in a relatively short space of time—so not ‘very clearly distinct’ but still ‘warrants special attention’.
At this point I think it’s easier to just discard the terminology altogether. For some agents, it’s reasonable to describe them as having goals. For others, it isn’t. Some of those goals are dangerous. Some aren’t.
Daniel Dennett’s Intentional stance is either a good analogy for the problem of “can’t define what has a utility function” or just a rewording of the same issue. Dennett’s original formulation doesn’t discuss different types of AI systems or utility functions, ranging in ‘explicit goal directedness’ all the way from expected-minmax game players to deep RL to purely random agents, but instead discusses physical systems ranging from thermostats up to humans. Either way, if you agree with Dennett’s formulation of the intentional stance I think you’d also agree that it doesn’t make much sense to speak of ’the utility function as necessarily well-defined.
Comparing the entirety of the Bostrom/Yudkowsky singleton intelligence explosion scenario to the slower more spread out scenario, it’s not clear that it’s easier to predictably alter the course of the future in the first compared to the second.
In the first, assuming you successfully set the goals of the singleton, the hard part is over and the future can be steered easily because there are, by definition, no more coordination problems to deal with. But in the first, a superintelligent AGI could explode on us out of nowhere with little warning and a ‘randomly rolled utility function’, so the amount of coordination we’d need pre-intelligence explosion might be very large.
In the second slower scenario, there are still ways to influence the development of AI—aside from massive global coordination and legislation, there may well be decision points where two developmental paths are comparable in terms of short-term usefulness but one is much better than the other in terms of alignment or the value of the long-term future.
Stuart Russell’s claim that we need to replace ‘the standard model’ of AI development is one such example—if he’s right, a concerted push now by a few researchers could alter how nearly all future AI systems are developed for the better. So different conditions have to be met for it to be possible to predictably alter the future long in advance on the slow transition model (multiple plausible AI development paths that could be universally adopted and have ethically different outcomes) compared to the fast transition model (the ability to anticipate when and where the intelligence explosion will arrive and do all the necessary alignment work in time), but its not obvious to me one is easier to meet than the other.
This might be another case (like ‘the AIs utility function’) where we should just retire the term as meaningless, but I think that ‘takeoff’ isn’t always a strictly defined interval, especially if we’re towards the medium-slow end. The start of the takeoff has a precise meaning only if you believe that RSI is an all-or-nothing property. In this graph from a post of mine, the light blue curve has an obvious start to the takeoff where the gradient discontinuously changes, but what about the yellow line? There clearly is a takeoff in that progress becomes very rapid, but there’s no obvious start point, but there is still a period very different from our current period that is reached in a relatively short space of time—so not ‘very clearly distinct’ but still ‘warrants special attention’.
Daniel Dennett’s Intentional stance is either a good analogy for the problem of “can’t define what has a utility function” or just a rewording of the same issue. Dennett’s original formulation doesn’t discuss different types of AI systems or utility functions, ranging in ‘explicit goal directedness’ all the way from expected-minmax game players to deep RL to purely random agents, but instead discusses physical systems ranging from thermostats up to humans. Either way, if you agree with Dennett’s formulation of the intentional stance I think you’d also agree that it doesn’t make much sense to speak of ’the utility function as necessarily well-defined.