In general, this post has prompted me to think more about the transition period between AI that’s weaker than humans and stronger than all of human civilization, and that’s been interesting! A lot of people assume that that takeoff will happen very quickly, but if it lasts for multiple years (or even decades) then the dynamics of that transition period could matter a lot, and trade is one aspect of that.
some stray thoughts on what that transition period could look like:
Some doomy-feeling states don’t immediately kill us. We might get an AI that’s able to defeat humanity before it’s able to cheaply replicate lots of human labor, because it gets a decisive strategic advantage via specialized skill in some random domain and can’t easily skill itself up in other domains.
When would an AI prefer to trade rather than coerce or steal?
maybe if the transition period is slow, and it knows it’s in the earlier part of the period, so reputation matters
maybe if it’s being cleverly watched or trained by the org building it, since they want to avoid bad press
maybe there’s some core of values you can imprint that leads to this? but maybe actually being able to solve this issue is basically equivalent to solving alignment, in which case you might as well do that.
In a transition period, powerful human orgs would find various ways to interface with AI and vice versa, since they would be super useful tools / partners for each other. Even if the transition period is short, it might be long enough to change things, e.g. by getting the world’s most powerful actors interested in building + using AI and not leaving it in the hands of a few AGI labs, by favoring labs that build especially good interfaces & especially valuable services, etc. (While in a world with a short take off rather than a long transition period, maybe big tech & governments don’t recognize what’s happening before ASI / doom.)
In general, this post has prompted me to think more about the transition period between AI that’s weaker than humans and stronger than all of human civilization, and that’s been interesting! A lot of people assume that that takeoff will happen very quickly, but if it lasts for multiple years (or even decades) then the dynamics of that transition period could matter a lot, and trade is one aspect of that.
some stray thoughts on what that transition period could look like:
Some doomy-feeling states don’t immediately kill us. We might get an AI that’s able to defeat humanity before it’s able to cheaply replicate lots of human labor, because it gets a decisive strategic advantage via specialized skill in some random domain and can’t easily skill itself up in other domains.
When would an AI prefer to trade rather than coerce or steal?
maybe if the transition period is slow, and it knows it’s in the earlier part of the period, so reputation matters
maybe if it’s being cleverly watched or trained by the org building it, since they want to avoid bad press
maybe there’s some core of values you can imprint that leads to this? but maybe actually being able to solve this issue is basically equivalent to solving alignment, in which case you might as well do that.
In a transition period, powerful human orgs would find various ways to interface with AI and vice versa, since they would be super useful tools / partners for each other. Even if the transition period is short, it might be long enough to change things, e.g. by getting the world’s most powerful actors interested in building + using AI and not leaving it in the hands of a few AGI labs, by favoring labs that build especially good interfaces & especially valuable services, etc. (While in a world with a short take off rather than a long transition period, maybe big tech & governments don’t recognize what’s happening before ASI / doom.)