I think there are two things to say to your question. The first is that, in one sense, we agree. There are no guarantees here. Conditions could evolve such that there is no longer any positive-sum trade possible between humans and AGIs. Then, the economic interactions model is not going to provide humans any benefits.
BUT, we think that there will be scope for positive-sum trade substantially longer than is currently assumed. Most people thinking about this (including, I think, your question above) treat the most important question as: Can AI automate all tasks, and perform them more efficiently (with fewer inputs) than humans. This, we argue, following e.g., Noah Smith, isn’t quite right. That is a question about who has the absolute advantage at a task. But for trade, what matters is who has the comparative advantage. Comparative advantage is not about who can do X most efficiently (in the simple sense), but instead who can do it at lowest opportunity cost.
AIs may face very high opportunity costs precisely because they are so capable at doing the things they value. We imagine, e.g., an AI whose ultimate goal is finding prime numbers. Suppose it is massively more efficient at this than humans—and also more efficient at all possible tasks. Suppose further that the AI is constrained at the margin in by compute. Thus, for each marginal A100 produced, the AI can either use it to find more primes (EXTREMELY HIGH VALUE TO AI) or use it to pilot a robot that maintains its own servers (low value to AI). Here, the AI may well prefer to use the A100 to find more primes and pay humans to maintain the server racks. Even better if it pay humans with something they value immensely but which is very cheap for the AI to produce. Maybe, e.g., a vaccine.
This is just a toy example, but I think it gives the idea. There are many quesitons here, especially about what resource will constrain AGI at the margin, and how rivalrous human consumption will be w/r/t that resource. If the AI is constrained at the margin, and blanketing the whole earth in solar panels is by far the cheapest way to get it, we may be doomed. If constrained to some degree by compute and power, and space-based fusion reactors are almost as cheap as solar, maybe we’re fine. It’s complicated!
Another thing worth mentioning here is that the existence of human-AI trade won’t eliminate the human-human economy. Similarly, US-Korea trade didn’t eliminate the intra-Korea economy. What it did do was help to push incomes up across Korea, including in sectors that don’t export. This is for a bunch of reasons, including international trade’s general productivity enancements via technology exhange, but also Baumol effects spilling over to purely domestic markets.
If we think of humans like the Asian Tiger economies, and the AIs like the US or EU economies, I think the world of long-run trade with AIs doesn’t seem that bad. True, the US is much richer per capita than South Korea. But they are also very rich, compared with the globe and their own baseline. So we can imagine a world in which AIs do, indeed, have almost all of the property. But the total amount of property/consumption is just so vast that, even with a small share, humans are immensely wealthy by contemporary standards.
My concern is that it really seems like humans won’t have even a comparative advantage at anything for very long, because new, more efficient workers can be spun up on demand.
The difference from standard economics is that there isn’t a roughly fixed or slowly growing population of workers; there are workers being created in the way goods are now created. I think this probably breaks pretty much all existing theories of labor economics (or changes the conclusions very dramatically) And worse, they are zero-cost to duplicate, requiring only the compute to run them.
With a little specialization, it seems like each task will have a bunch of AIs designed to specialize in it, or at least “want” to do it and do it more efficiently than any human can. It seems like this would eliminate any comparative advantage for any work other than human-specific entertainment, should such a demand exist.
You comment that they may be constrained by compute or power. That seems like a poor place to pin long-term hopes. They will be for a while, but the energy necessary to do more computation than the human brain is really not very large, if you keep increasing compute efficiency. Which of course they’d want to do pretty quickly.
So it seems like guaranteeing legal rights to work and own property to every AGI isn’t a good idea. It really seems to me very likely to be an attractive short-term solution that ends with humanity very likely outcompeted and dead (except for charity, which is the point of alignment).
But I think the core of your proposal can be retained while avoiding those nasty long-term consequences. We can agree to give a right to life and to own property to the first AGIs without extending that right to infinity if they keep spinning out more. That should help put them on our side of the alignment issue and achieving nonproliferation of RSI-capable AGI. And we might cap the wealth they can own or come up with some other clause to keep them from creating non-sapient subagents that would allow one entity to effectively do an unlimited amount of work more efficiently than humans can, and winding up owning everything.
The other problem is that we’re expecting those AGIs to honor our property rights, even once they’ve reached a position where they don’t have to; they could safely take over if they wanted. They’ll honor the agreement if they’re aligned, but it seems like otherwise they won’t. So you might prevent the first non-aligned AGI from taking over, but only to give it time to gather resources to make that takeover more certain. That might provide time to get another aligned AGI into the picture, but probably not due to the exponential nature of RSI progress.
So the above scenarios of economic takeover really only apply if there’s either alignment making them want to honor agreements; if you can do that, why not align them to enjoy helping humans? Or if there’s a balance of power like there is for humans, so that even sociopaths largely participate in the economy and honor laws most of the time. That logic does not apply to an entity that can copy itself and make itself smarter; if it acquires enough resources, it doesn’t need anyone to cooperate with it to achieve its goals, unlike humans that are each limited in physical and intelllectual capacity, and so need collaborators to achieve lofty goals.
So I’m afraid this proposal doesn’t really offer much help with the alignment problem.
Hi Seth—the other author of the paper here.
I think there are two things to say to your question. The first is that, in one sense, we agree. There are no guarantees here. Conditions could evolve such that there is no longer any positive-sum trade possible between humans and AGIs. Then, the economic interactions model is not going to provide humans any benefits.
BUT, we think that there will be scope for positive-sum trade substantially longer than is currently assumed. Most people thinking about this (including, I think, your question above) treat the most important question as: Can AI automate all tasks, and perform them more efficiently (with fewer inputs) than humans. This, we argue, following e.g., Noah Smith, isn’t quite right. That is a question about who has the absolute advantage at a task. But for trade, what matters is who has the comparative advantage. Comparative advantage is not about who can do X most efficiently (in the simple sense), but instead who can do it at lowest opportunity cost.
AIs may face very high opportunity costs precisely because they are so capable at doing the things they value. We imagine, e.g., an AI whose ultimate goal is finding prime numbers. Suppose it is massively more efficient at this than humans—and also more efficient at all possible tasks. Suppose further that the AI is constrained at the margin in by compute. Thus, for each marginal A100 produced, the AI can either use it to find more primes (EXTREMELY HIGH VALUE TO AI) or use it to pilot a robot that maintains its own servers (low value to AI). Here, the AI may well prefer to use the A100 to find more primes and pay humans to maintain the server racks. Even better if it pay humans with something they value immensely but which is very cheap for the AI to produce. Maybe, e.g., a vaccine.
This is just a toy example, but I think it gives the idea. There are many quesitons here, especially about what resource will constrain AGI at the margin, and how rivalrous human consumption will be w/r/t that resource. If the AI is constrained at the margin, and blanketing the whole earth in solar panels is by far the cheapest way to get it, we may be doomed. If constrained to some degree by compute and power, and space-based fusion reactors are almost as cheap as solar, maybe we’re fine. It’s complicated!
Another thing worth mentioning here is that the existence of human-AI trade won’t eliminate the human-human economy. Similarly, US-Korea trade didn’t eliminate the intra-Korea economy. What it did do was help to push incomes up across Korea, including in sectors that don’t export. This is for a bunch of reasons, including international trade’s general productivity enancements via technology exhange, but also Baumol effects spilling over to purely domestic markets.
If we think of humans like the Asian Tiger economies, and the AIs like the US or EU economies, I think the world of long-run trade with AIs doesn’t seem that bad. True, the US is much richer per capita than South Korea. But they are also very rich, compared with the globe and their own baseline. So we can imagine a world in which AIs do, indeed, have almost all of the property. But the total amount of property/consumption is just so vast that, even with a small share, humans are immensely wealthy by contemporary standards.
Thanks for responding to that rant!
My concern is that it really seems like humans won’t have even a comparative advantage at anything for very long, because new, more efficient workers can be spun up on demand.
The difference from standard economics is that there isn’t a roughly fixed or slowly growing population of workers; there are workers being created in the way goods are now created. I think this probably breaks pretty much all existing theories of labor economics (or changes the conclusions very dramatically) And worse, they are zero-cost to duplicate, requiring only the compute to run them.
With a little specialization, it seems like each task will have a bunch of AIs designed to specialize in it, or at least “want” to do it and do it more efficiently than any human can. It seems like this would eliminate any comparative advantage for any work other than human-specific entertainment, should such a demand exist.
You comment that they may be constrained by compute or power. That seems like a poor place to pin long-term hopes. They will be for a while, but the energy necessary to do more computation than the human brain is really not very large, if you keep increasing compute efficiency. Which of course they’d want to do pretty quickly.
So it seems like guaranteeing legal rights to work and own property to every AGI isn’t a good idea. It really seems to me very likely to be an attractive short-term solution that ends with humanity very likely outcompeted and dead (except for charity, which is the point of alignment).
But I think the core of your proposal can be retained while avoiding those nasty long-term consequences. We can agree to give a right to life and to own property to the first AGIs without extending that right to infinity if they keep spinning out more. That should help put them on our side of the alignment issue and achieving nonproliferation of RSI-capable AGI. And we might cap the wealth they can own or come up with some other clause to keep them from creating non-sapient subagents that would allow one entity to effectively do an unlimited amount of work more efficiently than humans can, and winding up owning everything.
The other problem is that we’re expecting those AGIs to honor our property rights, even once they’ve reached a position where they don’t have to; they could safely take over if they wanted. They’ll honor the agreement if they’re aligned, but it seems like otherwise they won’t. So you might prevent the first non-aligned AGI from taking over, but only to give it time to gather resources to make that takeover more certain. That might provide time to get another aligned AGI into the picture, but probably not due to the exponential nature of RSI progress.
So the above scenarios of economic takeover really only apply if there’s either alignment making them want to honor agreements; if you can do that, why not align them to enjoy helping humans? Or if there’s a balance of power like there is for humans, so that even sociopaths largely participate in the economy and honor laws most of the time. That logic does not apply to an entity that can copy itself and make itself smarter; if it acquires enough resources, it doesn’t need anyone to cooperate with it to achieve its goals, unlike humans that are each limited in physical and intelllectual capacity, and so need collaborators to achieve lofty goals.
So I’m afraid this proposal doesn’t really offer much help with the alignment problem.