Why would a coalition look very different from the world economy, be controlled by a few people, and be hard to form? My default expectation is that it would look much like the world economy. (With the most obvious changes being a fall in the labor share of income / increasing wage inequality.) A few big underlying disagreements:
I don’t think I agree that most progress in AI is driven by rare smart individuals talking to each other—I think it’s not very accurate as a description of current AI progress, that it will be even less true as AI progress becomes a larger share of the world economy, and that most “AI” progress is driven by compute/energy/data/other software rather than stuff that most looks like insight-driven AI progress.
Your toy model seems wrong: most projects make extensive use of other people’s private innovations, by trading with them. So the project that hoards the most innovations can still only be competitive if it trades with others (in order to get access to their hoarded innovations).
I think my more basic complaint is with the “pressure to make a profit over some timescale” model. It think it’s more like: you need inputs from the rest of the economy and so you trade with them. Right now deep learning moonshots don’t trade with the rest of the world because they don’t make anything of much value, but if they were creating really impactful technology then the projects which traded would be radically faster than the projects which just used their innovations in house. This is true even if all innovations are public, since they need access to physical capital.
I think any of these would be enough to carry my objection. (Though if you reject my first claim, and thought that rare smart individuals drove AI progress even then AI progress was overwhelmingly economically important, then you could imagine a sufficiently well-coordinated cartel of those rare smart individuals having a DSA.)
Thanks for the detailed reply! I should say at this point that I’m not particularly confident in my views on this topic; just trying to put forth my take on things in the spirit of improvement. So I wouldn’t be surprised if I end up thinking you are right after more discussion.
A coalition strong enough to prevent the world’s leading project from maintaining and lengthening its lead would need to have some way of preventing the leading project from accessing the innovations of the coalition. Otherwise the leading project will free-ride off the research done by the coalition. For this reason I think that a coalition would look very different from the world economy; in order to prevent the leading project from accessing innovations deployed in the world economy you would need to have an enforced universal embargo on them pretty much, and if you have that much political power, why stop there? Why not just annex them or shut them down?
A successful coalition (that isn’t politically powerful enough to embargo or annex or stop their rival) would need to be capable of preventing information from leaking out to the rival project, and that suggests to me that they would need to concentrate power in the hands of a few individuals (the CEOs and boards of the companies in the coalitions, for example). A distributed, more anarchic architecture would not be able to prevent leaks and spies.
And of course, even this hypothetical coalition would be hard to form, for all the usual reasons. Sure, the 2nd through 10th projects could gang up to decisively beat the leading project. But it’s also true that the 2nd through 10th most powerful nation-states could gang up to decisively beat the most powerful nation-state. Yet this isn’t the norm. More concretely, I expect some projects to ally with each other, but the result to be two or three coalitions of almost equal strength rather than many.
I agree that we seem to disagree about the importance of compute/energy/data vs. smart people talking to each other, and that this disagreement seems relevant. If AI progress was just a matter of compute, for example, then… well actually mightn’t there still be a decisive strategic advantage in that case? Wouldn’t one project have more compute than the others, and thus pull ahead so long as funds lasted?
This gets us into the toy model & its problems. I don’t think I understand your alternative model. I maybe don’t get what you mean by trading. Does one party giving money to another party in return for access to their technology or products count? If so, then I think my original model still stands: The leading project will be able to hoard technology/innovation and lengthen its lead over the rest of the world so long as it still has funding to buy the necessary stuff. I agree that it will be burning money fast if it doesn’t sell/trade its innovations and instead tries to hoard them, but empirically it seems that it’s quite possible for leading projects to go several years in this state.
“Right now deep learning moonshots don’t trade with the rest of the world because they don’t make anything of much value, but if they were creating really impactful technology then the projects which traded would be radically faster than the projects which just used their innovations in house.”
I think it depends not on how impactful their technology is but on how impactful their technology is relative to the perceived impact of hoarding it and going for a decisive strategic advantage. Technology hoarding happens sometimes even in very competitive industries for this reason, and it is the norm among militaries. It seems very possible to me that companies which are producing truly astounding AI technologies—stuff that seems plausibly only a few years away from human-level AGI—will have no problem finding deep-pocketed investors willing to throw money at them for a few years. Again, maybe governments will get involved, in which case this is almost trivial.
I think overall your model (as I understand it) is that people won’t be (successfully) moonshotting for AGI until AI progress is already making the economy grow very fast, and also at this point progress towards AGI will mostly be a matter of how much money you have to buy compute and stuff. So even a deep-pocketed funder like Google wouldn’t be able to compete for two years with shallow-pocketed, more short-sighted projects that sell their tech and reinvest the profits. Is this your view?
Wouldn’t one project have more compute than the others, and thus pull ahead so long as funds lasted?
To have “more compute than all the others” seems to require already being a large fraction of all the world’s spending (since a large fraction of spending is on computers—or whatever bundle of inputs is about to let this project take over the world—unless you are positing a really bad mispricing). At that point we are talking “coalition of states” rather than “project.”
I totally agree that it wouldn’t be crazy for a major world power to pull ahead of others technologically and eventually be able to win a war handily, and that will tend happen over shorter and shorter timescales if economic and technological progress accelerate.
(Or you might think the project is a small fraction of world compute but larger than any other project, but if economies of scale are in fact this critical, then you are again suggesting a really gigantic market failure. That’s not beyond the pale, but we should be focusing on why this crazy market failure is happening.)
OK, so you agree with me about major world powers (nation-states) but still disagree about companies? I think this means we are closer together than it seemed, because I also think that decisive strategic advantage is significantly more likely to happen if a nation-state gets involved than if it’s just some private company.
I didn’t say “more compute than all the others,” I said “more compute than the others,” by which I meant more compute than any particular other project, yeah. This is consistent with a large fraction of the world’s spending being on compute already. For example, today Deepmind (citation needed) has the largest compute budget of any AI project, but their compute budget is a tiny fraction of the world’s total.
I’m not sure whether or not I’m positing a gigantic market failure. Your claim is that if compute is so important for AI technology and AI technology is so useful, the market will either fail or find ways to get a large fraction of its budget spent on a single AI project? This single project would then be a potential source of DSA but it would also be so big already that it could take over the world by selling products instead? I’m putting question marks not out of sarcasm or anything, just genuine uncertainty about what your claim is. Before I can respond to it I need to understand it.
This gets us into the toy model & its problems. I don’t think I understand your alternative model. I maybe don’t get what you mean by trading. Does one party giving money to another party in return for access to their technology or products count? If so, then I think my original model still stands: The leading project will be able to hoard technology/innovation and lengthen its lead over the rest of the world so long as it still has funding to buy the necessary stuff.
The reason I let other people use my IP is because they pay me money, with which I can develop even more IP. If the leading project declines to do this, then it will have less IP than any of its normal competitors. If the leading project’s IP allows it to be significantly more productive than everyone else, then they could have just taken over the world through the normal mechanism of selling products. (Modulo leaks/spying.) As far as I can tell, until you are a large fraction of the world, the revenue you get from selling lets you grow faster, and I don’t think the toy model really undermines that typical argument (which has to go through leaks/spying, market frictions, etc.).
I am skeptical that selling products is sufficient to take over the world, no matter how good the products are. Eventually you raise too much attention and get nationalized or taxed or copied.
In light of your critiques I intend to write a much better version of this post in the future. Thanks! I wrote this one during MSFP as part of their blog post day event, so it was kinda rushed and has lots of room for improvement. I’m very glad to see so much engagement though; it inspires me to make said improvements. Perhaps in the course of doing so I’ll change my mind.
A coalition strong enough to prevent the world’s leading project from maintaining and lengthening its lead would need to have some way of preventing the leading project from accessing the innovations of the coalition. Otherwise the leading project will free-ride off the research done by the coalition. For this reason I think that a coalition would look very different from the world economy; in order to prevent the leading project from accessing innovations deployed in the world economy you would need to have an enforced universal embargo on them pretty much, and if you have that much political power, why stop there? Why not just annex them or shut them down?
Are you saying that the leading project can easily spy on other projects, but other projects can’t spy on it? Is this because the rest of the world is trading with each other, and trading opens up opportunities for spying? Some other reason I missed? I don’t think it’s usually the case that gains from rabbit-holing, in terms of protection from spying, are large enough to outweigh the costs from not trading. It seems weird to expect AI to change that, since you are arguing that the proportional importance of spying will go down, not up, because it won’t be accelerated as much.
If the leading project can’t spy on everyone else, then how does it differ from all of the other companies who are developing technology, keeping it private, and charging other people to use it? The leading project can use others’ technology when it pays them, just like they use each other’s technology when they pay each other. The leading project can choose not to sell its technology, but then it just has less money and so falls further and further behind in terms of compute etc. (and at any rate, it needs to be selling something to the other people in order to even be able to afford to use their technology).
(I may just be missing something about your model.)
Yes, if the coalition is a large fraction of the world then I am saying there is an asymmetry in that the leading project can more easily spy on that large fraction of the world than the other way round. This is because large fractions of the world contain many different people and groups, some of which will leak secrets (or sell secrets) to the leading project, unless extremely and unprecedentedly effective anti-leaking-and-spying measures are implemented across a large fraction of the world. It’s hard but doable for one corporation to keep trade secrets from the rest of the economy; how on earth can the rest of the economy keep trade secrets from a powerful corporation?
I don’t see how I’m arguing that the proportional importance of spying will go down. The proportional importance of spying will go up precisely because it won’t be accelerated as much as AI technology in general will be. (Why don’t I think spying will be accelerated as much as AI technology in general? I certainly agree that spying technology will be accelerated as much or more as AI technology. However I think that spying is a function of several things, only one of which is spying technology, the others being non-technology things like having literal human spies climb through ranks of enemy orgs and also having anti-spying technology.) I envision a future where spying is way more rewarding than any time in history, and yet nevertheless the actual amount of successful spying is less than 10x-100x more than in the past, due to the factors mentioned in the parenthesis.
“The leading project can choose not to sell its technology, but then it just has less money and so falls further and further behind in terms of compute etc. (and at any rate, it needs to be selling something to the other people in order to even be able to afford to use their technology).”
Again, my whole point is that this is only true in the long run. Yes, in the long run a project which relies on other sources of income to buy the things it needs to buy will lose money to projects which sell their innovations. But in the short run empirically it seems that projects can go for years on funding raised from investors and wealthy parent companies. I guess your point is that in a world where the economy is growing super fast due to AI, this won’t be true: any parent company or group of investors capable of funding the leading project at year X will be relative paupers by year X+3 unless their project has been selling its tech. Am I right in understanding you here?
(Miscellanous: I don’t think the leading project differs from all the other projects that develop tech and keep it private. Like Wei Dai said, insofar as a company can charge people to use tech without letting the secrets of how to build that tech escape, they will obviously do so. I think our disagreement is about your last two sentences, which I quoted above.)
It seems like there are strong reasons to expect that the post AI coalitions will look very different from the current world economy, though I agree that they might look like a world economy. For instance, imagine world GDP grows by 100x. It seems totally plausible that Google/TSMC/OpenAI revenue grows by 50x relative to typical other companies which only 2x revenue.
Then, power structures might be dramatically different from current power structures. (Even if the US Government is effectively co-running AI lab(s), I still expect that the power structures within AI labs could become considerably more powerful than any current corporate coalition. E.g., maybe 1 board member on the OpenAI board is more powerful than any current corporate board member today in terms of % control over the future of the world.)
Why would a coalition look very different from the world economy, be controlled by a few people, and be hard to form? My default expectation is that it would look much like the world economy. (With the most obvious changes being a fall in the labor share of income / increasing wage inequality.) A few big underlying disagreements:
I don’t think I agree that most progress in AI is driven by rare smart individuals talking to each other—I think it’s not very accurate as a description of current AI progress, that it will be even less true as AI progress becomes a larger share of the world economy, and that most “AI” progress is driven by compute/energy/data/other software rather than stuff that most looks like insight-driven AI progress.
Your toy model seems wrong: most projects make extensive use of other people’s private innovations, by trading with them. So the project that hoards the most innovations can still only be competitive if it trades with others (in order to get access to their hoarded innovations).
I think my more basic complaint is with the “pressure to make a profit over some timescale” model. It think it’s more like: you need inputs from the rest of the economy and so you trade with them. Right now deep learning moonshots don’t trade with the rest of the world because they don’t make anything of much value, but if they were creating really impactful technology then the projects which traded would be radically faster than the projects which just used their innovations in house. This is true even if all innovations are public, since they need access to physical capital.
I think any of these would be enough to carry my objection. (Though if you reject my first claim, and thought that rare smart individuals drove AI progress even then AI progress was overwhelmingly economically important, then you could imagine a sufficiently well-coordinated cartel of those rare smart individuals having a DSA.)
Thanks for the detailed reply! I should say at this point that I’m not particularly confident in my views on this topic; just trying to put forth my take on things in the spirit of improvement. So I wouldn’t be surprised if I end up thinking you are right after more discussion.
A coalition strong enough to prevent the world’s leading project from maintaining and lengthening its lead would need to have some way of preventing the leading project from accessing the innovations of the coalition. Otherwise the leading project will free-ride off the research done by the coalition. For this reason I think that a coalition would look very different from the world economy; in order to prevent the leading project from accessing innovations deployed in the world economy you would need to have an enforced universal embargo on them pretty much, and if you have that much political power, why stop there? Why not just annex them or shut them down?
A successful coalition (that isn’t politically powerful enough to embargo or annex or stop their rival) would need to be capable of preventing information from leaking out to the rival project, and that suggests to me that they would need to concentrate power in the hands of a few individuals (the CEOs and boards of the companies in the coalitions, for example). A distributed, more anarchic architecture would not be able to prevent leaks and spies.
And of course, even this hypothetical coalition would be hard to form, for all the usual reasons. Sure, the 2nd through 10th projects could gang up to decisively beat the leading project. But it’s also true that the 2nd through 10th most powerful nation-states could gang up to decisively beat the most powerful nation-state. Yet this isn’t the norm. More concretely, I expect some projects to ally with each other, but the result to be two or three coalitions of almost equal strength rather than many.
I agree that we seem to disagree about the importance of compute/energy/data vs. smart people talking to each other, and that this disagreement seems relevant. If AI progress was just a matter of compute, for example, then… well actually mightn’t there still be a decisive strategic advantage in that case? Wouldn’t one project have more compute than the others, and thus pull ahead so long as funds lasted?
This gets us into the toy model & its problems. I don’t think I understand your alternative model. I maybe don’t get what you mean by trading. Does one party giving money to another party in return for access to their technology or products count? If so, then I think my original model still stands: The leading project will be able to hoard technology/innovation and lengthen its lead over the rest of the world so long as it still has funding to buy the necessary stuff. I agree that it will be burning money fast if it doesn’t sell/trade its innovations and instead tries to hoard them, but empirically it seems that it’s quite possible for leading projects to go several years in this state.
“Right now deep learning moonshots don’t trade with the rest of the world because they don’t make anything of much value, but if they were creating really impactful technology then the projects which traded would be radically faster than the projects which just used their innovations in house.”
I think it depends not on how impactful their technology is but on how impactful their technology is relative to the perceived impact of hoarding it and going for a decisive strategic advantage. Technology hoarding happens sometimes even in very competitive industries for this reason, and it is the norm among militaries. It seems very possible to me that companies which are producing truly astounding AI technologies—stuff that seems plausibly only a few years away from human-level AGI—will have no problem finding deep-pocketed investors willing to throw money at them for a few years. Again, maybe governments will get involved, in which case this is almost trivial.
I think overall your model (as I understand it) is that people won’t be (successfully) moonshotting for AGI until AI progress is already making the economy grow very fast, and also at this point progress towards AGI will mostly be a matter of how much money you have to buy compute and stuff. So even a deep-pocketed funder like Google wouldn’t be able to compete for two years with shallow-pocketed, more short-sighted projects that sell their tech and reinvest the profits. Is this your view?
To have “more compute than all the others” seems to require already being a large fraction of all the world’s spending (since a large fraction of spending is on computers—or whatever bundle of inputs is about to let this project take over the world—unless you are positing a really bad mispricing). At that point we are talking “coalition of states” rather than “project.”
I totally agree that it wouldn’t be crazy for a major world power to pull ahead of others technologically and eventually be able to win a war handily, and that will tend happen over shorter and shorter timescales if economic and technological progress accelerate.
(Or you might think the project is a small fraction of world compute but larger than any other project, but if economies of scale are in fact this critical, then you are again suggesting a really gigantic market failure. That’s not beyond the pale, but we should be focusing on why this crazy market failure is happening.)
OK, so you agree with me about major world powers (nation-states) but still disagree about companies? I think this means we are closer together than it seemed, because I also think that decisive strategic advantage is significantly more likely to happen if a nation-state gets involved than if it’s just some private company.
I didn’t say “more compute than all the others,” I said “more compute than the others,” by which I meant more compute than any particular other project, yeah. This is consistent with a large fraction of the world’s spending being on compute already. For example, today Deepmind (citation needed) has the largest compute budget of any AI project, but their compute budget is a tiny fraction of the world’s total.
I’m not sure whether or not I’m positing a gigantic market failure. Your claim is that if compute is so important for AI technology and AI technology is so useful, the market will either fail or find ways to get a large fraction of its budget spent on a single AI project? This single project would then be a potential source of DSA but it would also be so big already that it could take over the world by selling products instead? I’m putting question marks not out of sarcasm or anything, just genuine uncertainty about what your claim is. Before I can respond to it I need to understand it.
The reason I let other people use my IP is because they pay me money, with which I can develop even more IP. If the leading project declines to do this, then it will have less IP than any of its normal competitors. If the leading project’s IP allows it to be significantly more productive than everyone else, then they could have just taken over the world through the normal mechanism of selling products. (Modulo leaks/spying.) As far as I can tell, until you are a large fraction of the world, the revenue you get from selling lets you grow faster, and I don’t think the toy model really undermines that typical argument (which has to go through leaks/spying, market frictions, etc.).
I am skeptical that selling products is sufficient to take over the world, no matter how good the products are. Eventually you raise too much attention and get nationalized or taxed or copied.
In light of your critiques I intend to write a much better version of this post in the future. Thanks! I wrote this one during MSFP as part of their blog post day event, so it was kinda rushed and has lots of room for improvement. I’m very glad to see so much engagement though; it inspires me to make said improvements. Perhaps in the course of doing so I’ll change my mind.
Are you saying that the leading project can easily spy on other projects, but other projects can’t spy on it? Is this because the rest of the world is trading with each other, and trading opens up opportunities for spying? Some other reason I missed? I don’t think it’s usually the case that gains from rabbit-holing, in terms of protection from spying, are large enough to outweigh the costs from not trading. It seems weird to expect AI to change that, since you are arguing that the proportional importance of spying will go down, not up, because it won’t be accelerated as much.
If the leading project can’t spy on everyone else, then how does it differ from all of the other companies who are developing technology, keeping it private, and charging other people to use it? The leading project can use others’ technology when it pays them, just like they use each other’s technology when they pay each other. The leading project can choose not to sell its technology, but then it just has less money and so falls further and further behind in terms of compute etc. (and at any rate, it needs to be selling something to the other people in order to even be able to afford to use their technology).
(I may just be missing something about your model.)
Yes, if the coalition is a large fraction of the world then I am saying there is an asymmetry in that the leading project can more easily spy on that large fraction of the world than the other way round. This is because large fractions of the world contain many different people and groups, some of which will leak secrets (or sell secrets) to the leading project, unless extremely and unprecedentedly effective anti-leaking-and-spying measures are implemented across a large fraction of the world. It’s hard but doable for one corporation to keep trade secrets from the rest of the economy; how on earth can the rest of the economy keep trade secrets from a powerful corporation?
I don’t see how I’m arguing that the proportional importance of spying will go down. The proportional importance of spying will go up precisely because it won’t be accelerated as much as AI technology in general will be. (Why don’t I think spying will be accelerated as much as AI technology in general? I certainly agree that spying technology will be accelerated as much or more as AI technology. However I think that spying is a function of several things, only one of which is spying technology, the others being non-technology things like having literal human spies climb through ranks of enemy orgs and also having anti-spying technology.) I envision a future where spying is way more rewarding than any time in history, and yet nevertheless the actual amount of successful spying is less than 10x-100x more than in the past, due to the factors mentioned in the parenthesis.
“The leading project can choose not to sell its technology, but then it just has less money and so falls further and further behind in terms of compute etc. (and at any rate, it needs to be selling something to the other people in order to even be able to afford to use their technology).”
Again, my whole point is that this is only true in the long run. Yes, in the long run a project which relies on other sources of income to buy the things it needs to buy will lose money to projects which sell their innovations. But in the short run empirically it seems that projects can go for years on funding raised from investors and wealthy parent companies. I guess your point is that in a world where the economy is growing super fast due to AI, this won’t be true: any parent company or group of investors capable of funding the leading project at year X will be relative paupers by year X+3 unless their project has been selling its tech. Am I right in understanding you here?
(Miscellanous: I don’t think the leading project differs from all the other projects that develop tech and keep it private. Like Wei Dai said, insofar as a company can charge people to use tech without letting the secrets of how to build that tech escape, they will obviously do so. I think our disagreement is about your last two sentences, which I quoted above.)
It seems like there are strong reasons to expect that the post AI coalitions will look very different from the current world economy, though I agree that they might look like a world economy. For instance, imagine world GDP grows by 100x. It seems totally plausible that Google/TSMC/OpenAI revenue grows by 50x relative to typical other companies which only 2x revenue.
Then, power structures might be dramatically different from current power structures. (Even if the US Government is effectively co-running AI lab(s), I still expect that the power structures within AI labs could become considerably more powerful than any current corporate coalition. E.g., maybe 1 board member on the OpenAI board is more powerful than any current corporate board member today in terms of % control over the future of the world.)