For a medium-large organization made of people who care at least of their own life and the life of their kin the cost of extinction will be so high that it will offset any benefits that they may hope to obtain.
If we consider a simple model where eventually the potential benefit of launching an AGI grows steadily with time, while the risk steadily drops, at some point the expected benefit will exceed the expected cost, and someone will launch an AGI. But because the private cost of extinction is only a small fraction of the social cost, even for a large organization, they will do this much sooner than they should.
Also consider that an organization is made up of individuals, and suffers from internal coordination problems. Take a company of a million employees, how many actually have a say in when the AGI gets launched?
See also this relevant recent article, about how “Putin is acting out of an apparent belief that increasing the nuclear threat to Europe, and as a result to his own country, is ultimately good for Russia and worth the risks. It is a gamble with the lives of hundreds of millions of Europeans, and perhaps many beyond, at stake.” How did we get so close to nuclear war, during the Cold War, and again now, if large organizations (whole countries, in this case) would never risk extinction in the hope of obtaining private benefits?
But a single purely selfish individual is unlikely to create a competitive AI project.
Suppose you’re right about large organizations being more responsible than I think they would be, then they’ll be holding off on launching AGI even when they have the capability to do so. At some point though that capability will filter down to smaller organizations and individuals. Maybe even immediately, if hardware is cheap by that point, and the last steps are purely algorithmic.
If we consider a simple model where eventually the potential benefit of launching an AGI grows steadily with time, while the risk steadily drops, at some point the expected benefit will exceed the expected cost, and someone will launch an AGI. But because the private cost of extinction is only a small fraction of the social cost, even for a large organization, they will do this much sooner than they should.
I’m not sure what point you are trying to make.
Yes, private organizations or national governments make decisions that are less socially optimal compared to a super-competent world-government ruled by a benevolent dictator that has somehow solved the interpersonal preferences comparison problem. That’s not a motte I will try to attack. But it seems to me that you are actually trying to defend the bailey that private organizations or national governments will engage in an arms race to launch a potentially dangerous AI as soon as they could disregarding reasonable safety concerns. This positions seems less defensible.
Suppose you’re right about large organizations being more responsible than I think they would be, then they’ll be holding off on launching AGI even when they have the capability to do so. At some point though that capability will filter down to smaller organizations and individuals. Maybe even immediately, if hardware is cheap by that point, and the last steps are purely algorithmic.
Expect government regulation.
Also note that the same argument can be made for nuclear power, nuclear weapons, chemical weapons or biological weapons. In principle individuals or small groups could build them, and there have been perhaps one instance of bioweapon attack (the 2001 anthrax mail attacks in the US) and a few instances of chemical attacks. But all of them were inefficient and ultimately caused little damage. In practice it seems that the actual expertise and organizational capabilities required to pull such things to a significant scale are non-trivial.
AI may be quite similar in this regard: even without malicious intent, going from research papers and proof-of-concept systems to a fully operational system capable of causing major damage will probably require significant engineering efforts.
If we consider a simple model where eventually the potential benefit of launching an AGI grows steadily with time, while the risk steadily drops, at some point the expected benefit will exceed the expected cost, and someone will launch an AGI. But because the private cost of extinction is only a small fraction of the social cost, even for a large organization, they will do this much sooner than they should.
Also consider that an organization is made up of individuals, and suffers from internal coordination problems. Take a company of a million employees, how many actually have a say in when the AGI gets launched?
See also this relevant recent article, about how “Putin is acting out of an apparent belief that increasing the nuclear threat to Europe, and as a result to his own country, is ultimately good for Russia and worth the risks. It is a gamble with the lives of hundreds of millions of Europeans, and perhaps many beyond, at stake.” How did we get so close to nuclear war, during the Cold War, and again now, if large organizations (whole countries, in this case) would never risk extinction in the hope of obtaining private benefits?
Suppose you’re right about large organizations being more responsible than I think they would be, then they’ll be holding off on launching AGI even when they have the capability to do so. At some point though that capability will filter down to smaller organizations and individuals. Maybe even immediately, if hardware is cheap by that point, and the last steps are purely algorithmic.
I’m not sure what point you are trying to make.
Yes, private organizations or national governments make decisions that are less socially optimal compared to a super-competent world-government ruled by a benevolent dictator that has somehow solved the interpersonal preferences comparison problem. That’s not a motte I will try to attack.
But it seems to me that you are actually trying to defend the bailey that private organizations or national governments will engage in an arms race to launch a potentially dangerous AI as soon as they could disregarding reasonable safety concerns. This positions seems less defensible.
Expect government regulation.
Also note that the same argument can be made for nuclear power, nuclear weapons, chemical weapons or biological weapons.
In principle individuals or small groups could build them, and there have been perhaps one instance of bioweapon attack (the 2001 anthrax mail attacks in the US) and a few instances of chemical attacks. But all of them were inefficient and ultimately caused little damage. In practice it seems that the actual expertise and organizational capabilities required to pull such things to a significant scale are non-trivial.
AI may be quite similar in this regard: even without malicious intent, going from research papers and proof-of-concept systems to a fully operational system capable of causing major damage will probably require significant engineering efforts.