Economic arguments, such as the principle of comparative advantage, are sometimes
invoked to argue that AGI would find it more beneficial to trade with us than to do
us harm. However, technological progress can drive the wages of workers below the
level needed for survival (Clark 2007; Freeman 2008; Brynjolfsson and McAfee 2011;
Miller 2012), and there is already a possible threat of technological unemployment
(Brynjolfsson and McAfee 2011). AGIs keeping humans around due to gains from
trade implicitly presumes that they would not have the will or the opportunity to simply
eliminate humans in order to replace them with a better trading partner, and then trade
with the new partner instead.
Humans already eliminate species with low economic value in order to make room
for more humans, such as when clearing a forest in order to build new homes. Clark
uses the example of horses in Britain: their population peaked in 1901, with 3.25 million
horses doing work such as plowing fields, hauling wagons and carriages short distances,
and carrying armies into battle. The internal combustion engine replaced so many of
them that by 1924 there were fewer than two million. Clark (2007) writes:
There was always a wage at which all these horses could have remained employed. But that wage was so low that it did not pay for their feed, and it
certainly did not pay enough to breed fresh generations of horses to replace
them. Horses were thus an early casualty of industrialization.
There are also ways to harm humans while still respecting their property rights, such as
by manipulating them into making bad decisions, or selling them addictive substances.
If AGIs were sufficiently smarter than humans, humans could be tricked into making a
series of trades that respected their property rights but left them with negligible assets
and caused considerable damage to their well-being.
What we said about this in Responses to Catastrophic AGI Risk:
Yep