All a self-improving AI needs is good enough YGBM (you gotta believe me) technology. Eliezer gets this, some of the commenters in this thread don’t. The key isn’t the ability to solve protein-folding, the key is the ability to manipulate a threshhold amount of us to its ends.
We may already functionally be there. You and Chiquita Brands International in a death match. My money is CBI killing you before you kill it. Companies and markets already manipulate our behavior, using reward incentives like the ones in Eliezer’s parable for us to engage in behavior that maximizes their persistence at the expense of ours. Whether they’re “intelligent”, “conscious”, or not isn’t as relevant to me as the fact that they may have a permanent persistence maximizing advantage over us. My money is probably on corporations and markets substrate jumping and leaving us behind as more likely than us substrate jumping and leaving our cellular and bacteriological medium behind.
Considering how much wealth can be generated at the moment by running a computer program shifting (numbers loosely representing) money around, and the ever-more-sophisticated ways that this can be done (obviously, to the detriment of many of the humans involved), I think it’s already in the process of substrate jumping. These things aren’t limited to human minds and tax law any more.
A point well taken. IT professionals have a phrase: “Problem exists between keyboard and chair.” Humans are insecure systems; we can be engineered remarkably easily.
Though I do think there is some hope of beating a corporation; you just need a lot of people on your side (a number comparable to the total number of customers, shareholders, and employees of the corporation).
All a self-improving AI needs is good enough YGBM (you gotta believe me) technology. Eliezer gets this, some of the commenters in this thread don’t. The key isn’t the ability to solve protein-folding, the key is the ability to manipulate a threshhold amount of us to its ends.
We may already functionally be there. You and Chiquita Brands International in a death match. My money is CBI killing you before you kill it. Companies and markets already manipulate our behavior, using reward incentives like the ones in Eliezer’s parable for us to engage in behavior that maximizes their persistence at the expense of ours. Whether they’re “intelligent”, “conscious”, or not isn’t as relevant to me as the fact that they may have a permanent persistence maximizing advantage over us. My money is probably on corporations and markets substrate jumping and leaving us behind as more likely than us substrate jumping and leaving our cellular and bacteriological medium behind.
History says that when United Fruit wants you dead, you will die. (see Latin America)
Considering how much wealth can be generated at the moment by running a computer program shifting (numbers loosely representing) money around, and the ever-more-sophisticated ways that this can be done (obviously, to the detriment of many of the humans involved), I think it’s already in the process of substrate jumping. These things aren’t limited to human minds and tax law any more.
A point well taken. IT professionals have a phrase: “Problem exists between keyboard and chair.” Humans are insecure systems; we can be engineered remarkably easily.
Though I do think there is some hope of beating a corporation; you just need a lot of people on your side (a number comparable to the total number of customers, shareholders, and employees of the corporation).