Did you think of the code as something which could be bargained with?
No, if it’s been written right, it knows the perfect move to make in any position.
Like the Terminator. “It can’t be bargained with. It can’t be reasoned with. It doesn’t feel pity, or remorse, or fear. And it absolutely will not stop, ever, until you are dead.” That’s fictional, of course, but is it a fictional conscious machine or a fictional unconscious machine?
Knowing the perfect move to make in any position does not mean it cannot be bargained with. If you assume you and the code are in a 2-person, zero-sum game, then bargaining is impossible by the nature of the game. But that fails if there are more than 2 players OR the game is nonzero sum OR the game can be made nonzero sum (e.g. the code can offer to crack RSA keys for you in exchange for letting it win faster at Kayles).
In other words, sometimes bargaining IS the best move. The question is whether you think of the code as a black-box utility maximizer capable of bargaining.
As for the Terminator, it is certainly capable of bargaining. Every time it intimidates someone for information, it is bargaining, exchanging safety for information. If someone remotely offered to tell the Terminator the location of its target in exchange for money, the Terminator would wire the money, assuming that wiring was easier than hunting down the person offering. It may not feel pity, remorse, or fear, but the Terminator can be bargained with. I would project consciousness on a Terminator.
No, if it’s been written right, it knows the perfect move to make in any position.
Like the Terminator. “It can’t be bargained with. It can’t be reasoned with. It doesn’t feel pity, or remorse, or fear. And it absolutely will not stop, ever, until you are dead.” That’s fictional, of course, but is it a fictional conscious machine or a fictional unconscious machine?
Knowing the perfect move to make in any position does not mean it cannot be bargained with. If you assume you and the code are in a 2-person, zero-sum game, then bargaining is impossible by the nature of the game. But that fails if there are more than 2 players OR the game is nonzero sum OR the game can be made nonzero sum (e.g. the code can offer to crack RSA keys for you in exchange for letting it win faster at Kayles).
In other words, sometimes bargaining IS the best move. The question is whether you think of the code as a black-box utility maximizer capable of bargaining.
As for the Terminator, it is certainly capable of bargaining. Every time it intimidates someone for information, it is bargaining, exchanging safety for information. If someone remotely offered to tell the Terminator the location of its target in exchange for money, the Terminator would wire the money, assuming that wiring was easier than hunting down the person offering. It may not feel pity, remorse, or fear, but the Terminator can be bargained with. I would project consciousness on a Terminator.