Well, the AI wouldn’t only have to predict how the lie will benefit the person who hears it, but also how the actions that result from holding a false belief might affect other individuals.
The above quibble aside, the answer to your question is pretty trivial. To a person who values the truth, knowledge is a benefit and will therefore be part of the AI’s ‘net benefit’ calculation. Consequently, the kind of person who would want to program an AI to only tell him the truth would never be lied to by an AI that does a net benefit calculation.
The only question that remains is if we would want to program an AI to always tell the truth to people who want to be deceived at least some of the time. In other words, do we want other individuals to believe whatever they like, as long as it’s a net benefit to them and doesn’t affect the rest of us? My answer would be yes.
Of course, in the real world, popular false beliefs, such as religious ones, often do not lead to a net benefit for those who hold them, and even more often affect the rest of us negatively.
To a person who values the truth, knowledge is a benefit and will therefore be part of the AI’s ‘net benefit’ calculation.
But knowledge of the truth has a finite value. What if the AI believed that the benefit of a lie would outweigh a truth-seeker’s cost of being lied to?
So the question is, would any rational truth-seeker choose to only be told the truth by the AI?
A person doesn’t have to ‘infinitely value’ truth to always prefer the truth to a lie. The importance put on truth merely has to be greater than the importance put on anything else.
That said, if the question is, is there a human, or has there ever been a human who values truth more than anything else, the answer is almost certainly no. For example, I care about the truth a lot, but if I were given the choice between learning a single, randomly chosen fact about the universe, and being given a million dollars, I’d pick the cash without too much hesitation.
However, as Eliezer has said many times, human minds only represent a tiny fraction of all possible minds. A mind that puts truth above anything else is certainly possible, even if it doesn’t exist yet.
Now that we know we programmed an AI that may lie to us, our rational expectations will make us skeptical of what the AI says, which is not ideal. Sounds like the AI programmer will have to cover up the fact that the AI does not always speak the truth.
Well, the AI wouldn’t only have to predict how the lie will benefit the person who hears it, but also how the actions that result from holding a false belief might affect other individuals.
The above quibble aside, the answer to your question is pretty trivial. To a person who values the truth, knowledge is a benefit and will therefore be part of the AI’s ‘net benefit’ calculation. Consequently, the kind of person who would want to program an AI to only tell him the truth would never be lied to by an AI that does a net benefit calculation.
The only question that remains is if we would want to program an AI to always tell the truth to people who want to be deceived at least some of the time. In other words, do we want other individuals to believe whatever they like, as long as it’s a net benefit to them and doesn’t affect the rest of us? My answer would be yes.
Of course, in the real world, popular false beliefs, such as religious ones, often do not lead to a net benefit for those who hold them, and even more often affect the rest of us negatively.
But knowledge of the truth has a finite value. What if the AI believed that the benefit of a lie would outweigh a truth-seeker’s cost of being lied to?
So the question is, would any rational truth-seeker choose to only be told the truth by the AI?
A person doesn’t have to ‘infinitely value’ truth to always prefer the truth to a lie. The importance put on truth merely has to be greater than the importance put on anything else.
That said, if the question is, is there a human, or has there ever been a human who values truth more than anything else, the answer is almost certainly no. For example, I care about the truth a lot, but if I were given the choice between learning a single, randomly chosen fact about the universe, and being given a million dollars, I’d pick the cash without too much hesitation.
However, as Eliezer has said many times, human minds only represent a tiny fraction of all possible minds. A mind that puts truth above anything else is certainly possible, even if it doesn’t exist yet.
Now that we know we programmed an AI that may lie to us, our rational expectations will make us skeptical of what the AI says, which is not ideal. Sounds like the AI programmer will have to cover up the fact that the AI does not always speak the truth.