If it is not, then what is it? If it is, then what calculations did it use to reach the above decision—what were the assigned probabilities to the scenarios mentioned?
It’s an expected utility maximizer, but it considers the expected utility of its decision process, not just the expected utility of individual decisions. In a world where there exist more known liars than known superhuman entities, and any liar can claim superhuman powers, any decision process that allows them to exploit you is of negative expected utility.
It’s like the professor who in the example agrees to accept a delayed essay that was delayed for the reason of a grandmother’s death, because this is a valid reason that will largely not be exploited, but not “I wanted to watch my favorite team play”, because lots of others students would be able to use the same excuse. The professor’s not just considering the individual decision, but whether decision process would be of negative utility in a more general manner.
It seems to me that you run into the mathematical problem again when trying to calculate the expected utility of its decision process. Some of the outcomes of the decision process are associated with utilities of 3^^^3.
It seems to me that you run into the mathematical problem again when trying to calculate the expected utility of its decision process. Some of the outcomes of the decision process are associated with utilities of 3^^^3.
Perhaps. I don’t have the math to see how the whole calculation would go.
But it seems to me that the utility of 3^^^3 is associated with a particular execution instance. However when evaluating the decision process as a whole (not the individual decision) the 3^^^3 utility mentioned by the mugger doesn’t have a privileged position over the the hypothetical malicious/lying individuals that can just even more easily talk about utilities or disutilities of 3^^^^3 or 3^^^^^3, or even have their signs reversed (so that they torture people if you submit to their demands despite their claims to the opposite).
So the result should ideally be a different decision process that is able to reject unsubstantiated claims by potentially-lying individuals completely, instead of just trying to fudge the “Probability” of the truth-value of the claim, or the calculated utility if the claim is true.
So the AI is not an expected utility maximizer?
If it is not, then what is it? If it is, then what calculations did it use to reach the above decision—what were the assigned probabilities to the scenarios mentioned?
It’s an expected utility maximizer, but it considers the expected utility of its decision process, not just the expected utility of individual decisions. In a world where there exist more known liars than known superhuman entities, and any liar can claim superhuman powers, any decision process that allows them to exploit you is of negative expected utility.
It’s like the professor who in the example agrees to accept a delayed essay that was delayed for the reason of a grandmother’s death, because this is a valid reason that will largely not be exploited, but not “I wanted to watch my favorite team play”, because lots of others students would be able to use the same excuse. The professor’s not just considering the individual decision, but whether decision process would be of negative utility in a more general manner.
It seems to me that you run into the mathematical problem again when trying to calculate the expected utility of its decision process. Some of the outcomes of the decision process are associated with utilities of 3^^^3.
Perhaps. I don’t have the math to see how the whole calculation would go.
But it seems to me that the utility of 3^^^3 is associated with a particular execution instance. However when evaluating the decision process as a whole (not the individual decision) the 3^^^3 utility mentioned by the mugger doesn’t have a privileged position over the the hypothetical malicious/lying individuals that can just even more easily talk about utilities or disutilities of 3^^^^3 or 3^^^^^3, or even have their signs reversed (so that they torture people if you submit to their demands despite their claims to the opposite).
So the result should ideally be a different decision process that is able to reject unsubstantiated claims by potentially-lying individuals completely, instead of just trying to fudge the “Probability” of the truth-value of the claim, or the calculated utility if the claim is true.