I think there’s thorough confusion between utilityA: utility as used in economics to try and predict humans (and predict them inaccurately), and the utilityB: utility as in the model based agent, where the utility is a mathematical function which takes in description of the world and which only refers to real world items if you read stuff into it that is not there and can not be put there.
Viciously maximizing some utilityB leads to, given sufficient capability, the vicious and ohh so dangerous modification of the inputs to utilityB function, i.e. wireheading.
The AIs as we know them, agents or tools, are not utilityA maximizers. We do not know how to make utilityA maximizer. The human intelligence also doesn’t seem to work as utilityA maximizer. It is likely the case that utilityA maximizer is a logical impossibility for agents embedded in the world, or at very least, requires very major advances in formalization of philosophy.
It is likely the case that utilityA maximizer is a logical impossibility for agents embedded in the world...
Very interesting and relevant! Can you elaborate or link? I think the case can be made based on Arrow’s theorem and its corollaries, but I’m not sure that’s what you have in mind.
I think there’s thorough confusion between utilityA: utility as used in economics to try and predict humans (and predict them inaccurately), and the utilityB: utility as in the model based agent, where the utility is a mathematical function which takes in description of the world and which only refers to real world items if you read stuff into it that is not there and can not be put there.
Viciously maximizing some utilityB leads to, given sufficient capability, the vicious and ohh so dangerous modification of the inputs to utilityB function, i.e. wireheading.
The AIs as we know them, agents or tools, are not utilityA maximizers. We do not know how to make utilityA maximizer. The human intelligence also doesn’t seem to work as utilityA maximizer. It is likely the case that utilityA maximizer is a logical impossibility for agents embedded in the world, or at very least, requires very major advances in formalization of philosophy.
Very interesting and relevant! Can you elaborate or link? I think the case can be made based on Arrow’s theorem and its corollaries, but I’m not sure that’s what you have in mind.