I don’t understand the science: but my understanding was that utility in this sense was whatever the AI was trying to maximise. If you make an AI that tries to maximise paperclips, it’s motivated to do that, not to maximise ‘doing whatever maximises my utility maximisation’. I think there’s an error here comes of mixing it up with pleasure, and thinking that the paperclip maximiser will ‘enjoy’ making paperclips but ultimately what it wants is enjoyment. Whereas in its motherboard of motherboards it really, really wants paperclips.
Well, we can draw a useful distinction here between “number of paperclips” and “perceived number of paperclips”. The AI only ever has access to the latter, and is probably a lot more aware of this than we are, since it can watch its own program running. The process of updating its model of the world toward greater accuracy is likely to be “painful” (in an appropriate sense), as the AI realizes progressively that there are fewer paperclips in the world than previously believed; it’s much easier, and a lot more rewarding, to simply decide that there are an arbitrarily large number of paperclips in the world already and it needn’t do anything.
If it’s designed to be concerned with a particular number in a spreadsheet , so to speak, absolutely. I have no idea if it’s possible to make the desire more connected to the world, to make the AI really care about something in the world.
I don’t understand the science: but my understanding was that utility in this sense was whatever the AI was trying to maximise. If you make an AI that tries to maximise paperclips, it’s motivated to do that, not to maximise ‘doing whatever maximises my utility maximisation’. I think there’s an error here comes of mixing it up with pleasure, and thinking that the paperclip maximiser will ‘enjoy’ making paperclips but ultimately what it wants is enjoyment. Whereas in its motherboard of motherboards it really, really wants paperclips.
Well, we can draw a useful distinction here between “number of paperclips” and “perceived number of paperclips”. The AI only ever has access to the latter, and is probably a lot more aware of this than we are, since it can watch its own program running. The process of updating its model of the world toward greater accuracy is likely to be “painful” (in an appropriate sense), as the AI realizes progressively that there are fewer paperclips in the world than previously believed; it’s much easier, and a lot more rewarding, to simply decide that there are an arbitrarily large number of paperclips in the world already and it needn’t do anything.
If it’s designed to be concerned with a particular number in a spreadsheet , so to speak, absolutely. I have no idea if it’s possible to make the desire more connected to the world, to make the AI really care about something in the world.