Well, we can draw a useful distinction here between “number of paperclips” and “perceived number of paperclips”. The AI only ever has access to the latter, and is probably a lot more aware of this than we are, since it can watch its own program running. The process of updating its model of the world toward greater accuracy is likely to be “painful” (in an appropriate sense), as the AI realizes progressively that there are fewer paperclips in the world than previously believed; it’s much easier, and a lot more rewarding, to simply decide that there are an arbitrarily large number of paperclips in the world already and it needn’t do anything.
If it’s designed to be concerned with a particular number in a spreadsheet , so to speak, absolutely. I have no idea if it’s possible to make the desire more connected to the world, to make the AI really care about something in the world.
Well, we can draw a useful distinction here between “number of paperclips” and “perceived number of paperclips”. The AI only ever has access to the latter, and is probably a lot more aware of this than we are, since it can watch its own program running. The process of updating its model of the world toward greater accuracy is likely to be “painful” (in an appropriate sense), as the AI realizes progressively that there are fewer paperclips in the world than previously believed; it’s much easier, and a lot more rewarding, to simply decide that there are an arbitrarily large number of paperclips in the world already and it needn’t do anything.
If it’s designed to be concerned with a particular number in a spreadsheet , so to speak, absolutely. I have no idea if it’s possible to make the desire more connected to the world, to make the AI really care about something in the world.