Right, the fact that dividing by the number of copies of you sometimes gives division by zero is another good reason for not doing it :-)
But your assessment of problem importance is interesting. I would’ve said it the other way around. In practice, we tend to quickly notice when we are being money pumped, and apply a patch on the fly, so at worst we only end up losing a bit of money, which is recoverable. Crazy policies on the other hand… usually do little damage because we compartmentalize, but when we fail to compartmentalize, the resulting loss may not be recoverable.
This is a valid point for humans (who are not utilitarians at all). but when thinking of an ideal, AI-safe ethics, money pumps are a great falw: because the AI will get pumped again, again, and again, and lose utility. Alternately, the AI will patch its system on the fly; and if we don’t know how it does this, it could end up with a crazy policy—its unpredictable.
Maybe; honestly, nobody knows yet. We’re still too far from being able to build an AI for which ethics would be a relevant concept, to be able to say what such an ethics should look like. For all we really know, perhaps if and when we get to that point, it might become apparent that a system of supervisor modules to perform on-the-fly patching to reliably keep things within sensible bounds is a better solution than trying for logically perfect ethics, for any mind operating under physically realistic constraints of data and computing power.
Right, the fact that dividing by the number of copies of you sometimes gives division by zero is another good reason for not doing it :-)
But your assessment of problem importance is interesting. I would’ve said it the other way around. In practice, we tend to quickly notice when we are being money pumped, and apply a patch on the fly, so at worst we only end up losing a bit of money, which is recoverable. Crazy policies on the other hand… usually do little damage because we compartmentalize, but when we fail to compartmentalize, the resulting loss may not be recoverable.
This is a valid point for humans (who are not utilitarians at all). but when thinking of an ideal, AI-safe ethics, money pumps are a great falw: because the AI will get pumped again, again, and again, and lose utility. Alternately, the AI will patch its system on the fly; and if we don’t know how it does this, it could end up with a crazy policy—its unpredictable.
Maybe; honestly, nobody knows yet. We’re still too far from being able to build an AI for which ethics would be a relevant concept, to be able to say what such an ethics should look like. For all we really know, perhaps if and when we get to that point, it might become apparent that a system of supervisor modules to perform on-the-fly patching to reliably keep things within sensible bounds is a better solution than trying for logically perfect ethics, for any mind operating under physically realistic constraints of data and computing power.