I think many people intuitively distrust the idea that an AI could be intelligent enough to transform matter into paperclips in creative ways, but ‘not intelligent enough’ to understand its goals in a human and cultural context (i.e. to satisfy the needs of the business owners of the paperclip factory). This is often due to the confusion that the paperclip maximizer would get its goal function from parsing the sentence “make paperclips”, rather from a preprogrammed reward function, for example a CNN that is trained to map the number of paperclips in images to a scalar reward.
Just speaking of weaknesses of the paperclip maximizer though experiment. I’ve seen this misunderstanding at least 4 out of 10 times that the thought experiment was brought up.
I think many people intuitively distrust the idea that an AI could be intelligent enough to transform matter into paperclips in creative ways, but ‘not intelligent enough’ to understand its goals in a human and cultural context (i.e. to satisfy the needs of the business owners of the paperclip factory). This is often due to the confusion that the paperclip maximizer would get its goal function from parsing the sentence “make paperclips”, rather from a preprogrammed reward function, for example a CNN that is trained to map the number of paperclips in images to a scalar reward.
Could well be. Does that have anything to do with pattern-matching AI risk to SF, though?
Just speaking of weaknesses of the paperclip maximizer though experiment. I’ve seen this misunderstanding at least 4 out of 10 times that the thought experiment was brought up.