I was surprised that the post focused on the difficulty of learning to classify things, rather than on the problems that would arise assuming the AI learned to classify smiling humans correctly. I’m not worried that the AI will tile the universe with smiley-faces. I’m worried the AI will tile the universe with smiling humans. Even with genuinely happy humans.
Humans can classify humans into happy and unhappy pretty well; superintelligent AI will be able to also. The hard problem is not identifying happiness; the hard problem is deciding what to maximize.
I was surprised that the post focused on the difficulty of learning to classify things, rather than on the problems that would arise assuming the AI learned to classify smiling humans correctly. I’m not worried that the AI will tile the universe with smiley-faces. I’m worried the AI will tile the universe with smiling humans. Even with genuinely happy humans.
Humans can classify humans into happy and unhappy pretty well; superintelligent AI will be able to also. The hard problem is not identifying happiness; the hard problem is deciding what to maximize.