To push back on this, I’m not sure that humanness is a “bug,” as you say. While we likely aren’t a pinnacle of intelligence in a fundamental sense, I do think that as humans have continued to advance, first through natural selection and now through… whatever it is we do now with culture and education and science, the parts of humanness that we care about have tended to increase in us, and not go away. So perhaps an AI optimized far beyond us, but starting in the same general neighborhood in the function space, would optimize to become not just superintelligent but superhuman in the sense that they would embody the things that we care about better than we do!
To push back on this, I’m not sure that humanness is a “bug,” as you say. While we likely aren’t a pinnacle of intelligence in a fundamental sense, I do think that as humans have continued to advance, first through natural selection and now through… whatever it is we do now with culture and education and science, the parts of humanness that we care about have tended to increase in us, and not go away. So perhaps an AI optimized far beyond us, but starting in the same general neighborhood in the function space, would optimize to become not just superintelligent but superhuman in the sense that they would embody the things that we care about better than we do!