Richard Ngo, as part of his comprehensive overview of Realism about Rationality and his skepticism of it, pointed out the very important fact that “there is no canonical way to scale [a human] up”, which means that you need to make arbitrary choices (with no real guiding hand available to you) of which modifications to make to a human if you want to make him/her more intelligent, knowledgeable, coherent etc.
I seem to recall hearing a phrase I liked, which appears to concisely summarize the concern as: “There’s no canonical way to scale me up.”
Does that sound right to you?
I mentioned it above :)