SHRDLU could parse ″pick up a big red block″ and respond with an action that seems appropriate, but could not understand those concepts in a more general environment.
Could the same thing be also true for humans, only that we have much larger environment where we can function properly?
When we discuss many-worlds interpretation, the meaning of probability, whether atoms have identity, et cetera… a superintelligence could observe our attempts and say: “Humans have some symbols hardcoded by evolution, they can also do some simple reasoning about them, but when they get too far away from their natural environment, they are completely lost.”
Could the same thing be also true for humans, only that we have much larger environment where we can function properly?
When we discuss many-worlds interpretation, the meaning of probability, whether atoms have identity, et cetera… a superintelligence could observe our attempts and say: “Humans have some symbols hardcoded by evolution, they can also do some simple reasoning about them, but when they get too far away from their natural environment, they are completely lost.”