My guess here is that there are some instrumentally convergent abstractions/algorithms which both a brain and a hypothetical AGI needs to use. But a brain will have implemented some of those as hacks on top of methods which evolved earlier, whereas an AI could implement those methods directly. So for instance, one could imagine the brain implementing simple causal reasoning as a hack on top of pre-existing temporal sequence capabilities. When designing an AI, it would probably make more sense to use causal DAGs as the fundamental, and then implement temporal sequences as abstract stick-dags which don’t support many (if any) counterfactuals.
Possibly better example: tree search and logic. Humans seem to handle these mostly as hacks on top of pattern-matchers and trigger-action pairs, but for an AI it makes more sense to implement tree search as a fundamental.
My guess here is that there are some instrumentally convergent abstractions/algorithms which both a brain and a hypothetical AGI needs to use. But a brain will have implemented some of those as hacks on top of methods which evolved earlier, whereas an AI could implement those methods directly. So for instance, one could imagine the brain implementing simple causal reasoning as a hack on top of pre-existing temporal sequence capabilities. When designing an AI, it would probably make more sense to use causal DAGs as the fundamental, and then implement temporal sequences as abstract stick-dags which don’t support many (if any) counterfactuals.
Possibly better example: tree search and logic. Humans seem to handle these mostly as hacks on top of pattern-matchers and trigger-action pairs, but for an AI it makes more sense to implement tree search as a fundamental.