While many (most?) humans clearly seem to be able to make use of internal sub-agents, it is not clear how these themselves are realized in the brain. There is no obvious agent module in the brain. It seems more plausible that agentyness is a natural abstraction. If so, these concepts are not pure, i.e., without influence from circumstantial evidence. Sure, higher levels of abstraction might be learned that may increasingly shed the circumstantial details. But can we or the learning mechanism make use of this fact? There is no clearly analyzable parameter for “agent 1 responsible for topic X” that could be tweaked. At least not until the AI builds interpretability for itself and/or builds a successor AI with explicit structures for agents (if that is even possible).
While many (most?) humans clearly seem to be able to make use of internal sub-agents, it is not clear how these themselves are realized in the brain. There is no obvious agent module in the brain. It seems more plausible that agentyness is a natural abstraction. If so, these concepts are not pure, i.e., without influence from circumstantial evidence. Sure, higher levels of abstraction might be learned that may increasingly shed the circumstantial details. But can we or the learning mechanism make use of this fact? There is no clearly analyzable parameter for “agent 1 responsible for topic X” that could be tweaked. At least not until the AI builds interpretability for itself and/or builds a successor AI with explicit structures for agents (if that is even possible).