But I guess that instrumental convergence will still eventually lead to either
all shards acquiring more and more instrumental structure (neuronal weights within shards getting optimized for that), or
shards that are directly instrumental will take more and more weight overall.
One can see that in regular human adult development. The heuristics children use are simpler and more of the type “searching for nice things in nice ways” or even seeing everything thru a niceness lens. While adults have more pure strategies, e.g., planning as a shard of its own. Most humans just die before they reach convergence. And there are probably also other aspects. Enlightenment may be a state where pure shards become an option.
But I guess that instrumental convergence will still eventually lead to either
all shards acquiring more and more instrumental structure (neuronal weights within shards getting optimized for that), or
shards that are directly instrumental will take more and more weight overall.
One can see that in regular human adult development. The heuristics children use are simpler and more of the type “searching for nice things in nice ways” or even seeing everything thru a niceness lens. While adults have more pure strategies, e.g., planning as a shard of its own. Most humans just die before they reach convergence. And there are probably also other aspects. Enlightenment may be a state where pure shards become an option.