Not sure if you’ve ever taken a class on electricity & magnetism, but one of the central notions is the conservative vector field—electric fields being the standard example. You take an electron, and drag it around the electric field. Sometimes you’ll have to push it against the field, sometimes the field will push it along for you. You add up all the energy spent pushing (or energy extracted when the field pushes it for you), and find an interesting result: the energy spent moving the electron from point A to point B is completely independent of the path taken. Any two paths from A to B will require exactly the same energy expenditure.
That’s a pretty serious constraint on the field—the vast majority of possible vector fields are not conservative.
It’s also exactly the same constraint as a utility function: a vector field is conservative if-and-only-if it is acyclic, in the sense of having zero circulation around any closed curve. Indeed, this means that conservative vector fields can be viewed as utility functions: the field itself is the gradient of a “utility function” (called the potential field), and it accepts any local “trade” which increases utility—i.e. moving an electron up the gradient of the utility function. Conversely, if we have preferences represented by local preferences in a (finite-dimensional) vector space, then we can summarize those preferences with a utility function if-and-only-if the field is conservative.
My point is: acyclicity is a major constraint on a system’s behavior. It is definitely not the case that “everything can be represented as having a utility function”.
Now, there is a separate piece to your concern: when people talk about subagent theories of mind, they think that the brain is actually implemented using subagents, not merely behaving in a manner equivalent to having subagents. It’s a variant of the behavior vs architecture question. In this case, we can partially answer the question: subagent architectures have a relative advantage over most non-subagent architectures in that the subagent architectures won’t throw away resources via cyclic preferences, whereas most of the non-subagent architectures will. The only non-subagent architectures which don’t throw away resources are those whose behavior just so happens to be equivalent to subagents.
If a system with a subagent architecture is evolving, then it will mostly be exploring different configurations of subagents—so any configuration it explores will at least not throw away resources. On the other hand, with a non-subagent architecture, we’d expect that there’s some surface in configuration space which happens to implement agent-like behavior, and any changes which move off that surface will throw away at least some resources—and any single-nucleotide change is likely to move off the surface. In other words, a subagent architecture is more likely to have a nice evolutionary path from wherever it starts to the maximum-fitness design, whereas a non-subagent architecture may not have such a smooth path. As an evolutionary analogue to the behavior vs architecture question, I’d conjecture: subagent-like behavior generally won’t evolve without subagent-like architecture, because it’s so much easier to explore efficient designs within a subagent architecture.
Not sure if you’ve ever taken a class on electricity & magnetism, but one of the central notions is the conservative vector field—electric fields being the standard example. You take an electron, and drag it around the electric field. Sometimes you’ll have to push it against the field, sometimes the field will push it along for you. You add up all the energy spent pushing (or energy extracted when the field pushes it for you), and find an interesting result: the energy spent moving the electron from point A to point B is completely independent of the path taken. Any two paths from A to B will require exactly the same energy expenditure.
That’s a pretty serious constraint on the field—the vast majority of possible vector fields are not conservative.
It’s also exactly the same constraint as a utility function: a vector field is conservative if-and-only-if it is acyclic, in the sense of having zero circulation around any closed curve. Indeed, this means that conservative vector fields can be viewed as utility functions: the field itself is the gradient of a “utility function” (called the potential field), and it accepts any local “trade” which increases utility—i.e. moving an electron up the gradient of the utility function. Conversely, if we have preferences represented by local preferences in a (finite-dimensional) vector space, then we can summarize those preferences with a utility function if-and-only-if the field is conservative.
My point is: acyclicity is a major constraint on a system’s behavior. It is definitely not the case that “everything can be represented as having a utility function”.
Now, there is a separate piece to your concern: when people talk about subagent theories of mind, they think that the brain is actually implemented using subagents, not merely behaving in a manner equivalent to having subagents. It’s a variant of the behavior vs architecture question. In this case, we can partially answer the question: subagent architectures have a relative advantage over most non-subagent architectures in that the subagent architectures won’t throw away resources via cyclic preferences, whereas most of the non-subagent architectures will. The only non-subagent architectures which don’t throw away resources are those whose behavior just so happens to be equivalent to subagents.
If a system with a subagent architecture is evolving, then it will mostly be exploring different configurations of subagents—so any configuration it explores will at least not throw away resources. On the other hand, with a non-subagent architecture, we’d expect that there’s some surface in configuration space which happens to implement agent-like behavior, and any changes which move off that surface will throw away at least some resources—and any single-nucleotide change is likely to move off the surface. In other words, a subagent architecture is more likely to have a nice evolutionary path from wherever it starts to the maximum-fitness design, whereas a non-subagent architecture may not have such a smooth path. As an evolutionary analogue to the behavior vs architecture question, I’d conjecture: subagent-like behavior generally won’t evolve without subagent-like architecture, because it’s so much easier to explore efficient designs within a subagent architecture.