The idea in the first part of the post is that, if we try to abstract just one of the two agent-instances, then the abstraction fails because low-level details of the calculations in the two instances are correlated; the point of abstraction is (roughly) that low-level details should be uncorrelated between different abstract components.
But then we ask why that’s a problem; in what situations does the one-instance agent abstraction actually return incorrect predictions? What queries does it not answer correctly? Answer: queries which involve correlations between the low-level calculations within the two agent-instances.
But do we actually care about those particular queries? Like, think about it from the agent’s point of view. It only cares about maximizing the outcome, and the outcome doesn’t depend on the low-level details of the two agents’ calculations; the outcome only depends on the functional form of the agents’ computations (i.e. their input-output behavior).
I guess what is confusing me is that you seem to have provided a reason why we shouldn’t just care about high-level functional behaviour (because this might miss correlations between the low-level components), then in the next sentence you’re acting as though this is irrelevant?
Yeah, I worried that would be confusing when writing the OP. Glad you left a comment, I’m sure other people are confused by it too.
The reason why we shouldn’t just care about high-level functional behavior is that the abstraction leaks. Problem is, it’s not clear that the abstraction leaks in a way which is relevant to the outcome. And if the leak isn’t relevant to the outcome, then it’s not clear why we need to care about it—or (more importantly) why the agents themselves would care about it.
Please keep asking for clarification if that doesn’t help explain it; I want to make this make sense for everyone else too.
The idea in the first part of the post is that, if we try to abstract just one of the two agent-instances, then the abstraction fails because low-level details of the calculations in the two instances are correlated; the point of abstraction is (roughly) that low-level details should be uncorrelated between different abstract components.
But then we ask why that’s a problem; in what situations does the one-instance agent abstraction actually return incorrect predictions? What queries does it not answer correctly? Answer: queries which involve correlations between the low-level calculations within the two agent-instances.
But do we actually care about those particular queries? Like, think about it from the agent’s point of view. It only cares about maximizing the outcome, and the outcome doesn’t depend on the low-level details of the two agents’ calculations; the outcome only depends on the functional form of the agents’ computations (i.e. their input-output behavior).
Does that make sense?
I guess what is confusing me is that you seem to have provided a reason why we shouldn’t just care about high-level functional behaviour (because this might miss correlations between the low-level components), then in the next sentence you’re acting as though this is irrelevant?
Yeah, I worried that would be confusing when writing the OP. Glad you left a comment, I’m sure other people are confused by it too.
The reason why we shouldn’t just care about high-level functional behavior is that the abstraction leaks. Problem is, it’s not clear that the abstraction leaks in a way which is relevant to the outcome. And if the leak isn’t relevant to the outcome, then it’s not clear why we need to care about it—or (more importantly) why the agents themselves would care about it.
Please keep asking for clarification if that doesn’t help explain it; I want to make this make sense for everyone else too.
Ah, I think I now get where you are coming from