EDIT: This answer isn’t very good, see my other one.
Good question. We could easily draw a diagram in which the two are separate—we’d have the “agent” node reading from one cloud and then influencing things outside of that cloud. But that case isn’t very interesting—most of what we call “agenty” behavior, and especially the diagonalization issues, are about the case where the actual model and the agent’s beliefs coincide. In particular, if we’re talking about ideal game-theoretic agents, we usually assume that both the rules of the game and each agent’s strategy are common knowledge—including off-equilibrium behavior.
So, for idealized game-theoretic agents, there is no separation between the actual model and the agent’s model—interventions on the actual model are reflected in the agent’s model.
That said, in the low-level model, the map and the territory will presumably always be separate. “When do they coincide?” is implicitly wrapped up in the question “when do non-agenty models abstract into agenty models?”. I view the potential mismatch between the two models as an abstraction failure—if they don’t match, then the agency-abstraction is broken.
I think there’s a contradiction here. Idealized game-theoretic agents are the OPPOSITE of what we call “agenty behavior”. Executing a knowable strategy is the trivial part. Agenty-ness in the human rationalist sense is about cases where the model is too complicated to reflectively know or analyze formally.
First of all, these are definitely not opposites, and game-theoretic agency is about much more than just “executing a knowable strategy”. The basic point of embedded agency and the like is that, because of reflectivity etc, idealized game-theoretic agent behavior can only exist (or even be approximated) in the real world at an abstract level which throws out some information about the underlying territory. Game theoretic agency is still the original goal of the exercise; reflectivity and whatnot enter the picture because they’re a constraint, not because they’re part of what we mean by “agentiness”.
In terms of human rationality—of the sort in the sequences—the recurring theme is that we want to approximate idealized game-theoretic agency as best we can, despite complicated models, reflectivity, etc. Again, game-theoretic agency is the original goal; approximations enter the picture because complexity is a constraint. Nothing about that is contradictory.
Tying it back to the OP: we have a low-level model which may be too complex for “the agent” to represent/reason about directly. We abstract that into a high-level model. The agent is then an idealized game-theoretic agent within the high-level model, but the high-level model itself is lossy. The agent’s own model coincides with the high-level model—that’s the meaning of the clouds. But that still leaves the question of whether and to what extent the high-level model accurately reflects the low-level model—that’s the abstraction part.
Interesting take. When I see “agenty” used on this site and related blogs, it usually seems to map to something like self-actualization or percieved locus of control, more psychological frameworks. I’d not thought too much about how different (or similar) it was to “agent” in decision theory and game-theoretical usage, which is not about the feeling of control, but about behavior selection according to legible reasoning.
Again, decision theory/game theory are not about “executing a knowable strategy” or “behavior selection according to legible reasoning”. They’re about what goal-directed behavior means, especially under partial information and in the presence of other goal-directed systems. The theory of decisions/games is the theory of how to achieve goals. Whether a legible strategy achieves a goal is mostly incidental to decision/game theory—there are some games where legibility/illegibility could convey an advantage, but that’s not really something that most game theorists study.
EDIT: This answer isn’t very good, see my other one.
Good question. We could easily draw a diagram in which the two are separate—we’d have the “agent” node reading from one cloud and then influencing things outside of that cloud. But that case isn’t very interesting—most of what we call “agenty” behavior, and especially the diagonalization issues, are about the case where the actual model and the agent’s beliefs coincide. In particular, if we’re talking about ideal game-theoretic agents, we usually assume that both the rules of the game and each agent’s strategy are common knowledge—including off-equilibrium behavior.
So, for idealized game-theoretic agents, there is no separation between the actual model and the agent’s model—interventions on the actual model are reflected in the agent’s model.
That said, in the low-level model, the map and the territory will presumably always be separate. “When do they coincide?” is implicitly wrapped up in the question “when do non-agenty models abstract into agenty models?”. I view the potential mismatch between the two models as an abstraction failure—if they don’t match, then the agency-abstraction is broken.
I think there’s a contradiction here. Idealized game-theoretic agents are the OPPOSITE of what we call “agenty behavior”. Executing a knowable strategy is the trivial part. Agenty-ness in the human rationalist sense is about cases where the model is too complicated to reflectively know or analyze formally.
First of all, these are definitely not opposites, and game-theoretic agency is about much more than just “executing a knowable strategy”. The basic point of embedded agency and the like is that, because of reflectivity etc, idealized game-theoretic agent behavior can only exist (or even be approximated) in the real world at an abstract level which throws out some information about the underlying territory. Game theoretic agency is still the original goal of the exercise; reflectivity and whatnot enter the picture because they’re a constraint, not because they’re part of what we mean by “agentiness”.
In terms of human rationality—of the sort in the sequences—the recurring theme is that we want to approximate idealized game-theoretic agency as best we can, despite complicated models, reflectivity, etc. Again, game-theoretic agency is the original goal; approximations enter the picture because complexity is a constraint. Nothing about that is contradictory.
Tying it back to the OP: we have a low-level model which may be too complex for “the agent” to represent/reason about directly. We abstract that into a high-level model. The agent is then an idealized game-theoretic agent within the high-level model, but the high-level model itself is lossy. The agent’s own model coincides with the high-level model—that’s the meaning of the clouds. But that still leaves the question of whether and to what extent the high-level model accurately reflects the low-level model—that’s the abstraction part.
Interesting take. When I see “agenty” used on this site and related blogs, it usually seems to map to something like self-actualization or percieved locus of control, more psychological frameworks. I’d not thought too much about how different (or similar) it was to “agent” in decision theory and game-theoretical usage, which is not about the feeling of control, but about behavior selection according to legible reasoning.
Again, decision theory/game theory are not about “executing a knowable strategy” or “behavior selection according to legible reasoning”. They’re about what goal-directed behavior means, especially under partial information and in the presence of other goal-directed systems. The theory of decisions/games is the theory of how to achieve goals. Whether a legible strategy achieves a goal is mostly incidental to decision/game theory—there are some games where legibility/illegibility could convey an advantage, but that’s not really something that most game theorists study.