I don’t quite understand what you mean by “doing something so well we get a strange feeling”. Would you include e.g. optimal play in an endgame of chess, which was precomputed, as an instance?
In that case, I’d guess that maybe the generator of the feeling is the fact that the moves are pre-computed. Much of the optimisation was done before hand, and is not visible, it is the kind of thing you could imagine being hardcoded.
And I think the reason that hardcoded behaviour feels non-agentic is because we have the ability to hold this optimal policy in our heads and fully understand it. Any description of “goals” and “beliefs” and “optimisation” which might generate this policy is about as long as the policy itself, and both are short enough that we can understand them.
Agency seems like a sweet spot. The system is complex enough that we need to ascribe simplified descriptions of “goals” and “beliefs” and “intentionality” to predict its behaviour and not simple enough that we can completely simulate its behaviour in our heads, or perhaps even with any computation method we feel we can fully understand. But it is not so complex that a “goal+belief+intention” description is beyond our capacity, and its actions seem inscrutable.
How’s this related to making mistakes? Well, it the “agent” description seems to better account for errors than the system being a perfectly predictable optimisation process with the simple goal of “get to the bottom”. And its optimisation is not so rapid that it feels like we’re just watching the end policy. We get to see it “thinking” live. As a result of this, I’d suspect that if I understood the algorithm better, I’d feel like it was less agentic.
I like the idea of agency being some sweet spot between being too simple and too complex, yes. Though I’m not sure I agree that if we can fully understand the algorithm, then we won’t view it as an agent. I think the algorithm for this point particle is simple enough for us to fully understand, but due to the stochastic nature of the optimization algorithm, we can never fully predict it. So I guess I’d say agency isn’t a sweet spot in the amount of computation needed, but rather in the amount of stochasticity perhaps?
As for other examples of “doing something so well we get a strange feeling,” the chess example wouldn’t be my go-to, since the action space there is somehow “small” since it is discrete and finite. I’m more thinking of the difference between a human ballet dancer, and an ideal robotic ballet dancer—that slight imperfection makes the human somehow relatable for us. E.g., in CGI you have to make your animated characters make some unnecessary movements, each step must be different than any other, etc. We often admire hand-crafted art more than perfect machine-generated decorations for the same sort of minute asymmetry that makes it relatable, and thus admirable. In voice recording, you often record the song twice for the L and R channels, rather than just copying (see ‘double tracking’) - the slight differences make the sound “bigger” and “more alive.” Etc, etc.
I think that if we fully understood the algorithm and had chunked it in our heads so we could just imagine manipulating it any way we liked, then I think we would view it as less agenty. But of course, a lot of our intuitions are rough heuristics and they might misfire in various ways and make us think “agent!” in a way we don’t reflectively endorse (like we don’t endorse “a smiley face--> a person”).
Or, you know, my attempted abstraction of my agent intuitions fails in some way. I think that the stochasticity thing might play a part in being agent. Like, maybe because most agents are power seeking and power seeking behaviour is about leaving lots of options live and thus increasing other’s uncertainty about your future actions. Wasn’t there that paper about entropy which someone linked to in the comments of one of TurnTrout’s “rethinking impact” posts? It was about modeling entropy in a way that shared mathematical structure with impact measures. Of course, there’s also some kinds of logical uncertainty when you model an agent modeling you.
As for the example of dancing, CGI and music I’d say that’s more about “natural/human” vs “unnatural/inhuman” than “agent” vs “not-agent”, though there’s a large inner product between the two axis.
Some thoughts:
I don’t quite understand what you mean by “doing something so well we get a strange feeling”. Would you include e.g. optimal play in an endgame of chess, which was precomputed, as an instance?
In that case, I’d guess that maybe the generator of the feeling is the fact that the moves are pre-computed. Much of the optimisation was done before hand, and is not visible, it is the kind of thing you could imagine being hardcoded.
And I think the reason that hardcoded behaviour feels non-agentic is because we have the ability to hold this optimal policy in our heads and fully understand it. Any description of “goals” and “beliefs” and “optimisation” which might generate this policy is about as long as the policy itself, and both are short enough that we can understand them.
Agency seems like a sweet spot. The system is complex enough that we need to ascribe simplified descriptions of “goals” and “beliefs” and “intentionality” to predict its behaviour and not simple enough that we can completely simulate its behaviour in our heads, or perhaps even with any computation method we feel we can fully understand. But it is not so complex that a “goal+belief+intention” description is beyond our capacity, and its actions seem inscrutable.
How’s this related to making mistakes? Well, it the “agent” description seems to better account for errors than the system being a perfectly predictable optimisation process with the simple goal of “get to the bottom”. And its optimisation is not so rapid that it feels like we’re just watching the end policy. We get to see it “thinking” live. As a result of this, I’d suspect that if I understood the algorithm better, I’d feel like it was less agentic.
I like the idea of agency being some sweet spot between being too simple and too complex, yes. Though I’m not sure I agree that if we can fully understand the algorithm, then we won’t view it as an agent. I think the algorithm for this point particle is simple enough for us to fully understand, but due to the stochastic nature of the optimization algorithm, we can never fully predict it. So I guess I’d say agency isn’t a sweet spot in the amount of computation needed, but rather in the amount of stochasticity perhaps?
As for other examples of “doing something so well we get a strange feeling,” the chess example wouldn’t be my go-to, since the action space there is somehow “small” since it is discrete and finite. I’m more thinking of the difference between a human ballet dancer, and an ideal robotic ballet dancer—that slight imperfection makes the human somehow relatable for us. E.g., in CGI you have to make your animated characters make some unnecessary movements, each step must be different than any other, etc. We often admire hand-crafted art more than perfect machine-generated decorations for the same sort of minute asymmetry that makes it relatable, and thus admirable. In voice recording, you often record the song twice for the L and R channels, rather than just copying (see ‘double tracking’) - the slight differences make the sound “bigger” and “more alive.” Etc, etc.
Does this make sense?
I think that if we fully understood the algorithm and had chunked it in our heads so we could just imagine manipulating it any way we liked, then I think we would view it as less agenty. But of course, a lot of our intuitions are rough heuristics and they might misfire in various ways and make us think “agent!” in a way we don’t reflectively endorse (like we don’t endorse “a smiley face--> a person”).
Or, you know, my attempted abstraction of my agent intuitions fails in some way. I think that the stochasticity thing might play a part in being agent. Like, maybe because most agents are power seeking and power seeking behaviour is about leaving lots of options live and thus increasing other’s uncertainty about your future actions. Wasn’t there that paper about entropy which someone linked to in the comments of one of TurnTrout’s “rethinking impact” posts? It was about modeling entropy in a way that shared mathematical structure with impact measures. Of course, there’s also some kinds of logical uncertainty when you model an agent modeling you.
As for the example of dancing, CGI and music I’d say that’s more about “natural/human” vs “unnatural/inhuman” than “agent” vs “not-agent”, though there’s a large inner product between the two axis.
just updated the post to add this clarification about “too perfect”—thanks for your question!