I like the idea of agency being some sweet spot between being too simple and too complex, yes. Though I’m not sure I agree that if we can fully understand the algorithm, then we won’t view it as an agent. I think the algorithm for this point particle is simple enough for us to fully understand, but due to the stochastic nature of the optimization algorithm, we can never fully predict it. So I guess I’d say agency isn’t a sweet spot in the amount of computation needed, but rather in the amount of stochasticity perhaps?
As for other examples of “doing something so well we get a strange feeling,” the chess example wouldn’t be my go-to, since the action space there is somehow “small” since it is discrete and finite. I’m more thinking of the difference between a human ballet dancer, and an ideal robotic ballet dancer—that slight imperfection makes the human somehow relatable for us. E.g., in CGI you have to make your animated characters make some unnecessary movements, each step must be different than any other, etc. We often admire hand-crafted art more than perfect machine-generated decorations for the same sort of minute asymmetry that makes it relatable, and thus admirable. In voice recording, you often record the song twice for the L and R channels, rather than just copying (see ‘double tracking’) - the slight differences make the sound “bigger” and “more alive.” Etc, etc.
I think that if we fully understood the algorithm and had chunked it in our heads so we could just imagine manipulating it any way we liked, then I think we would view it as less agenty. But of course, a lot of our intuitions are rough heuristics and they might misfire in various ways and make us think “agent!” in a way we don’t reflectively endorse (like we don’t endorse “a smiley face--> a person”).
Or, you know, my attempted abstraction of my agent intuitions fails in some way. I think that the stochasticity thing might play a part in being agent. Like, maybe because most agents are power seeking and power seeking behaviour is about leaving lots of options live and thus increasing other’s uncertainty about your future actions. Wasn’t there that paper about entropy which someone linked to in the comments of one of TurnTrout’s “rethinking impact” posts? It was about modeling entropy in a way that shared mathematical structure with impact measures. Of course, there’s also some kinds of logical uncertainty when you model an agent modeling you.
As for the example of dancing, CGI and music I’d say that’s more about “natural/human” vs “unnatural/inhuman” than “agent” vs “not-agent”, though there’s a large inner product between the two axis.
I like the idea of agency being some sweet spot between being too simple and too complex, yes. Though I’m not sure I agree that if we can fully understand the algorithm, then we won’t view it as an agent. I think the algorithm for this point particle is simple enough for us to fully understand, but due to the stochastic nature of the optimization algorithm, we can never fully predict it. So I guess I’d say agency isn’t a sweet spot in the amount of computation needed, but rather in the amount of stochasticity perhaps?
As for other examples of “doing something so well we get a strange feeling,” the chess example wouldn’t be my go-to, since the action space there is somehow “small” since it is discrete and finite. I’m more thinking of the difference between a human ballet dancer, and an ideal robotic ballet dancer—that slight imperfection makes the human somehow relatable for us. E.g., in CGI you have to make your animated characters make some unnecessary movements, each step must be different than any other, etc. We often admire hand-crafted art more than perfect machine-generated decorations for the same sort of minute asymmetry that makes it relatable, and thus admirable. In voice recording, you often record the song twice for the L and R channels, rather than just copying (see ‘double tracking’) - the slight differences make the sound “bigger” and “more alive.” Etc, etc.
Does this make sense?
I think that if we fully understood the algorithm and had chunked it in our heads so we could just imagine manipulating it any way we liked, then I think we would view it as less agenty. But of course, a lot of our intuitions are rough heuristics and they might misfire in various ways and make us think “agent!” in a way we don’t reflectively endorse (like we don’t endorse “a smiley face--> a person”).
Or, you know, my attempted abstraction of my agent intuitions fails in some way. I think that the stochasticity thing might play a part in being agent. Like, maybe because most agents are power seeking and power seeking behaviour is about leaving lots of options live and thus increasing other’s uncertainty about your future actions. Wasn’t there that paper about entropy which someone linked to in the comments of one of TurnTrout’s “rethinking impact” posts? It was about modeling entropy in a way that shared mathematical structure with impact measures. Of course, there’s also some kinds of logical uncertainty when you model an agent modeling you.
As for the example of dancing, CGI and music I’d say that’s more about “natural/human” vs “unnatural/inhuman” than “agent” vs “not-agent”, though there’s a large inner product between the two axis.
just updated the post to add this clarification about “too perfect”—thanks for your question!