Probably the single biggest blackbox in classic decision theory is the Cartesian assumption of an ‘agent’ which is an atomic entity sealed inside a barrier which is communicating with/acting in an environment which has no access to or prediction about the agent. This is a black box not for convenience, nor for generality, but because if you open it, you just see a little symbol labeled ‘agent’, and beyond that, you just have to shrug and say, ‘magic’.
The ‘agent’ node is where a lot of reflexive problems like Newcomb or smoking lesion or AIXI wireheading come in. In comparison to the problem that ‘we have no way at all to define an “agent” in a rigorous way reducible to atoms and handling correctly cases like the environment screwing with the agent’s brain’, problems like “where do we get a probability” are much easier—standard reductionist moves like ‘the brain is just a gigantic Bayesian model/deep ensemble and that’s how you can assign probabilities to everything you can think of’ look like they work, whereas for defining an agent with permeable boundaries & which is made up of the exact same parts as the environment, we’re left struggling to even say what a solution might look like.
(It would need to handle all of the dynamics inside and outside an agent, from memes colonizing a brain, to clonal lineages of blood or immune cells colonizing your body over a lifetime, to commensals like gut bacteria, to bacteriophages deciding whether to lyse or lie quiescent inside them, to infections evolving within an individual such as the COVID strains that emerged within immuno-compromised individuals, to passive latent infections waiting for immune weakness or old age to spring back to life, to ‘neural darwinism’ of neurons & connections, to transmission-biasing genes or jumping transposons or sex-biased genes or antagonistic pleiotropic genes or… And this is just a small selection of the stuff which is ‘under the skin’! Where is the ‘agent’ which emerges out of all these different levels & units of analysis which one could potentially apply Price’s equation or Bellman equations or VNM coherency? We get away with ignoring this most of the time and modeling the ‘obvious’ agents the ‘obvious’ way—but why does this work so well, why do sometimes you do in fact have to unsee an ‘agent’ as a bunch of sub-agents, and what justifies each particular level of analysis, beyond just the obvious pragmatic response of ‘this particular approach seems to be what’s useful in this case’?)
Hence, Agent Foundations and trying to nail down workable abstractions which can handle, say, cellular automata agents, and which can be composed and copied while preserving agency or identity.
Seems unlikely to me but I do wonder: after we break down the agent all the way to atoms and build it back up level by level, will we realize that the physics sim was all we needed all along? Perhaps just a coarser sim for bigger stuff?
Probably the single biggest blackbox in classic decision theory is the Cartesian assumption of an ‘agent’ which is an atomic entity sealed inside a barrier which is communicating with/acting in an environment which has no access to or prediction about the agent.
I’ve always understood this to be the equivalent of spherical cows.
Of course it doesn’t, and can’t actually, exist in the real world.
Considering that for real humans even something as simple as severing the connection between left and right hemispheres of the brain causes identity to break down, there likely are no human ‘agents’ in the strict sense.
Probably the single biggest blackbox in classic decision theory is the Cartesian assumption of an ‘agent’ which is an atomic entity sealed inside a barrier which is communicating with/acting in an environment which has no access to or prediction about the agent. This is a black box not for convenience, nor for generality, but because if you open it, you just see a little symbol labeled ‘agent’, and beyond that, you just have to shrug and say, ‘magic’.
The ‘agent’ node is where a lot of reflexive problems like Newcomb or smoking lesion or AIXI wireheading come in. In comparison to the problem that ‘we have no way at all to define an “agent” in a rigorous way reducible to atoms and handling correctly cases like the environment screwing with the agent’s brain’, problems like “where do we get a probability” are much easier—standard reductionist moves like ‘the brain is just a gigantic Bayesian model/deep ensemble and that’s how you can assign probabilities to everything you can think of’ look like they work, whereas for defining an agent with permeable boundaries & which is made up of the exact same parts as the environment, we’re left struggling to even say what a solution might look like.
(It would need to handle all of the dynamics inside and outside an agent, from memes colonizing a brain, to clonal lineages of blood or immune cells colonizing your body over a lifetime, to commensals like gut bacteria, to bacteriophages deciding whether to lyse or lie quiescent inside them, to infections evolving within an individual such as the COVID strains that emerged within immuno-compromised individuals, to passive latent infections waiting for immune weakness or old age to spring back to life, to ‘neural darwinism’ of neurons & connections, to transmission-biasing genes or jumping transposons or sex-biased genes or antagonistic pleiotropic genes or… And this is just a small selection of the stuff which is ‘under the skin’! Where is the ‘agent’ which emerges out of all these different levels & units of analysis which one could potentially apply Price’s equation or Bellman equations or VNM coherency? We get away with ignoring this most of the time and modeling the ‘obvious’ agents the ‘obvious’ way—but why does this work so well, why do sometimes you do in fact have to unsee an ‘agent’ as a bunch of sub-agents, and what justifies each particular level of analysis, beyond just the obvious pragmatic response of ‘this particular approach seems to be what’s useful in this case’?)
Hence, Agent Foundations and trying to nail down workable abstractions which can handle, say, cellular automata agents, and which can be composed and copied while preserving agency or identity.
Seems unlikely to me but I do wonder: after we break down the agent all the way to atoms and build it back up level by level, will we realize that the physics sim was all we needed all along? Perhaps just a coarser sim for bigger stuff?
I’ve always understood this to be the equivalent of spherical cows.
Of course it doesn’t, and can’t actually, exist in the real world.
Considering that for real humans even something as simple as severing the connection between left and right hemispheres of the brain causes identity to break down, there likely are no human ‘agents’ in the strict sense.