Embedded Agency is the problem that an understanding of the theory of rational agents must account for the fact that the agents we create (and we ourselves) are inside the world or universe we are trying to affect, and not separated from it. This is in contrast with much current basic theory of AI or Rationality (such as Solomonoff induction or Bayesianism) which implicitly supposes a separation between the agent and the-things-the-agent-has-beliefs about. In other words, agents in this universe do not have Cartesian or dualistic boundaries like much of philosophy assumes, and are instead reductionist, that is agents are made up of non-agent parts like bits and atoms.
Embedded Agency is not a fully formalized research agenda, but Scott Garrabrant and Abram Demski have written the canonical explanation of the idea in their sequence Embedded Agency. This points to many of the core confusions we have about rational agency and attempts to tie them into a single picture.
Could someone clear up the following for me:
In which universe? The “current basic theory of AI” universe?
Could we add an intuitive explanation about what is meant by “Cartesian or dualistic boundaries”?
Not clear if “agents are made up of non-agent parts like bits and atoms” is for Embedded Agency or the old frame.
1. In our universe, as opposed to the “current basic theory of AI” universe.
2. From Arbital:
3. For embedded agency. In the old frame agents aren’t really made of anything.