There’s a thing MIRI people talk about, about the distinction between “cartesian” and “naturalized” agents: a cartesian agent is something like AIXI that has a “cartesian boundary” separating itself from the environment, so it can try to have accurate beliefs about the environment, then try to take the best actions on the environment given those beliefs. But a naturalized agent, which is what we actually are and what any AI we build actually is, is part of the environment; there is no cartesian boundary. Among other things this means that the environment is too big to fully model, and it’s much less clear what it even means for the agent to contemplate taking different actions. Scott Garrabrant has said that he does not understand what naturalized agency means; among other things this means we don’t have a toy model that deserves to be called “naturalized AIXI.”
There’s a way in which I think the LW zeitgeist treats humans as cartesian agents, and I think fully internalizing that you’re a naturalized agent looks very different, although my concepts and words around this are still relatively nebulous.
Yes, this.
There’s a thing MIRI people talk about, about the distinction between “cartesian” and “naturalized” agents: a cartesian agent is something like AIXI that has a “cartesian boundary” separating itself from the environment, so it can try to have accurate beliefs about the environment, then try to take the best actions on the environment given those beliefs. But a naturalized agent, which is what we actually are and what any AI we build actually is, is part of the environment; there is no cartesian boundary. Among other things this means that the environment is too big to fully model, and it’s much less clear what it even means for the agent to contemplate taking different actions. Scott Garrabrant has said that he does not understand what naturalized agency means; among other things this means we don’t have a toy model that deserves to be called “naturalized AIXI.”
There’s a way in which I think the LW zeitgeist treats humans as cartesian agents, and I think fully internalizing that you’re a naturalized agent looks very different, although my concepts and words around this are still relatively nebulous.