I can’t actually understand/grok/predict what it is like to not exist, but I know that if I die, I will not learn or act anymore. That seems to be all that naturalized reasoning can give me, and all that is necessary for an AI too.
A naturalized agent’s hypotheses can be about world-states that include the agent, or world-states that don’t include the agent. A Cartesian agent’s hypotheses are all about the agent’s internal states, and different possible causes for those states, so the idea of ‘world-states that don’t include the agent’ can’t be directly represented. Even a halting program in AIXI’s hypothesis space isn’t really a prediction about how a world without AIXI would look; it’s more a prediction about how Everything (including AIXI) could come to an end.
Our ultimate goal in building an AI isn’t to optimize the internal features of the AI; it’s to optimize the rest of the world, with the AI functioning as a tool. So it seems likely that we’ll want our AI’s beliefs to look like pictures of an objective world (in which agents like the AI happen to exist, sometimes).
A Cartesian agent’s hypotheses are all about the agent’s internal states, and different possible causes for those states, so the idea of ‘world-states that don’t include the agent’ can’t be directly represented.
A sequence predictor’s predictions are all about the agent’s input tape states*, and different possible causes for those states. The hypotheses are programs that implement entire models of the Universe, and these can definitely directly represent world-states which don’t include the agent.
* More realistically, the states of the registers where the sensor data is placed.
ETA: I wonder if this intuition is caused by that fact that I am a practicing Bayesian statistician, so the distinction between posterior distributions and posterior predictive distributions is more salient to me.
I can’t actually understand/grok/predict what it is like to not exist, but I know that if I die, I will not learn or act anymore. That seems to be all that naturalized reasoning can give me, and all that is necessary for an AI too.
A naturalized agent’s hypotheses can be about world-states that include the agent, or world-states that don’t include the agent. A Cartesian agent’s hypotheses are all about the agent’s internal states, and different possible causes for those states, so the idea of ‘world-states that don’t include the agent’ can’t be directly represented. Even a halting program in AIXI’s hypothesis space isn’t really a prediction about how a world without AIXI would look; it’s more a prediction about how Everything (including AIXI) could come to an end.
Our ultimate goal in building an AI isn’t to optimize the internal features of the AI; it’s to optimize the rest of the world, with the AI functioning as a tool. So it seems likely that we’ll want our AI’s beliefs to look like pictures of an objective world (in which agents like the AI happen to exist, sometimes).
A sequence predictor’s predictions are all about the agent’s input tape states*, and different possible causes for those states. The hypotheses are programs that implement entire models of the Universe, and these can definitely directly represent world-states which don’t include the agent.
* More realistically, the states of the registers where the sensor data is placed.
ETA: I wonder if this intuition is caused by that fact that I am a practicing Bayesian statistician, so the distinction between posterior distributions and posterior predictive distributions is more salient to me.