The problem is that when we expect an AI to build a model of the environment from scratch
Is this a wise approach? What does “scratch” mean?
“Don’t worry about the rest” isn’t something we want an AI to do. If its utility function makes no explicit reference to the rest of the universe, it has no incentive not to replace it with more computing power that it can use to better optimize the region that it does care about.
That’s what the boundary conditions are for. A fully formalized version of “don’t trust as valid any computations run outside of your region” seems like the easiest way to disincentivize the AI from trying to run computations in the rest of the universe.
Is this a wise approach? What does “scratch” mean?
What I had in mind while writing this was Solomonoff induction. If the AI’s model of the universe could be any computable program, it is hard to detect even a paperclip (impossible in full generality due to Rice’s theorem). On LW, the phrase ‘ontological crisis’ is used to refer to the problem of translating a utility function described in terms of one model of the universe into something that can be use in a different, presumably more accurate, model of the universe. The transition from classical physics to quantum mechanics is an illustrative example; why should or shouldn’t our decisions under many worlds be approximately the same as they would be in a classical universe?
As for whether this is a good idea, it seems much harder, if even possible, to build an AI that doesn’t need to navigate such transitions as it is to build one that can do so.
That’s what the boundary conditions are for. A fully formalized version of “don’t trust as valid any computations run outside of your region” seems like the easiest way to disincentivize the AI from trying to run computations in the rest of the universe.
This still seems very dangerous. If there is a boundary beyond which it has no incentive to preserve anything, I think that at least some things outside of that boundary get destroyed by default. Concretely, what if the AI creates self-replicating nanobots and has some system within its region to prevent them from replicating uncontrollably, but there is no such protection in place in the rest of the universe?
Is this a wise approach? What does “scratch” mean?
That’s what the boundary conditions are for. A fully formalized version of “don’t trust as valid any computations run outside of your region” seems like the easiest way to disincentivize the AI from trying to run computations in the rest of the universe.
What I had in mind while writing this was Solomonoff induction. If the AI’s model of the universe could be any computable program, it is hard to detect even a paperclip (impossible in full generality due to Rice’s theorem). On LW, the phrase ‘ontological crisis’ is used to refer to the problem of translating a utility function described in terms of one model of the universe into something that can be use in a different, presumably more accurate, model of the universe. The transition from classical physics to quantum mechanics is an illustrative example; why should or shouldn’t our decisions under many worlds be approximately the same as they would be in a classical universe?
As for whether this is a good idea, it seems much harder, if even possible, to build an AI that doesn’t need to navigate such transitions as it is to build one that can do so.
This still seems very dangerous. If there is a boundary beyond which it has no incentive to preserve anything, I think that at least some things outside of that boundary get destroyed by default. Concretely, what if the AI creates self-replicating nanobots and has some system within its region to prevent them from replicating uncontrollably, but there is no such protection in place in the rest of the universe?