People often ask why MIRI researchers think decision theory is relevant for AGI safety. I, too, often wonder myself whether it’s as likely to be relevant as, say, program synthesis. But the basic argument for the relevance of decision theory was explained succinctly in Levitt (1999):
If robots are to put to more general uses, they will need to operate without human intervention, outdoors, on roads and in normal industrial and residential environments where unpredictable physical and visual events routinely occur. It will not be practical, or even safe, to halt robotic actions whenever the robot encounters an unexpected event or ambiguous visual interpretation.
Currently, commercial robots determine their actions mostly by control-theoretic feedback. Control-theoretic algorithms require the possibilities of what can happen in the world be represented in models embodied in software programs that allow the robot to pre-determine an appropriate action response to any task-relevant occurrence of visual events. When robots are used in open, uncontrolled environments, it will not be possible to provide the robot with a priori models of all the objects and dynamical events that might occur.
In order to decide what actions to take in response to un-modeled, unexpected or ambiguously interpreted events events in the world, robots will need to augment their processing beyond controlled feedback response, and engage in decision processes.
People often ask why MIRI researchers think decision theory is relevant for AGI safety. I, too, often wonder myself whether it’s as likely to be relevant as, say, program synthesis. But the basic argument for the relevance of decision theory was explained succinctly in Levitt (1999):