I don’t see why a machine that is able to make plans is the same as a machine that is able to execute those plans. For example, I can envision a machine that is able to generate the text describing with a lot of detail how to damage the economy of a country X and not necessarily having the power to execute it unless there are humans behind implementing those actions. Imagination and action are different things.
I suspect one of the generators of disagreements here is that MIRI folks don’t think imagination and action are (fundamentally) different things.
Like, there’s an intuitive human distinction between “events that happen inside your brain” and “events that happen outside your brain”. And there’s an intuitive human distinction between “controlling the direction of thoughts inside your brain so that you can reach useful conclusions” and “controlling the direction of events outside your brain so that you can reach useful outcomes”.
But it isn’t trivial to get an AGI system to robustly recognize and respect that exact distinction, so that it optimizes only ‘things inside its head’ (while nonetheless producing outputs that are useful for external events and are entangled with information about the external world). And it’s even less trivial to make an AGI system robustly incapable of acting on the physical world, while having all the machinery for doing amazing reasoning about the physical world, and for taking all the internal actions required to perform that reasoning.
I suspect one of the generators of disagreements here is that MIRI folks don’t think imagination and action are (fundamentally) different things.
Like, there’s an intuitive human distinction between “events that happen inside your brain” and “events that happen outside your brain”. And there’s an intuitive human distinction between “controlling the direction of thoughts inside your brain so that you can reach useful conclusions” and “controlling the direction of events outside your brain so that you can reach useful outcomes”.
But it isn’t trivial to get an AGI system to robustly recognize and respect that exact distinction, so that it optimizes only ‘things inside its head’ (while nonetheless producing outputs that are useful for external events and are entangled with information about the external world). And it’s even less trivial to make an AGI system robustly incapable of acting on the physical world, while having all the machinery for doing amazing reasoning about the physical world, and for taking all the internal actions required to perform that reasoning.