When I build a bridge in a game, I get an in-game reward. I don’t get easier transport to anywhere. If I neglect to build the bridge or play the game at all, I still get to use all the bridges otherwise available to me. ‘Real’ bridges are at the top level of reality available to me. Even the simulation hypothesis does not make these bridges a game.
Why do I want to cross the bridge? To not be bored, to find my love, or to meet some other human value. The AI could do that for me too, and cut out the need for transport. If we follow that logic even a short way, it would be obvious that we don’t want the AI doing certain things for us. If there is danger of us being harmed because the FAI could help but won’t it need merely help a little more, getting closer to those things we want to do ourselves. If we’re in danger of being harmed by our own laziness, it need only back off. (It might do this at the level of the entires species, for all time, so individuals might be bored or angry or not cross rivers as soon as they would like, but it might optimize for everybody moment to moment.)
If there are things we couldn’t stand to have a machine do, and couldn’t stand for it to not help us with, I think those would be incoherent volitions.
When I build a bridge in a game, I get an in-game reward. I don’t get easier transport to anywhere. If I neglect to build the bridge or play the game at all, I still get to use all the bridges otherwise available to me. ‘Real’ bridges are at the top level of reality available to me. Even the simulation hypothesis does not make these bridges a game.
Why do I want to cross the bridge? To not be bored, to find my love, or to meet some other human value. The AI could do that for me too, and cut out the need for transport. If we follow that logic even a short way, it would be obvious that we don’t want the AI doing certain things for us. If there is danger of us being harmed because the FAI could help but won’t it need merely help a little more, getting closer to those things we want to do ourselves. If we’re in danger of being harmed by our own laziness, it need only back off. (It might do this at the level of the entires species, for all time, so individuals might be bored or angry or not cross rivers as soon as they would like, but it might optimize for everybody moment to moment.)
If there are things we couldn’t stand to have a machine do, and couldn’t stand for it to not help us with, I think those would be incoherent volitions.