If it’s important to me that my children have food, I will take the steps I think will lead to my children being fed.
My reward function in this case is whatever structures in my mind reinforce the taking of actions that are associated in certain ways with the structures that represent my children having food. Maybe there’s a subjective component to that (“warm glow”), maybe there isn’t.
A sufficiently advanced neuroscience allows me to point to structures in my own brain and say “Ah, see? That is where my preference for my children to have food is computed, that is where my belief that earning a salary increases the chances my children have food is computed, that is where my increased inclination to earn a salary is computed,” and so on and so forth. That is, it lets me identify the neural substrate(s) of my utility function(s).
So Omega hands me the appropriately advanced neuroscience and there I am, standing in front of the console that controls the appropriate machinery, knowing full well that the only reason I care about my child being fed is those circuits I’m seeing on the screen—that, for example, if an accidental brain lesion were to disrupt those circuits, I would no longer care whether my child were fed or not.
Omega’s gadget also allows me to edit those structures so that I no longer care about whether my child is fed. There’s the button right there. Do I press it?
If it’s important to me that my children have food, I will take the steps I think will lead to my children being fed.
My reward function in this case is whatever structures in my mind reinforce the taking of actions that are associated in certain ways with the structures that represent my children having food. Maybe there’s a subjective component to that (“warm glow”), maybe there isn’t.
A sufficiently advanced neuroscience allows me to point to structures in my own brain and say “Ah, see? That is where my preference for my children to have food is computed, that is where my belief that earning a salary increases the chances my children have food is computed, that is where my increased inclination to earn a salary is computed,” and so on and so forth. That is, it lets me identify the neural substrate(s) of my utility function(s).
So Omega hands me the appropriately advanced neuroscience and there I am, standing in front of the console that controls the appropriate machinery, knowing full well that the only reason I care about my child being fed is those circuits I’m seeing on the screen—that, for example, if an accidental brain lesion were to disrupt those circuits, I would no longer care whether my child were fed or not.
Omega’s gadget also allows me to edit those structures so that I no longer care about whether my child is fed. There’s the button right there. Do I press it?
I can’t see why I would.
Would you?