The agent, being of intelligence at least similar to a person’s, would determine that, unless it can guarantee the new more powerful agent is aligned to its goals, it shouldn’t improve itself.
People generally do not conclude that. Some things under the umbrella of “self-development” are indeed about pursuing the same goals more effectively, but a lot is about doing things that change those goals. The more “spiritual” or drug-induced experiences are undertaken for exactly that end. You can talk about your own Extrapolated Volition, but that seems to mean in practice changing yourself in ways you endorse after the fact, even if you would not have in advance.
I guess I don’t really see that in myself. If you offered me a brain chip that would make me smarter but made me stop caring for my family I simply wouldn’t do it. Maybe I’d meditate to make want to watch less TV, but that’s because watching TV isn’t really in what I’d consider my “core” desires.
If I know the change in advance then of course I won’t endorse it. But if I get my smartness upgraded, and as a result of being smarter come to discard some of my earlier values as trash, what then?
All changes of beliefs or values feel like I got closer to the truth. It is as if I carry around my own personal lightbulb, illuminating my location on the landscape of possible ideas and leaving everything that differs from it in the darkness of error. But the beacon moves with me. Wherever I stand, that is what I think right.
To have correct beliefs, I can attend to the process of how I got there — is it epistemically sound? — rather than complacently observe the beacon of enlightenment centre on whatever the place that I stand. But how shall I know what are really the right values?
People generally do not conclude that. Some things under the umbrella of “self-development” are indeed about pursuing the same goals more effectively, but a lot is about doing things that change those goals. The more “spiritual” or drug-induced experiences are undertaken for exactly that end. You can talk about your own Extrapolated Volition, but that seems to mean in practice changing yourself in ways you endorse after the fact, even if you would not have in advance.
What you do changes who you are.
I guess I don’t really see that in myself. If you offered me a brain chip that would make me smarter but made me stop caring for my family I simply wouldn’t do it. Maybe I’d meditate to make want to watch less TV, but that’s because watching TV isn’t really in what I’d consider my “core” desires.
If I know the change in advance then of course I won’t endorse it. But if I get my smartness upgraded, and as a result of being smarter come to discard some of my earlier values as trash, what then?
All changes of beliefs or values feel like I got closer to the truth. It is as if I carry around my own personal lightbulb, illuminating my location on the landscape of possible ideas and leaving everything that differs from it in the darkness of error. But the beacon moves with me. Wherever I stand, that is what I think right.
To have correct beliefs, I can attend to the process of how I got there — is it epistemically sound? — rather than complacently observe the beacon of enlightenment centre on whatever the place that I stand. But how shall I know what are really the right values?