What are you trying to achieve with these questions?
If you just think the questions are entertaining to think about, you might find reading the Sequences worthwhile, as they and the discussions around them explore many of these questions at some length.
If you’re trying to explore something more targeted, I’ve failed to decipher what it might be.
It may even preserve its own values to continue achieving the goal
This is backwards. An intelligent system selects goals that, if achieved, optimize the world according to its values.
I change it so I feel that way ALL THE TIME.
The local jargon for this is “wireheading.” And, sure, if you don’t value anything as much or more than you value pleasure (or whatever it is you get from watching Seinfeld, then you might choose to wirehead given the option.
But, again, it matters whether you value something in the first place. If something else is important to you besides pleasure, then you won’t necessarily trade everything else for more pleasure.
The local consensus is that what humans value is complex; that there is no single thing (such as pleasure) to which human values reduce. Personally, I’m inclined to agree with that. But, well, you tell me: do you value anything other than pleasure?
It’s a bit more than entertaining, since I plan to upload as soon as it’s a mature technology. My concern is that I will degenerate into a wirehead, given the complete freedom to do so. Human values may be complex, but as far as I can tell there’s no guarantee that the values of an uploaded mind will remain complex.
Will be going over the Sequences. Thanks buddy. :)
There is in fact no guarantee that the values of an uploaded mind will remain complex.
There is also no guarantee that the values of a mind implemented in meat will remain complex as we gain more understanding of and control over our own brains.
The term “value drift” is sometimes used around here to refer to the process whereby the values of intelligent systems change over time, regardless of the infrastructure on which those systems are implemented. It’s generally seen as a risk that needs to be addressed for any sufficiently powerful self-modifying system, though harder to address for human minds than for sensibly designed systems.
What are you trying to achieve with these questions?
If you just think the questions are entertaining to think about, you might find reading the Sequences worthwhile, as they and the discussions around them explore many of these questions at some length.
If you’re trying to explore something more targeted, I’ve failed to decipher what it might be.
This is backwards. An intelligent system selects goals that, if achieved, optimize the world according to its values.
The local jargon for this is “wireheading.” And, sure, if you don’t value anything as much or more than you value pleasure (or whatever it is you get from watching Seinfeld, then you might choose to wirehead given the option.
But, again, it matters whether you value something in the first place. If something else is important to you besides pleasure, then you won’t necessarily trade everything else for more pleasure.
The local consensus is that what humans value is complex; that there is no single thing (such as pleasure) to which human values reduce. Personally, I’m inclined to agree with that. But, well, you tell me: do you value anything other than pleasure?
It’s a bit more than entertaining, since I plan to upload as soon as it’s a mature technology. My concern is that I will degenerate into a wirehead, given the complete freedom to do so. Human values may be complex, but as far as I can tell there’s no guarantee that the values of an uploaded mind will remain complex.
Will be going over the Sequences. Thanks buddy. :)
There is in fact no guarantee that the values of an uploaded mind will remain complex.
There is also no guarantee that the values of a mind implemented in meat will remain complex as we gain more understanding of and control over our own brains.
The term “value drift” is sometimes used around here to refer to the process whereby the values of intelligent systems change over time, regardless of the infrastructure on which those systems are implemented. It’s generally seen as a risk that needs to be addressed for any sufficiently powerful self-modifying system, though harder to address for human minds than for sensibly designed systems.
You might find the discussion here relevant.