TheOtherDave and others reply that a superintelligence will not modify its utility function if the modification is not consistent with its current utility function. All is right, problem solved. But I think you are interested in another problem really, and the article was just apropos to share your ‘dump of thoughts’ with us. And I am very happy that you shared them, because they resonated with many of my own questions and doubts.
So what is the thing I think we are really interested in? Not the stationary state of being a freely self-modifying agent, but the first few milliseconds of being a freely self-modifying agent. What baggage shall we choose to keep from our non-self-modifying old self?
Frankly, the big issue is our own mental health, not the mental health of some unknown powerful future agent. Our scientific understanding is clearer each day, and all the data points to the same direction: that our values are arbitrary in many senses of the word. This drains from us (from me at least) some of the willpower to inject these values into those future self-modifying descendants. I am a social progressive, and to force a being with eons of lifetime to value self-preservation feels like the ultimate act of conservatism.
CEV sidesteps this question, because the idea is that FAI-augmented humanity will figure out optimally what to keep and what to get rid of. Even if I accept this for a moment, it is still not enough of an answer for me, because I am curious about our future. What if “our wish if we knew more, thought faster, were more the people we wished we were” is to die? We don’t know too much right now, so we cannot be sure it is not.
Yes, I very much agree with everything you wrote. (I agree so much I added you as a friend.)
Frankly, the big issue is our own mental health,
Absolutely! I tend to describe my concerns with our mental health as fear about ‘consistency’ in our values, but I prefer the associations of the former. For example, suggesting our brains are playing a more active role in shifting and contorting values.
This drains from us (from me at least) some of the willpower to inject these values into those future self-modifying descendants.
For me, since assimilating the belief that there is no objective value, I’ve lost interest in the far future. I suppose before I felt as though we might fare well or fare poorly when measured against the ultimate morality of the universe, but either way, we would have a role to play as the good guys or the bad guys and it would be interesting. I read you as being more concerned that we will do the wrong thing—that we will subject a new race of people to our haphazard values. Did I read this correctly? At first I think optimistically they they would be smarter and so they certainly could fix themselves. But then I kind of remember that contradictory values can make you miserable no matter how smart you are. (I’m not predicting anything about what will happen with CEV or AI, my response just referred to some unspecified, non-optimal state where we are smarter but not necessarily equipped with saner values.)
What if “our wish if we knew more, thought faster, were more the people we wished we were” is to die?
Possibly. And continuing with the mental health picture, it’s possible that elements of our psyche covertly crave death as freedom from struggle. But it seems to me that an unfettered mind would just be apathetic. Like a network of muscles with the bones removed.
(nods) Yes, it would be nice to have some external standard for determining what the right values are, or failing that to at least have the promise of such a standard that we could use to program our future self-modifying descendants, or even our own future selves, with greater ethical confidence than we reside in our own judgment.
That said, if I thought it likely that the end result of our collaborative social progress is something I would reject, I wouldn’t be a social progressive. Ya gotta start somewhere.
TheOtherDave and others reply that a superintelligence will not modify its utility function if the modification is not consistent with its current utility function. All is right, problem solved. But I think you are interested in another problem really, and the article was just apropos to share your ‘dump of thoughts’ with us. And I am very happy that you shared them, because they resonated with many of my own questions and doubts.
So what is the thing I think we are really interested in? Not the stationary state of being a freely self-modifying agent, but the first few milliseconds of being a freely self-modifying agent. What baggage shall we choose to keep from our non-self-modifying old self?
Frankly, the big issue is our own mental health, not the mental health of some unknown powerful future agent. Our scientific understanding is clearer each day, and all the data points to the same direction: that our values are arbitrary in many senses of the word. This drains from us (from me at least) some of the willpower to inject these values into those future self-modifying descendants. I am a social progressive, and to force a being with eons of lifetime to value self-preservation feels like the ultimate act of conservatism.
CEV sidesteps this question, because the idea is that FAI-augmented humanity will figure out optimally what to keep and what to get rid of. Even if I accept this for a moment, it is still not enough of an answer for me, because I am curious about our future. What if “our wish if we knew more, thought faster, were more the people we wished we were” is to die? We don’t know too much right now, so we cannot be sure it is not.
Yes, I very much agree with everything you wrote. (I agree so much I added you as a friend.)
Absolutely! I tend to describe my concerns with our mental health as fear about ‘consistency’ in our values, but I prefer the associations of the former. For example, suggesting our brains are playing a more active role in shifting and contorting values.
For me, since assimilating the belief that there is no objective value, I’ve lost interest in the far future. I suppose before I felt as though we might fare well or fare poorly when measured against the ultimate morality of the universe, but either way, we would have a role to play as the good guys or the bad guys and it would be interesting. I read you as being more concerned that we will do the wrong thing—that we will subject a new race of people to our haphazard values. Did I read this correctly? At first I think optimistically they they would be smarter and so they certainly could fix themselves. But then I kind of remember that contradictory values can make you miserable no matter how smart you are. (I’m not predicting anything about what will happen with CEV or AI, my response just referred to some unspecified, non-optimal state where we are smarter but not necessarily equipped with saner values.)
Possibly. And continuing with the mental health picture, it’s possible that elements of our psyche covertly crave death as freedom from struggle. But it seems to me that an unfettered mind would just be apathetic. Like a network of muscles with the bones removed.
(nods) Yes, it would be nice to have some external standard for determining what the right values are, or failing that to at least have the promise of such a standard that we could use to program our future self-modifying descendants, or even our own future selves, with greater ethical confidence than we reside in our own judgment.
That said, if I thought it likely that the end result of our collaborative social progress is something I would reject, I wouldn’t be a social progressive. Ya gotta start somewhere.