Unfortunately, the interaction between these components is extraordinarily complex, and many puzzles remain.
This seems pretty important regarding CEV?
What about if I don’t want to do something, don’t like to experience doing it but feel that I am morally obligated to do it? In what category does such a drive fall, or is one just confused in that case? It seems to me that it also has to do with pleasure, but a solely negative incentive. One might feel worse not doing it but one won’t feel good either. For example, if you kick the fat man onto the rails facing the Trolley problem you’ll feel bad but maybe not as bad as if you didn’t do it. Or is such thinking not part of the wanting/liking/learning system at all? The problem with this is, what is it that CEV is going to extrapolate, wanting/liking or what we think we ought to want? It seems that Yudkowsky only advocates CEV if it turns out that humanity decides to do what it ought to do and not what it might on some other level want to do:
...generic procedures meant to prevent CEV from running if 80% of humanity turns out to be selfish bastards… (Eliezer Yudkowsky 14 February 2011 06:43:09 AM)
An example would be what humanity wants to do with the limited resources in the universe versus what it ought to do with them. What if it turns out that humanity wants to waste all resources to support a single galactic civilisation for a few billion years rather than 100 times as many beings for as long as physically possible but with a reduced average amount of pleasure?
So is the SIAI going to extrapolate our reward system or high-level ethical decision making? Decision-utility versus experience-utility?
This seems pretty important regarding CEV?
What about if I don’t want to do something, don’t like to experience doing it but feel that I am morally obligated to do it? In what category does such a drive fall, or is one just confused in that case? It seems to me that it also has to do with pleasure, but a solely negative incentive. One might feel worse not doing it but one won’t feel good either. For example, if you kick the fat man onto the rails facing the Trolley problem you’ll feel bad but maybe not as bad as if you didn’t do it. Or is such thinking not part of the wanting/liking/learning system at all? The problem with this is, what is it that CEV is going to extrapolate, wanting/liking or what we think we ought to want? It seems that Yudkowsky only advocates CEV if it turns out that humanity decides to do what it ought to do and not what it might on some other level want to do:
An example would be what humanity wants to do with the limited resources in the universe versus what it ought to do with them. What if it turns out that humanity wants to waste all resources to support a single galactic civilisation for a few billion years rather than 100 times as many beings for as long as physically possible but with a reduced average amount of pleasure?
So is the SIAI going to extrapolate our reward system or high-level ethical decision making? Decision-utility versus experience-utility?
Yes—Yvain pointed this out also.