It’s a good thing—from their point of view. They probably think that there should be more paperclips. The term “should” makes sense in the context of a set of preferences.
No, it’s a paperclip-maximizing thing. From their point of view, and ours. No disagreement. They just care about what’s paperclip-maximizing, not what’s good.
IMO, in this context, “good” just means “favoured by this moral system”. An action that “should” be performed is just one that would be morally obligatory—according to the specified moral system. Both terms are relative to a set of moral standards.
I was talking as though a paperclip maximiser would have morals that reflected their values. You were apparently assuming the opposite. Which perspective is better would depend on which particular paperclip maximiser was being examined.
Personally, I think there are often good reasons for morals and values being in tune with one another.
It’s a good thing—from their point of view. They probably think that there should be more paperclips. The term “should” makes sense in the context of a set of preferences.
No, it’s a paperclip-maximizing thing. From their point of view, and ours. No disagreement. They just care about what’s paperclip-maximizing, not what’s good.
This is not a real point of disagreement.
IMO, in this context, “good” just means “favoured by this moral system”. An action that “should” be performed is just one that would be morally obligatory—according to the specified moral system. Both terms are relative to a set of moral standards.
I was talking as though a paperclip maximiser would have morals that reflected their values. You were apparently assuming the opposite. Which perspective is better would depend on which particular paperclip maximiser was being examined.
Personally, I think there are often good reasons for morals and values being in tune with one another.