In other words, it sounds like you’re saying human values are complex because all values are complex. But that doesn’t by itself prove that human values are much more complex than those of a pleasure maximizer, which is a claim that I take to be included in the “value is complex”/”thou art godshatter” idea.
ETA: Though your point may well be sufficient to support the particular implication that we’re unlikely to get the values right by hard-coding them or having them randomly emerge.
Though your point may well be sufficient to support the particular implication that we’re unlikely to get the values right by hard-coding them or having them randomly emerge.
If someone thinks it’s pretty likely that their preferences (as endorsed in reflective equilibrium) can be captured by some form of utilitarianism, then their time could easily be better spent trying to narrow down the exact form of utilitarianism (with the aim of hard-coding it) instead of trying to figure out how to specify and implement something like CEV.
Utilitarians haven’t made that much progress on these issues in the last 60 years, and there are a lot of them. Do you have a further argument for why this could easily be the case?
There’s little progress in the sense of resolving disagreements, but if someone thinks what is important is not public consensus on a particular form of utilitarianism, but just what they would endorse in reflective equilibrium as an individual, and they already have fairly strong intuitions on what kind of utilitarianism is right (which some people seem to), then that seems sufficient to make “narrow down the exact form of utilitarianism” a potentially better plan than “implement CEV” from their perspective.
I meant that there’s been little progress in the sense of generating theories precise enough to offer concrete recommendations, things that might be coded into an AI, e.g. formal criteria for identifying preferences, pains, and pleasures in the world (beyond pointing to existing humans and animals, which doesn’t pin down the content of utilitronium), or a clear way to value different infinitely vast worlds (with all the rearrangement issues discussed in Bostrom’s “Infinitarian Challenge” paper). This isn’t just a matter of persistent moral disagreement, but a lack of any comprehensive candidates that actually tell you what to do in particular situations rather than having massive lacunae that are filled in by consideration of individual cases and local intuitions.
then that seems sufficient to make “narrow down the exact form of utilitarianism” a potentially better plan than “implement CEV” from their perspective
This seems to me more about the “C” than the “EV.” I think such a utilitarian should still be strongly concerned with having at least their reflective equilibrium extrapolated. Even a little uncertainty about many dimensions means probably going wrong, and it seems that reasonable uncertainty about several of these things (e.g. infinite worlds and implications for probability and ethics) is in fact large.
I meant that there’s been little progress in the sense of generating theories precise enough to offer concrete recommendations, things that might be coded into an AI, e.g. formal criteria for identifying preferences, pains, and pleasures in the world (beyond pointing to existing humans and animals, which doesn’t pin down the content of utilitronium)
One could argue that until recently there has been little motivation amongst utilitarians to formulate such precise theories, so you can’t really count all of the past 60 years as evidence against this being doable in the next few decades. Some of the problems weren’t even identified until recently, and others, like how to identify pain and pleasure, could be informed by recent or ongoing science. And of course these difficulties have to be compared with the difficulties of EV. Perhaps I should just say that it’s not nearly as obvious that “hard-coding” is a bad idea, if “complexity of value” refers to the complexity of a precise formulation of utilitarianism, for example, as opposed to the complexity of “Godshatter”.
Even a little uncertainty about many dimensions means probably going wrong, and it seems that reasonable uncertainty about several of these things (e.g. infinite worlds and implications for probability and ethics) is in fact large.
Is it plausible that someone could reasonably interpret lack of applicable intuitions along some dimensions as indifference, instead of uncertainty?
In other words, it sounds like you’re saying human values are complex because all values are complex.
Yes, I’m more confident of this than of the godshatter thesis. With respect to the godshatter thesis, it seems fairly likely to me that reflective equilibrium does not wash away all our desires’ complexity, because many values are self-defending, e.g. desire for varied experiences, protection of loved ones, etc. They seem not much more arbitrary than any form of hedonism particular enough to answer all the questions above, and so likely to rise or fall together (under conditions of reflection and self-control).
Many beliefs are also Self-defending, but that is little reason to hold onto them. There is no Reason for the same principle not to apply to beliefs about morality, which values implicitly are. Thus their Self-defending-ness doesn’t necessarily mean that reflective equilibrium doesn’t just throw them out as false theories rather than updating on them as raw data, though of course they are both. Or so one perspective would argue.
In other words, it sounds like you’re saying human values are complex because all values are complex. But that doesn’t by itself prove that human values are much more complex than those of a pleasure maximizer, which is a claim that I take to be included in the “value is complex”/”thou art godshatter” idea.
ETA: Though your point may well be sufficient to support the particular implication that we’re unlikely to get the values right by hard-coding them or having them randomly emerge.
If someone thinks it’s pretty likely that their preferences (as endorsed in reflective equilibrium) can be captured by some form of utilitarianism, then their time could easily be better spent trying to narrow down the exact form of utilitarianism (with the aim of hard-coding it) instead of trying to figure out how to specify and implement something like CEV.
Utilitarians haven’t made that much progress on these issues in the last 60 years, and there are a lot of them. Do you have a further argument for why this could easily be the case?
There’s little progress in the sense of resolving disagreements, but if someone thinks what is important is not public consensus on a particular form of utilitarianism, but just what they would endorse in reflective equilibrium as an individual, and they already have fairly strong intuitions on what kind of utilitarianism is right (which some people seem to), then that seems sufficient to make “narrow down the exact form of utilitarianism” a potentially better plan than “implement CEV” from their perspective.
I meant that there’s been little progress in the sense of generating theories precise enough to offer concrete recommendations, things that might be coded into an AI, e.g. formal criteria for identifying preferences, pains, and pleasures in the world (beyond pointing to existing humans and animals, which doesn’t pin down the content of utilitronium), or a clear way to value different infinitely vast worlds (with all the rearrangement issues discussed in Bostrom’s “Infinitarian Challenge” paper). This isn’t just a matter of persistent moral disagreement, but a lack of any comprehensive candidates that actually tell you what to do in particular situations rather than having massive lacunae that are filled in by consideration of individual cases and local intuitions.
This seems to me more about the “C” than the “EV.” I think such a utilitarian should still be strongly concerned with having at least their reflective equilibrium extrapolated. Even a little uncertainty about many dimensions means probably going wrong, and it seems that reasonable uncertainty about several of these things (e.g. infinite worlds and implications for probability and ethics) is in fact large.
One could argue that until recently there has been little motivation amongst utilitarians to formulate such precise theories, so you can’t really count all of the past 60 years as evidence against this being doable in the next few decades. Some of the problems weren’t even identified until recently, and others, like how to identify pain and pleasure, could be informed by recent or ongoing science. And of course these difficulties have to be compared with the difficulties of EV. Perhaps I should just say that it’s not nearly as obvious that “hard-coding” is a bad idea, if “complexity of value” refers to the complexity of a precise formulation of utilitarianism, for example, as opposed to the complexity of “Godshatter”.
Is it plausible that someone could reasonably interpret lack of applicable intuitions along some dimensions as indifference, instead of uncertainty?
Yes, I’m more confident of this than of the godshatter thesis. With respect to the godshatter thesis, it seems fairly likely to me that reflective equilibrium does not wash away all our desires’ complexity, because many values are self-defending, e.g. desire for varied experiences, protection of loved ones, etc. They seem not much more arbitrary than any form of hedonism particular enough to answer all the questions above, and so likely to rise or fall together (under conditions of reflection and self-control).
Many beliefs are also Self-defending, but that is little reason to hold onto them. There is no Reason for the same principle not to apply to beliefs about morality, which values implicitly are. Thus their Self-defending-ness doesn’t necessarily mean that reflective equilibrium doesn’t just throw them out as false theories rather than updating on them as raw data, though of course they are both. Or so one perspective would argue.