I’m moderately sure what my values are, to some approximation. More importantly, I’m even more sure that, whatever my values are, they are not so extremely different from the values of most people that I should wage some kind of war against the majority instead of trying to arrive at a reasonable compromise. And, in the unlikely event that most people (including me) will turn out to be some kind of utilitarians after all, it’s not a problem: value aggregation will then produce a universe which is pretty good for utilitarians.
I’m moderately sure what my values are, to some approximation. More importantly, I’m even more sure that, whatever my values are, they are not so extremely different from the values of most people [...]
Maybe you’re just not part of the target audience of my OP then… but from my perspective, if I determine my values through the kind of process described in the first quote, and most people determine their values through the kind of process described in the second quote, it seems quite likely that the values end up being very different.
[...] that I should wage some kind of war against the majority instead of trying to arrive at a reasonable compromise.
The kind of solution I have in mind is not “waging war” but for example, solving metaphilososphy and building an AI that can encourage philosophical reflection in humans or enhance people’s philosophical abilities.
And, in the unlikely possibility that most people (including me) will turn out to be some kind of utilitarians after all, it’s not a problem: value aggregation will then produce a universe which is pretty good for utilitarians.
What if you turn out to be some kind of utilitarian but most people don’t (because you’re more like the first group in the OP and they’re more like the second group), or most people will eventually turn out to be some kind of utilitarian in a world without AI, but in a world with AI, this will happen?
I don’t think people determine their values through either process. I think that they already have values, which are to a large extent genetic and immutable. Instead, these processes determine what values they pretend to have for game-theory reasons. So, the big difference between the groups is which “cards” they hold and/or what strategy they pursue, not an intrinsic difference in values.
But also, if we do model values as the result of some long process of reflection, and you’re worried about the AI disrupting or insufficiently aiding this process, then this is already a single-user alignment issue and should be analyzed in that context first. The presumed differences in moralities are not the main source of the problem here.
I don’t think people determine their values through either process. I think that they already have values, which are to a large extent genetic and immutable. Instead, these processes determine what values they pretend to have for game-theory reasons. So, the big difference between the groups is which “cards” they hold and/or what strategy they pursue, not an intrinsic difference in values.
This is not a theory that’s familiar to me. Why do you think this is true? Have you written more about it somewhere or can link to a more complete explanation?
But also, if we do model values as the result of some long process of reflection, and you’re worried about the AI disrupting or insufficiently aiding this process, then this is already a single-user alignment issue and should be analyzed in that context first. The presumed differences in moralities are not the main source of the problem here.
This seems reasonable to me. (If this was meant to be an argument against something I said, there may have been anther miscommuncation, but I’m not sure it’s worth tracking that down.)
This is not a theory that’s familiar to me. Why do you think this is true? Have you written more about it somewhere or can link to a more complete explanation?
I considering writing about this for a while, but so far I don’t feel sufficiently motivated. So, the links I posted upwards in the thread are the best I have, plus vague gesturing in the directions of Hansonian signaling theories, Jaynes’ theory of consciousness and Yudkowsky’s belief in belief.
I’m moderately sure what my values are, to some approximation. More importantly, I’m even more sure that, whatever my values are, they are not so extremely different from the values of most people that I should wage some kind of war against the majority instead of trying to arrive at a reasonable compromise. And, in the unlikely event that most people (including me) will turn out to be some kind of utilitarians after all, it’s not a problem: value aggregation will then produce a universe which is pretty good for utilitarians.
Maybe you’re just not part of the target audience of my OP then… but from my perspective, if I determine my values through the kind of process described in the first quote, and most people determine their values through the kind of process described in the second quote, it seems quite likely that the values end up being very different.
The kind of solution I have in mind is not “waging war” but for example, solving metaphilososphy and building an AI that can encourage philosophical reflection in humans or enhance people’s philosophical abilities.
What if you turn out to be some kind of utilitarian but most people don’t (because you’re more like the first group in the OP and they’re more like the second group), or most people will eventually turn out to be some kind of utilitarian in a world without AI, but in a world with AI, this will happen?
I don’t think people determine their values through either process. I think that they already have values, which are to a large extent genetic and immutable. Instead, these processes determine what values they pretend to have for game-theory reasons. So, the big difference between the groups is which “cards” they hold and/or what strategy they pursue, not an intrinsic difference in values.
But also, if we do model values as the result of some long process of reflection, and you’re worried about the AI disrupting or insufficiently aiding this process, then this is already a single-user alignment issue and should be analyzed in that context first. The presumed differences in moralities are not the main source of the problem here.
This is not a theory that’s familiar to me. Why do you think this is true? Have you written more about it somewhere or can link to a more complete explanation?
This seems reasonable to me. (If this was meant to be an argument against something I said, there may have been anther miscommuncation, but I’m not sure it’s worth tracking that down.)
I considering writing about this for a while, but so far I don’t feel sufficiently motivated. So, the links I posted upwards in the thread are the best I have, plus vague gesturing in the directions of Hansonian signaling theories, Jaynes’ theory of consciousness and Yudkowsky’s belief in belief.
Isn’t this the main thesis of “The righteous mind”?