i share this sentiment to an extent, though i’m usually more concerned with “partial but botched alignment”. see 1, 2.
that said, i agree many people want very bad things, but i’m somewhat hopeful that the kind of person who is likely to end up being who the AI is aligned to would be somewhat reasonable and cosmopolitan and respect the values of other moral patients, especially under CEV.
but that’s a very flimsy/hopeful argument.
a better argument would be that CEV is more of a decision process than “a continuously-existing person in control, in the usual sense” — i would think the CEV would bootstrap to a better, aligned and cosmopolitan decision process. even if there was a person out there to whom i would genuinely wish nontrivial suffering, which i don’t believe there is, i think my CEV would be less likely preoccupied with that and moreso concerned with coming up with general principles that make everything okay.
but, it’s good to see other people come to this reasoning. incidentally, i’d be up for having more thorough conversations about this.
i share this sentiment to an extent, though i’m usually more concerned with “partial but botched alignment”. see 1, 2.
that said, i agree many people want very bad things, but i’m somewhat hopeful that the kind of person who is likely to end up being who the AI is aligned to would be somewhat reasonable and cosmopolitan and respect the values of other moral patients, especially under CEV.
but that’s a very flimsy/hopeful argument.
a better argument would be that CEV is more of a decision process than “a continuously-existing person in control, in the usual sense” — i would think the CEV would bootstrap to a better, aligned and cosmopolitan decision process. even if there was a person out there to whom i would genuinely wish nontrivial suffering, which i don’t believe there is, i think my CEV would be less likely preoccupied with that and moreso concerned with coming up with general principles that make everything okay.
but, it’s good to see other people come to this reasoning. incidentally, i’d be up for having more thorough conversations about this.