It has been a few years since I read that sequence in full, but my impression was that Eliezer thought there were some basic pieces that human morality is made out of, and some common ways of finding/putting those pieces together, though they needn’t be exactly the same. If you run this process for long enough, using the procedures humans use to construct their values, then you’d end up in some relatively small space compared to the space of all possible goals, or all evolutionarily fit goals for superintelligences etc.
So too for a psychopath. This seems plausible to me. I don’t expect a psychopath to wind up optimizing for paper-clips on reflection. But I also don’t expect to be happy in a world that ranks very high accordting to a psychopath’s values-on-reflection. Plausibly, I wouldn’t even exist in such a world.
It has been a few years since I read that sequence in full, but my impression was that Eliezer thought there were some basic pieces that human morality is made out of, and some common ways of finding/putting those pieces together, though they needn’t be exactly the same. If you run this process for long enough, using the procedures humans use to construct their values, then you’d end up in some relatively small space compared to the space of all possible goals, or all evolutionarily fit goals for superintelligences etc.
So too for a psychopath. This seems plausible to me. I don’t expect a psychopath to wind up optimizing for paper-clips on reflection. But I also don’t expect to be happy in a world that ranks very high accordting to a psychopath’s values-on-reflection. Plausibly, I wouldn’t even exist in such a world.