Just to clarify (perhaps unnecessarily): by an attractor I mean a moral framework from which you wouldn’t want to self-modify radically in any direction. There do exist many distinct attractors in the space of ‘abstracted idealized dynamics’, as Eliezer notes for the unfortunate Pebblesorters: they might modify their subgoals, but never approach a morality indifferent to the cardinality of pebble heaps.
Eliezer’s claim of moral convergence and the CEV, as I understand it, is that most humans are psychologically constituted so that our moral frameworks lie in the ‘basin’ of a single attractor; thus the incremental self-modifications of cultural history have an ultimate destination which a powerful AI could deduce.
I suspect, however, that the position is more chaotic than this; that there are distinct avenues of moral progress which will lead us to different attractors. In your terms, since our current right is after all not entirely comprehensive and consistent, we could find that both right1 and right2 are both right extrapolations from right, and that right can’t judge unequivocally which one is better.
Vladimir,
Just to clarify (perhaps unnecessarily): by an attractor I mean a moral framework from which you wouldn’t want to self-modify radically in any direction. There do exist many distinct attractors in the space of ‘abstracted idealized dynamics’, as Eliezer notes for the unfortunate Pebblesorters: they might modify their subgoals, but never approach a morality indifferent to the cardinality of pebble heaps.
Eliezer’s claim of moral convergence and the CEV, as I understand it, is that most humans are psychologically constituted so that our moral frameworks lie in the ‘basin’ of a single attractor; thus the incremental self-modifications of cultural history have an ultimate destination which a powerful AI could deduce.
I suspect, however, that the position is more chaotic than this; that there are distinct avenues of moral progress which will lead us to different attractors. In your terms, since our current right is after all not entirely comprehensive and consistent, we could find that both right1 and right2 are both right extrapolations from right, and that right can’t judge unequivocally which one is better.