@Eliezer: “I’m sure that Archimedes of Syracuse thought that Syracuse had lots of incredibly important philosophical and cultural differences with the Romans who were attacking his city. Had it fallen to Archimedes to build an AI, he might well have been tempted to believe that the whole fate of humanity would depend on whether the extrapolated volition of Syracuse or of Rome came to rule the world—due to all those incredibly important philosophical differences.”
- I don’t see how adding the romans to the CEV algorithm would have made much of a difference either way. If Archimedes had chosen himself and a few thousand people who he deemed to be “wise”, I suspect things would have gone somewhat better than if he had decided, on grounds of “non-jerkness”, to extrapolate the volition of the whole of humanity at that time, including all the barbarians, hunter gatherers, cannibal pygmies, etc.
Why can we expect this to generalize? Well, people who design AIs self-select for intelligence. People who design AIs with friendliness in mind self-select for intelligence, caution etc.
@Eliezer: “I’m sure that Archimedes of Syracuse thought that Syracuse had lots of incredibly important philosophical and cultural differences with the Romans who were attacking his city. Had it fallen to Archimedes to build an AI, he might well have been tempted to believe that the whole fate of humanity would depend on whether the extrapolated volition of Syracuse or of Rome came to rule the world—due to all those incredibly important philosophical differences.”
- I don’t see how adding the romans to the CEV algorithm would have made much of a difference either way. If Archimedes had chosen himself and a few thousand people who he deemed to be “wise”, I suspect things would have gone somewhat better than if he had decided, on grounds of “non-jerkness”, to extrapolate the volition of the whole of humanity at that time, including all the barbarians, hunter gatherers, cannibal pygmies, etc.
Why can we expect this to generalize? Well, people who design AIs self-select for intelligence. People who design AIs with friendliness in mind self-select for intelligence, caution etc.