Your argumentation based on the orthogonality principle is clear to me. But even if the utility function includes human values (fostering humankind, preserving a sustainable habitat on earth for humans, protecting humans against unfriendly AI developments, solving the control problem) strong egoistic traits are needed to remain superior to other upcoming AIs. Ben Goertzel coined the term “global AI Nanny” for a similar concept.
How would we get notion of existence of a little interfering FAI singleton?
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
How would we get notion of existence of a little interfering FAI singleton?
The AI’s values would likely have to be specifically chosen to get this outcome; something like “let human development continue normally, except for blocking existential catastrophes”. Something like that won’t impact what you’re trying to do, unless that involves destroying society or something equally problematic.
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
Above hypothetical singleton AI would end up either sabotaging the project, or containing the resulting AI. It wouldn’t have to stop the UFAI before release, necessarily; with enough of a hardware headstart, later safe containment can be guaranteed fine. Either way, the intervention needn’t involve attacking humans; interfering with just the AI’s hardware can accomplish the same result. And certainly the development project shouldn’t get much chance to fight back; terms like “interdiction”, “containment”, “sabotage”, and maybe “police action” (though that one has unfortunate anthropomorphic connotations) are a better fit than “war”.
Your argumentation based on the orthogonality principle is clear to me. But even if the utility function includes human values (fostering humankind, preserving a sustainable habitat on earth for humans, protecting humans against unfriendly AI developments, solving the control problem) strong egoistic traits are needed to remain superior to other upcoming AIs. Ben Goertzel coined the term “global AI Nanny” for a similar concept.
How would we get notion of existence of a little interfering FAI singleton?
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
The AI’s values would likely have to be specifically chosen to get this outcome; something like “let human development continue normally, except for blocking existential catastrophes”. Something like that won’t impact what you’re trying to do, unless that involves destroying society or something equally problematic.
Above hypothetical singleton AI would end up either sabotaging the project, or containing the resulting AI. It wouldn’t have to stop the UFAI before release, necessarily; with enough of a hardware headstart, later safe containment can be guaranteed fine. Either way, the intervention needn’t involve attacking humans; interfering with just the AI’s hardware can accomplish the same result. And certainly the development project shouldn’t get much chance to fight back; terms like “interdiction”, “containment”, “sabotage”, and maybe “police action” (though that one has unfortunate anthropomorphic connotations) are a better fit than “war”.