The FOOM—singleton theory intrinsically assumes egoistic AIs.
No, that’s wrong. The speed of takeoff is largely a technical question; from a strategic planning POV, going through a rapid takeoff likely makes sense regardless of what your goals are (unless your friendliness design is incomplete /and/ you have corrigibility aspects; but that’s a very special case).
As for what you do once you’re done, that does indeed depend on your goals; but forming a singleton doesn’t imply egoism or egocentrism of any kind. Your goals can still be entirely focused on other entities in society; it’s just that if have certain invariants you want to enforce on them (could be anything, really; things like “no murder”, “no extensive torture”, “no destroying society” would be unoffensive and relevant examples) - or indeed, more generally, certain aspects to optimize for—it helps a lot if you can stay in ultimate control to do these things.
As Bostrom explains in his footnotes, there are many kinds of singletons. In general, it simply refers to an entity that has attained and keeps ultimate power in society. How much or how little it uses that power to control any part of the world is independent of that, and some singletons would interfere little with the rest of society.
Your argumentation based on the orthogonality principle is clear to me. But even if the utility function includes human values (fostering humankind, preserving a sustainable habitat on earth for humans, protecting humans against unfriendly AI developments, solving the control problem) strong egoistic traits are needed to remain superior to other upcoming AIs. Ben Goertzel coined the term “global AI Nanny” for a similar concept.
How would we get notion of existence of a little interfering FAI singleton?
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
How would we get notion of existence of a little interfering FAI singleton?
The AI’s values would likely have to be specifically chosen to get this outcome; something like “let human development continue normally, except for blocking existential catastrophes”. Something like that won’t impact what you’re trying to do, unless that involves destroying society or something equally problematic.
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
Above hypothetical singleton AI would end up either sabotaging the project, or containing the resulting AI. It wouldn’t have to stop the UFAI before release, necessarily; with enough of a hardware headstart, later safe containment can be guaranteed fine. Either way, the intervention needn’t involve attacking humans; interfering with just the AI’s hardware can accomplish the same result. And certainly the development project shouldn’t get much chance to fight back; terms like “interdiction”, “containment”, “sabotage”, and maybe “police action” (though that one has unfortunate anthropomorphic connotations) are a better fit than “war”.
No, that’s wrong. The speed of takeoff is largely a technical question; from a strategic planning POV, going through a rapid takeoff likely makes sense regardless of what your goals are (unless your friendliness design is incomplete /and/ you have corrigibility aspects; but that’s a very special case).
As for what you do once you’re done, that does indeed depend on your goals; but forming a singleton doesn’t imply egoism or egocentrism of any kind. Your goals can still be entirely focused on other entities in society; it’s just that if have certain invariants you want to enforce on them (could be anything, really; things like “no murder”, “no extensive torture”, “no destroying society” would be unoffensive and relevant examples) - or indeed, more generally, certain aspects to optimize for—it helps a lot if you can stay in ultimate control to do these things.
As Bostrom explains in his footnotes, there are many kinds of singletons. In general, it simply refers to an entity that has attained and keeps ultimate power in society. How much or how little it uses that power to control any part of the world is independent of that, and some singletons would interfere little with the rest of society.
Your argumentation based on the orthogonality principle is clear to me. But even if the utility function includes human values (fostering humankind, preserving a sustainable habitat on earth for humans, protecting humans against unfriendly AI developments, solving the control problem) strong egoistic traits are needed to remain superior to other upcoming AIs. Ben Goertzel coined the term “global AI Nanny” for a similar concept.
How would we get notion of existence of a little interfering FAI singleton?
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
The AI’s values would likely have to be specifically chosen to get this outcome; something like “let human development continue normally, except for blocking existential catastrophes”. Something like that won’t impact what you’re trying to do, unless that involves destroying society or something equally problematic.
Above hypothetical singleton AI would end up either sabotaging the project, or containing the resulting AI. It wouldn’t have to stop the UFAI before release, necessarily; with enough of a hardware headstart, later safe containment can be guaranteed fine. Either way, the intervention needn’t involve attacking humans; interfering with just the AI’s hardware can accomplish the same result. And certainly the development project shouldn’t get much chance to fight back; terms like “interdiction”, “containment”, “sabotage”, and maybe “police action” (though that one has unfortunate anthropomorphic connotations) are a better fit than “war”.