Diametral opposing to theft and war szenarios you discuss in
your paper
“Rational Altruist—Why might the future be good?”:
How much altruism do we expect?
[...] my median expectation is that the future is much more altruistic than the present.
I fully agree with you and this aspect is lacking in Bostrums book. The FOOM—singleton theory intrinsically assumes
egoistic AIs.
Altruism is for me one of the core ingredience towards sustainably incorporating friendly AIs into society. I support
your view that the future will be more altruistic than the present: AIs will have more memory to remember
behavior of their contacts. The Dunbar’s number of social contacts will be higher. Social contacts recognize altruistic behavior and remember
this good deed for the future. The wider the social net the higher is the reward for altruistic behavior.
The result shows that, as predicted, even when controlling for a range of individual and relationship factors,
the network factor (number of connections) makes a significant contribution to altruism, thus showing that individuals are more likely to be altruistic to better-connected members of their social networks.
The idea of AIs and humans monitoring AIs in a constitutional society is not new. Stephen Omohundro presented it in October 2007 at the Stanford EE380
Computer Systems Colloquium on “Self-Improving Artificial Intelligence and the Future of Computing”.
Q: What about malicious mutations [of the utility function]?
Stephen Omohundro:
Dealing with malicious things is very important.
There is an organization—Eliezer is here in the back—he called it the Singularity Institute
for Artificial Intelligence, which is trying to ensure that the consequences of these kinds of
systems are immune to malicious agents and to accidental unintended consequences. And it is one of the great
challenges right now because if you assume that this kind of system is possible and has the kinds
of powers we are talking about, it can be useable for great good but also for bad purposes.
And so finding a structure which is stable—and I think I agree with Eric [Baum?]-
that the ultimate kind of solution that makes more
sense to me is essentially have a large ecology of intelligent agents and humans. Such that in a
kind of a constitution that everybody follows:
Humans probably will not be able
to monitor AIs, because they are thinking faster and more powerfully,
but AIs could monitor AIs.
So we set up a structure so that each entity wants to obey the “law”, wants to
follow the constitution, wants to respect all the various rights that we would to decide on.
And if somebody starts violating the law that they have an interest in stopping them from doing that.
The hope is that we can create basically a stable future of society with these kinds of entities.
The thinking of this is just beginning on that. I think a lot of input is needed from economists,
is needed from psychologists, [...] and sociologists [...] as well as computer systems engineers. I mean we really need
input from a wide variety of vizpoints.
The FOOM—singleton theory intrinsically assumes egoistic AIs.
No, that’s wrong. The speed of takeoff is largely a technical question; from a strategic planning POV, going through a rapid takeoff likely makes sense regardless of what your goals are (unless your friendliness design is incomplete /and/ you have corrigibility aspects; but that’s a very special case).
As for what you do once you’re done, that does indeed depend on your goals; but forming a singleton doesn’t imply egoism or egocentrism of any kind. Your goals can still be entirely focused on other entities in society; it’s just that if have certain invariants you want to enforce on them (could be anything, really; things like “no murder”, “no extensive torture”, “no destroying society” would be unoffensive and relevant examples) - or indeed, more generally, certain aspects to optimize for—it helps a lot if you can stay in ultimate control to do these things.
As Bostrom explains in his footnotes, there are many kinds of singletons. In general, it simply refers to an entity that has attained and keeps ultimate power in society. How much or how little it uses that power to control any part of the world is independent of that, and some singletons would interfere little with the rest of society.
Your argumentation based on the orthogonality principle is clear to me. But even if the utility function includes human values (fostering humankind, preserving a sustainable habitat on earth for humans, protecting humans against unfriendly AI developments, solving the control problem) strong egoistic traits are needed to remain superior to other upcoming AIs. Ben Goertzel coined the term “global AI Nanny” for a similar concept.
How would we get notion of existence of a little interfering FAI singleton?
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
How would we get notion of existence of a little interfering FAI singleton?
The AI’s values would likely have to be specifically chosen to get this outcome; something like “let human development continue normally, except for blocking existential catastrophes”. Something like that won’t impact what you’re trying to do, unless that involves destroying society or something equally problematic.
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
Above hypothetical singleton AI would end up either sabotaging the project, or containing the resulting AI. It wouldn’t have to stop the UFAI before release, necessarily; with enough of a hardware headstart, later safe containment can be guaranteed fine. Either way, the intervention needn’t involve attacking humans; interfering with just the AI’s hardware can accomplish the same result. And certainly the development project shouldn’t get much chance to fight back; terms like “interdiction”, “containment”, “sabotage”, and maybe “police action” (though that one has unfortunate anthropomorphic connotations) are a better fit than “war”.
Diametral opposing to theft and war szenarios you discuss in your paper “Rational Altruist—Why might the future be good?”:
I fully agree with you and this aspect is lacking in Bostrums book. The FOOM—singleton theory intrinsically assumes egoistic AIs.
Altruism is for me one of the core ingredience towards sustainably incorporating friendly AIs into society. I support your view that the future will be more altruistic than the present: AIs will have more memory to remember behavior of their contacts. The Dunbar’s number of social contacts will be higher. Social contacts recognize altruistic behavior and remember this good deed for the future. The wider the social net the higher is the reward for altruistic behavior.
Recent research confirms this perspective: Curry, O., & Dunbar, R. I. M. (2011). Altruism in networks: the effect of connections. Biology Letters, 7(5), 651-653:
The idea of AIs and humans monitoring AIs in a constitutional society is not new. Stephen Omohundro presented it in October 2007 at the Stanford EE380 Computer Systems Colloquium on “Self-Improving Artificial Intelligence and the Future of Computing”.
I transcribed part of the Q&A of his talk (starting 51:43)
Stephen Omohundro:
No, that’s wrong. The speed of takeoff is largely a technical question; from a strategic planning POV, going through a rapid takeoff likely makes sense regardless of what your goals are (unless your friendliness design is incomplete /and/ you have corrigibility aspects; but that’s a very special case).
As for what you do once you’re done, that does indeed depend on your goals; but forming a singleton doesn’t imply egoism or egocentrism of any kind. Your goals can still be entirely focused on other entities in society; it’s just that if have certain invariants you want to enforce on them (could be anything, really; things like “no murder”, “no extensive torture”, “no destroying society” would be unoffensive and relevant examples) - or indeed, more generally, certain aspects to optimize for—it helps a lot if you can stay in ultimate control to do these things.
As Bostrom explains in his footnotes, there are many kinds of singletons. In general, it simply refers to an entity that has attained and keeps ultimate power in society. How much or how little it uses that power to control any part of the world is independent of that, and some singletons would interfere little with the rest of society.
Your argumentation based on the orthogonality principle is clear to me. But even if the utility function includes human values (fostering humankind, preserving a sustainable habitat on earth for humans, protecting humans against unfriendly AI developments, solving the control problem) strong egoistic traits are needed to remain superior to other upcoming AIs. Ben Goertzel coined the term “global AI Nanny” for a similar concept.
How would we get notion of existence of a little interfering FAI singleton?
Do we accept that this FAI wages military war against a sandboxed secret unfriendly AI development project?
The AI’s values would likely have to be specifically chosen to get this outcome; something like “let human development continue normally, except for blocking existential catastrophes”. Something like that won’t impact what you’re trying to do, unless that involves destroying society or something equally problematic.
Above hypothetical singleton AI would end up either sabotaging the project, or containing the resulting AI. It wouldn’t have to stop the UFAI before release, necessarily; with enough of a hardware headstart, later safe containment can be guaranteed fine. Either way, the intervention needn’t involve attacking humans; interfering with just the AI’s hardware can accomplish the same result. And certainly the development project shouldn’t get much chance to fight back; terms like “interdiction”, “containment”, “sabotage”, and maybe “police action” (though that one has unfortunate anthropomorphic connotations) are a better fit than “war”.