A Singleton AI is not a stable equilibrium and therefore it is highly unlikely that a Singleton AI will dominate our future light cone (90%).
Superhuman intelligence will not give an AI an insurmountable advantage over collective humanity (75%).
Intelligent entities with values radically different to humans will be much more likely to engage in trade and mutual compromise than to engage in violence and aggression directed at humans (60%).
I’m curious, do you also think that a singleton is a desirable outcome? It’s possible my thinking is biased because I view this outcome as a dystopia and so underestimate it’s probability due to motivated cognition.
Funny you should mention it; that’s exactly what I was thinking. I have a friend (also named matt, incidentally) who I strongly believe is guilty of motivated cognition about the desirability of a singleton AI (he thinks it is likely, and therefore is biased toward thinking it would be good) and so I leaped naturally to the ad hominem attack you level against yourself. :-)
I’m curious, do you also think that a singleton is a desirable outcome? It’s possible my thinking is biased because I view this outcome as a dystopia and so underestimate it’s probability due to motivated cognition.
Most of them, no. Some, yes. Particularly since the alternative is the inevitable loss of everything that is valuable to me in the universe.
This is incredibly tangential, but I was talking to a friend earlier and I realized how difficult it is to instill in someone the desire for altruism. Her reasoning was basically, “Yeah… I feel like I should care about cancer, and I do care a little, but honestly, I don’t really care.” This sort of off-hand egoism is something I wasn’t used to; most smart people try to rationalize selfishness with crazy beliefs. But it’s hard to argue with “I just don’t care” other than to say “I bet you will have wanted to have cared”, which is gramatically horrible and a pretty terrible argument.
Stable equilibrium here does not refer to a property of a mind. It refers to a state of the universe. I’ve elaborated on this view a little here before but I can’t track the comment down at the moment.
Essentially my reasoning is that in order to dominate the physical universe an AI will need to deal with fundamental physical restrictions such as the speed of light. This means it will have spatially distributed sub-agents pursuing sub-goals intended to further its own goals. In some cases these sub-goals may involve conflict with other agents (this would be particularly true during the initial effort to become a singleton).
Maintaining strict control over sub-agents imposes restrictions on the design and capabilities of sub-agents which means it is likely that they will be less effective at achieving their sub-goals than sub-agents without such design restrictions. Sub-agents with significant autonomy may pursue actions that conflict with the higher level goals of the singleton.
Human (and biological) history is full of examples of this essential conflict. In military scenarios for example there is a tradeoff between tight centralized control and combat effectiveness—units that have a degree of authority to take decisions in the field without the delays or overhead imposed by communication times are generally more effective than those with very limited freedom to act without direct orders.
Essentially I don’t think a singleton AI can get away from the principal-agent problem. Variations on this essential conflict exist throughout the human and natural worlds and appear to me to be fundamental consequences of the nature of our universe.
Ant colonies don’t generally exhibit the principal-agent problem. I’d say with high certainty that the vast majority of our trouble with it is due to having the selfishness of an individual replicator hammered into each of us by our evolution.
I’m not a biologist, but given that animal bodies exhibit principal-agent problems, e.g., auto-immune diseases and cancers, I suspect ant colonies (and large AI’s) would also have these problems.
Cancer is a case where an engineered genome could improve over an evolved one. We’ve managed to write software (for the most vital systems) that can copy without error, with such high probability that we expect never to see that part malfunction.
One reason that evolution hasn’t constructed sufficiently good error correction is that the most obvious way to do this makes the genome totally incapable of new mutations, which works great until the niche changes.
However, an AI-subagent would need to be able to adjust itself to unexpected conditions, and thus can’t simply rely on digital copying to prevent malfunctions.
So you agree that it’s possible in principle for a singleton AI to remain a singleton (provided it starts out alone in the cosmos), but you believe it would sacrifice significant adaptability and efficiency by doing so. Perhaps; I don’t know either way.
But the AI might make that sacrifice if it concludes that (eventually) losing singleton status would cost its values far more than the sacrifice is worth (e.g. if losing singleton status consigns the universe to a Hansonian hardscrapple race to burn the cosmic commons(pdf) rather than a continued time of plenty).
I believe it would at the very least have to sacrifice at least all adaptability by doing so, as in only sending out nodes with all instructions in ROM and instructions to periodically rest all non-ROM memory and shelf-destruct if it notices any failures of its triple redundancy ROM. As well as an extremely strong directive against anything that would let nodes store long term state.
I don’t know whether ant colonies exhibit principal-agent problems (though I’d expect that they do to some degree) but I know there is evidence of nepotism in queen rearing in bee colonies where individuals are not all genetically identical (evidence of workers favouring the most closely related larvae when selecting larvae to feed royal jelly to create a queen).
The fact that ants from different colonies commonly exhibit aggression towards each other indicates limits to scaling such high levels of group cohesion. Though supercolonies do appear to exist they have not come to total dominance.
The largest and most complex examples of group coordination we know of are large human organizations and these show much greater levels of internal goal conflicts than much simpler and more spatially concentrated insect colonies.
A Singleton AI is not a stable equilibrium and therefore it is highly unlikely that a Singleton AI will dominate our future light cone (90%).
Superhuman intelligence will not give an AI an insurmountable advantage over collective humanity (75%).
Intelligent entities with values radically different to humans will be much more likely to engage in trade and mutual compromise than to engage in violence and aggression directed at humans (60%).
I want to upvote each of these points a dozen times. Then another few for the first.
It’s the most stable equilibrium I can conceive of. ie. More stable than if all evidence of life was obliterated from the universe.
I guess I’m playing the game right then :)
I’m curious, do you also think that a singleton is a desirable outcome? It’s possible my thinking is biased because I view this outcome as a dystopia and so underestimate it’s probability due to motivated cognition.
Funny you should mention it; that’s exactly what I was thinking. I have a friend (also named matt, incidentally) who I strongly believe is guilty of motivated cognition about the desirability of a singleton AI (he thinks it is likely, and therefore is biased toward thinking it would be good) and so I leaped naturally to the ad hominem attack you level against yourself. :-)
Most of them, no. Some, yes. Particularly since the alternative is the inevitable loss of everything that is valuable to me in the universe.
This is incredibly tangential, but I was talking to a friend earlier and I realized how difficult it is to instill in someone the desire for altruism. Her reasoning was basically, “Yeah… I feel like I should care about cancer, and I do care a little, but honestly, I don’t really care.” This sort of off-hand egoism is something I wasn’t used to; most smart people try to rationalize selfishness with crazy beliefs. But it’s hard to argue with “I just don’t care” other than to say “I bet you will have wanted to have cared”, which is gramatically horrible and a pretty terrible argument.
I respect blatant apathy a whole hell of a lot more than masked apathy, which is how I would qualify the average person’s altruism.
I agree with your second. Was your third supposed to be high or low? I think it’s low, but not unreasonably so.
I expected the third to be higher than most less wrongers would estimate.
I’m almost certainly missing some essential literature, but what does it mean for a mind to be a stable equilibrium?
I’m almost certainly missing some essential literature, but what does it mean for a mind to be a stable equilibrium?
Stable equilibrium here does not refer to a property of a mind. It refers to a state of the universe. I’ve elaborated on this view a little here before but I can’t track the comment down at the moment.
Essentially my reasoning is that in order to dominate the physical universe an AI will need to deal with fundamental physical restrictions such as the speed of light. This means it will have spatially distributed sub-agents pursuing sub-goals intended to further its own goals. In some cases these sub-goals may involve conflict with other agents (this would be particularly true during the initial effort to become a singleton).
Maintaining strict control over sub-agents imposes restrictions on the design and capabilities of sub-agents which means it is likely that they will be less effective at achieving their sub-goals than sub-agents without such design restrictions. Sub-agents with significant autonomy may pursue actions that conflict with the higher level goals of the singleton.
Human (and biological) history is full of examples of this essential conflict. In military scenarios for example there is a tradeoff between tight centralized control and combat effectiveness—units that have a degree of authority to take decisions in the field without the delays or overhead imposed by communication times are generally more effective than those with very limited freedom to act without direct orders.
Essentially I don’t think a singleton AI can get away from the principal-agent problem. Variations on this essential conflict exist throughout the human and natural worlds and appear to me to be fundamental consequences of the nature of our universe.
Ant colonies don’t generally exhibit the principal-agent problem. I’d say with high certainty that the vast majority of our trouble with it is due to having the selfishness of an individual replicator hammered into each of us by our evolution.
I’m not a biologist, but given that animal bodies exhibit principal-agent problems, e.g., auto-immune diseases and cancers, I suspect ant colonies (and large AI’s) would also have these problems.
Cancer is a case where an engineered genome could improve over an evolved one. We’ve managed to write software (for the most vital systems) that can copy without error, with such high probability that we expect never to see that part malfunction.
One reason that evolution hasn’t constructed sufficiently good error correction is that the most obvious way to do this makes the genome totally incapable of new mutations, which works great until the niche changes.
However, an AI-subagent would need to be able to adjust itself to unexpected conditions, and thus can’t simply rely on digital copying to prevent malfunctions.
So you agree that it’s possible in principle for a singleton AI to remain a singleton (provided it starts out alone in the cosmos), but you believe it would sacrifice significant adaptability and efficiency by doing so. Perhaps; I don’t know either way.
But the AI might make that sacrifice if it concludes that (eventually) losing singleton status would cost its values far more than the sacrifice is worth (e.g. if losing singleton status consigns the universe to a Hansonian hardscrapple race to burn the cosmic commons(pdf) rather than a continued time of plenty).
I believe it would at the very least have to sacrifice at least all adaptability by doing so, as in only sending out nodes with all instructions in ROM and instructions to periodically rest all non-ROM memory and shelf-destruct if it notices any failures of its triple redundancy ROM. As well as an extremely strong directive against anything that would let nodes store long term state.
Remember, you’re the one trying to prove impossibility of a task here. Your inability to imagine a solution to the problem is only very weak evidence.
I don’t know whether ant colonies exhibit principal-agent problems (though I’d expect that they do to some degree) but I know there is evidence of nepotism in queen rearing in bee colonies where individuals are not all genetically identical (evidence of workers favouring the most closely related larvae when selecting larvae to feed royal jelly to create a queen).
The fact that ants from different colonies commonly exhibit aggression towards each other indicates limits to scaling such high levels of group cohesion. Though supercolonies do appear to exist they have not come to total dominance.
The largest and most complex examples of group coordination we know of are large human organizations and these show much greater levels of internal goal conflicts than much simpler and more spatially concentrated insect colonies.
I’m analogizing a singleton to a single ant colony, not to a supercolony.
I agree with your first two, but am dubious about your third.
Two points that influence my thinking on that claim:
Gains from trade have the potential to be greater with greater difference in values between the two trading agents.
Destruction tends to be cheaper than creation. Intelligent agents that recognize this have an incentive to avoid violent conflict.