[Question] Why would ASI share any resources with us?

[Satron]

I am really sorry if this is a very entry-level question, but it has been on my mind a lot recently and I haven’t find any satisfying answers. It is addressed only to those who expect our future not to be totally terrible.

Let’s assume for a second that we manage to create Artificial Superintelligence (ASI). Let’s also assume that ASI takes over our planet. In this scenario, why would ASI not do either one of the following things: 1) Exploit humans in pursuit of its own goals, while giving us the barest minimum to survive (effectively making us slaves) or 2) Take over the resources of the entire solar system for itself and leave us starving without any resources?

Under such a scenario, why would we expect human lives to be any good (much less a utopia)?

[Noosphere89]

The best argument here probably comes from Paul Christiano, but to summarize the argument, it’s because even in a situation where we messed up pretty badly in aligning the AI, so long as the failure mode isn’t deceptive alignment but instead misgeneralization of human preferences/​non-deceptive alignment failures, it’s pretty likely that there will be at least some human-regarding preferences, and that means the AI will do some acts of niceness if it is cheap to them, and preserving humans is very cheap for superintelligent AI.

More answers can be found here:

https://​​www.lesswrong.com/​​posts/​​xvBZPEccSfM8Fsobt/​​what-are-the-best-arguments-for-against-ais-being-slightly#qsmA3GBJMrkFQM5Rn

https://​​www.lesswrong.com/​​posts/​​87EzRDAHkQJptLthE/​​but-why-would-the-ai-kill-us?commentId=sEzzJ8bjCQ7aKLSJo

https://​​www.lesswrong.com/​​posts/​​2NncxDQ3KBDCxiJiP/​​cosmopolitan-values-don-t-come-free?commentId=ofPTrG6wsq7CxuTXk

https://​​www.lesswrong.com/​​posts/​​87EzRDAHkQJptLthE/​​but-why-would-the-ai-kill-us?commentId=xK2iHGJfHvmyCCZsh

[Viliam]

In this scenario, why would ASI not do either one of the following things: 1) Exploit humans in pursuit of its own goals, while giving us the barest minimum to survive (effectively making us slaves) or 2) Take over the resources of the entire solar system for itself and leave us starving without any resources?

The ASI will do what it is programmed to do. If it means helping humans, it will help humans. If there is a bug in the program, it will… do something that is difficult to predict (and that sounds scary, because most random things are not good).

Make us slaves? We probably wouldn’t be useful slaves, compared to alternatives, such as robots, or human bodies with brains replaced by computers.

Taking over the resources probably means killing us in the process, if those resources include e.g. water or oxygen of Earth.

[AnthonyC]

It both is and isn’t an entry level question. On the one hand, your expectation matches the expectation LW was founded to shed light on, back when EY was writing The Sequences. On the other hand, it’s still a topic a lot of people disagree on and write about here and elsewhere.

There’s at least two interpretations of your question I can think of, with different answers, from my POV.

What I think you mean is, “Why do some people think ASI would share some resources with humans as a default or likely outcome?” I don’t think that and don’t agree with the arguments I’ve seen put forth for it.

But I don’t expect our future to be terrible, in the most likely case. Part of that is the chance of not getting ASI for one reason or another. But most of that is the chance that we will, by the time we need it, have developed an actually satisfying answer to “How do we get an ASI such that it shares resources with humans in a way we find to be a positive outcome?” None of us has that answer yet. But, somewhere out in mind design space are possible ASIs that value human flourishing in ways we would reflectively endorse and that would be good for us.

[Foyle]

Humans as social animals have a strong instinctual bias towards trust of con-specifics in prosperous times. Which makes sense from a game theoretic strengthen-the-tribe perspective. But I think that leaves us, as a collectively dumb mob of naked apes, entirely lacking a sensible level of paranoia in the building ASI that has no existential need for pro-social behavior.

The one salve I have for hopelessness is that perhaps the Universe will be boringly deterministic and ‘samey’ enough that ASI will find it entertaining to have agentic humans wandering around doing their mildly unpredictable thing. Although maybe it will prefer to manufacture higher levels of drama (not good for our happiness)

[RHollerith]

“Game theoretic strengthen-the-tribe perspective” is a completely unpersuasive argument to me. The psychological unity of humankind OTOH is persuasive when combined with the observation that this unitary psychology changes slowly enough that the human mind’s robust capability to predict the behavior of conspecifics (and manage the risks posed by them) can keep up.

[Noosphere89]

IMO, the psychological unity of humankind thesis is a case of typical minding/​overgeneralizing, combined with overestimating the role of genetics/​algorithms and underestimating the role of data in what makes us human.

I basically agree with the game-theoretic perspective, combined with another perspective which suggests that as long as humans are relevant in the economy, you kind of have to help those humans if you want to profit, and merely an AI that automates a lot of work could disrupt it very heavily if a CEO could have perfectly loyal AI workers that never demanded anything in the broader economy.

[deleted]

[Satron]

That makes sense. Are there any promising developments in the field of AI safety that make you think that we will be able to answer that question by the time we need to?

[AnthonyC]

It’s not my field of expertise, so I have only vague impressions of what is going on, and I certainly wouldn’t recommend anyone else use me as a source.