Someone said they dislike bullet point lists. Here is the same piece formatted as paragraphs. Do you prefer it? (in which case I will edit and change it)
Carlsmith tackles two linked questions:
How should we behave towards future beings (future humans, AIs etc)?
What should our priors be about how AIs will behave towards us?
Let’s consider the first question—what should our poise be towards the future? Perhaps we are worried about the future being less valuable than it could be or that humans will be killed or outcompeted by AI.
The blog posts contain a range of answers to this question, broadly categorised as follows:
We could seize power over the future (see here and here)
We could adopt a different poise which centred around notions like growth/harmony/ “attunement” (here and here)
Let’s look at each. Generally each point links to a blog or pair of blog posts.
Trusting in basic goodness, eg god or the universe. I might think God holds the future in His hands, or more broadly that things tend to be okay. Carlsmith considers a distrust of this as a feature of Yudkowskianism, which he labels “deep atheism”. Not merely not trusting in God, but not trusting that things will be ‘okay’ unless we make them so. For similar reasons, Yudkowskians don’t assume AIs will be good. AIs will be good. For them this isn’t a good answer.
However, without the above, most justifications for seizing control of the future look like those of the AIs. I too would be trying to gain the most resources for my aims at the cost of others, regardless of their ethical stances. In this sense, the AIs aren’t bad because they foom, they are bad because they are.. not us. However this looks worrying like a justification that Stalin or the paperclippers could use. See here and here.
There are therefore a number of ways of dealing with the future, with a number of flaws.
But there is a second parallel discussion, about how might AIs treat us. Perhaps because our imagination of our future selves informs how we imagine AIs.
For instance if we assume that we cannot trust things (not 1,3) then it’s very easy to see AIs as a tool or competitor. Either it is more powerful than us or we are more powerful than it.
However, if there is a meaningful position on (5). There may be other ways to relate to AIs and future people. Here we might not control them and they might not control us. We might relates as gentle aliens (eg octopus), as dead-but-not-supreme nature (like a bear). Or something even more other than that. Something we cannot imagine but should attempt to.
Note that this doesn’t mean we shouldn’t fear AIs, they might still be capable of ruining the future, but this poise feels different.
Someone said they dislike bullet point lists. Here is the same piece formatted as paragraphs. Do you prefer it? (in which case I will edit and change it)
Carlsmith tackles two linked questions:
How should we behave towards future beings (future humans, AIs etc)?
What should our priors be about how AIs will behave towards us?
Let’s consider the first question—what should our poise be towards the future? Perhaps we are worried about the future being less valuable than it could be or that humans will be killed or outcompeted by AI.
The blog posts contain a range of answers to this question, broadly categorised as follows:
We could accept that all future beings will be alien to us and stop worrying (see here and here)
We rely on moral systems or concepts of goodness/niceness
We could seize power over the future (see here and here)
We could adopt a different poise which centred around notions like growth/harmony/ “attunement” (here and here)
Let’s look at each. Generally each point links to a blog or pair of blog posts.
Trusting in basic goodness, eg god or the universe. I might think God holds the future in His hands, or more broadly that things tend to be okay. Carlsmith considers a distrust of this as a feature of Yudkowskianism, which he labels “deep atheism”. Not merely not trusting in God, but not trusting that things will be ‘okay’ unless we make them so. For similar reasons, Yudkowskians don’t assume AIs will be good. AIs will be good. For them this isn’t a good answer.
Next, I might decide this isn’t fixable. Hanson argues future people of any stripe might be as deeply alien to us as we are to, say the Ancient Greeks. He doesn’t expect a future we consider good to be possible or, likely, desirable. Carlsmith notes that Yudkowskians don’t hold this view and muses why. Are they avoiding how alien future people will be? Do they have a clear notion of good that’s robust over time? ( Yudkowsky doensn’t seem to think so). Or are they avoiding thinking about something uncomfortable?
Many answers seem to rely on moral systems, but these present their own problems. Moral systems vary wildly at edge cases meaning that at the scale of the future, many beliefs systems would advocate for seizing control against others. We fear paperclipping partially because it is involuntary and aesthetically dull. But the arguments also extend to law abiding and even relatively joyful beings taking increasing control of the future via legal and positive sum means.
However, without the above, most justifications for seizing control of the future look like those of the AIs. I too would be trying to gain the most resources for my aims at the cost of others, regardless of their ethical stances. In this sense, the AIs aren’t bad because they foom, they are bad because they are.. not us. However this looks worrying like a justification that Stalin or the paperclippers could use. See here and here.
Finally, Carlsmith posits a hidden fifth option, for which we current lack good concepts. He points to a notion of trust/ growth/ balance/ attunement. He talks about the colour, ‘green’ from Magic the Gathering, which is about growing in harmony with complex systems, sometimes trusting, sometimes acting. He notes rationalists and EAs are quite historically inimical to this (favouring ‘blue’ and ‘black’). He repeatedly tries to point at this missing way of being.
There are therefore a number of ways of dealing with the future, with a number of flaws.
But there is a second parallel discussion, about how might AIs treat us. Perhaps because our imagination of our future selves informs how we imagine AIs.
For instance if we assume that we cannot trust things (not 1,3) then it’s very easy to see AIs as a tool or competitor. Either it is more powerful than us or we are more powerful than it.
However, if there is a meaningful position on (5). There may be other ways to relate to AIs and future people. Here we might not control them and they might not control us. We might relates as gentle aliens (eg octopus), as dead-but-not-supreme nature (like a bear). Or something even more other than that. Something we cannot imagine but should attempt to.
Note that this doesn’t mean we shouldn’t fear AIs, they might still be capable of ruining the future, but this poise feels different.
In conclusion, this is my shortest summary of this set of blogs (though there is much much more in there). We should consider other ways to be towards those we could control but who might control us. We should consider other possible relationships towards AI. In AI discourse there is a lack of clarity in notions of attunement, respect, harmony in relation to the sub-optimal choices of other conscious beings. It is possible that this affects our priors about what AI might be like, possibly pushing towards a worse equilibrium.