« Then you believe in universally compelling arguments processed by a ghost in the machine. For every possible mind whose utility function assigns terminal value +1, mind design space contains an equal and opposite mind whose utility function assigns terminal value −1. »
That’s true but that doesn’t say anything about the sustainability of that given mind design (the possibility for the mind design to survive, either by having the individual to survive, or to create new individuals with similar mind designs).
A mind design that would value its own death very positively would commit suicide (unless unable to) and stop existing. A mind design that value pain (in the abstract meaning, dealing damages to its own physical support) would be much less likely to survive.
Many other possible mind designs would become completely tied in contradictions and chaos, and be unable to achieve anything. Think about a mind without “modus ponens”, or a mind for which the values are chosen randomly and change every nanosecond...
And it gets even narrower when the mind is not alone, but has to live in society with other similar minds.
So while mind design space contains the exact opposite of every possible terminal values, the “coherent mind design space” (minds that are coherent enough to not self-destruct, not paralyze themselves and not collapse into chaos) is probably more limited.
But that still is so big that it contains minds like the paper-clip optimizer wanting the tile the solar system with paper-clips, no doubt about that one.
« Then you believe in universally compelling arguments processed by a ghost in the machine. For every possible mind whose utility function assigns terminal value +1, mind design space contains an equal and opposite mind whose utility function assigns terminal value −1. »
That’s true but that doesn’t say anything about the sustainability of that given mind design (the possibility for the mind design to survive, either by having the individual to survive, or to create new individuals with similar mind designs).
A mind design that would value its own death very positively would commit suicide (unless unable to) and stop existing. A mind design that value pain (in the abstract meaning, dealing damages to its own physical support) would be much less likely to survive.
Many other possible mind designs would become completely tied in contradictions and chaos, and be unable to achieve anything. Think about a mind without “modus ponens”, or a mind for which the values are chosen randomly and change every nanosecond...
And it gets even narrower when the mind is not alone, but has to live in society with other similar minds.
So while mind design space contains the exact opposite of every possible terminal values, the “coherent mind design space” (minds that are coherent enough to not self-destruct, not paralyze themselves and not collapse into chaos) is probably more limited.
But that still is so big that it contains minds like the paper-clip optimizer wanting the tile the solar system with paper-clips, no doubt about that one.