What’s wrong with the AI making life into a RPG (or multiple thereof)? People like stories and they like levelling up, collecting stuff, crafting, competing, etc. A story doesn’t have to be pure fun (and those sort of stories are boring anyway).
E.g. Eliezer seems to think it’s not the perfect future: “The presence or absence of an external puppet master can affect my valuation of an otherwise fixed outcome. Even if people wouldn’t know they were being manipulated, it would matter to my judgment of how well humanity had done with its future. This is an important ethical issue, if you’re dealing with agents powerful enough to helpfully tweak people’s futures without their knowledge”.
Also, you write:
If we want to have a shot at creating a truly enduring culture (of the kind that is needed to get us off this planet and out into the galaxy)
If we really want this, we have to restrain from spending our whole lives playing the best RPG possible.
Never mind AI, they’re contradictory when executed by us. We aren’t robots following a prioritised script and an AI wouldn’t be either.
Consider human rules “you are allowed to lie to someone for the sake of their own utility” and “everyone should be able to take control of their own life”. We know that lies about serious things never turn out good, so we lie only about things of little importance, and little lies like “yes grandma, that was very tasty” doesn’t contradict the second rule. This looks different when you are an ultimate deceiver.
-
E.g. Eliezer seems to think it’s not the perfect future: “The presence or absence of an external puppet master can affect my valuation of an otherwise fixed outcome. Even if people wouldn’t know they were being manipulated, it would matter to my judgment of how well humanity had done with its future. This is an important ethical issue, if you’re dealing with agents powerful enough to helpfully tweak people’s futures without their knowledge”.
Also, you write:
If we really want this, we have to restrain from spending our whole lives playing the best RPG possible.
Consider human rules “you are allowed to lie to someone for the sake of their own utility” and “everyone should be able to take control of their own life”. We know that lies about serious things never turn out good, so we lie only about things of little importance, and little lies like “yes grandma, that was very tasty” doesn’t contradict the second rule. This looks different when you are an ultimate deceiver.
-