TurnTrout comments on My Objections to “We’re All Gonna Die with Eliezer Yudkowsky”

[TurnTrout]

Here are some of my disagreements with List of Lethalities. I’ll quote item one:

“Humans don’t explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn’t produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about, and it seems to me that there are deep theoretical reasons to expect it to happen again”

(Evolution) → (human values) is not the only case of inner alignment failure which we know about. I have argued that human values themselves are inner alignment failures on the human reward system. This has happened billions of times in slightly different learning setups.