I agree that “Moral Realism AI” was a bit of a misnomer and I’ve changed it to “Convergent Morality AI”.
Your scenario seems highly specific. Could you try to rephrase it in about three sentences, as in the other scenarios?
I’m a bit wary about adding a lot of future scenarios that are outside of our reality and want the scenarios to focus on the future of our universe. However, I do think there is space for a scenario where our reality ends as it has achieved its goals (as in your scenario, I think?).
thanks for changing the name of that scenario. Mine is not just highly specific, it happens to be true in great part: feel free to look at the work of Alan Gewirth and subsequent discussion (the references are all actual).
The main point is indeed that the Orthogonality Thesis is false: for a sufficiently high level of intelligence, human or machine, the Golden Rule is binding. This rules out several of the scenarios now listed (and may help readers to redistribute the probability mass they assign to the remaining ones).
I agree that “Moral Realism AI” was a bit of a misnomer and I’ve changed it to “Convergent Morality AI”.
Your scenario seems highly specific. Could you try to rephrase it in about three sentences, as in the other scenarios?
I’m a bit wary about adding a lot of future scenarios that are outside of our reality and want the scenarios to focus on the future of our universe. However, I do think there is space for a scenario where our reality ends as it has achieved its goals (as in your scenario, I think?).
Dear Bart,
thanks for changing the name of that scenario. Mine is not just highly specific, it happens to be true in great part: feel free to look at the work of Alan Gewirth and subsequent discussion (the references are all actual).
That reality ends when a particular goal is achieved is an old idea (see e.g. https://en.wikipedia.org/wiki/The_Nine_Billion_Names_of_God) In that respect, the scenario I’m discussing is more in line with your “Partially aligned AGI” scenario.
The main point is indeed that the Orthogonality Thesis is false: for a sufficiently high level of intelligence, human or machine, the Golden Rule is binding. This rules out several of the scenarios now listed (and may help readers to redistribute the probability mass they assign to the remaining ones).