Just want to note that I’ve been pushing for (what I think is) a proper amount of uncertainty about “realism about rationality” for a long time. Here’s a collection of quotes from just my top-level posts, arguing against various items in your list:
Is this realistic for human rationalist wannabes? It seems wildly implausible to me that two humans can communicate all of the information they have that is relevant to the truth of some statement just by repeatedly exchanging degrees of belief about it, except in very simple situations. You need to know the other agent’s information partition exactly in order to narrow down which element of the information partition he is in from his probability declaration, and he needs to know that you know so that he can deduce what inference you’re making, in order to continue to the next step, and so on. One error in this process and the whole thing falls apart. It seems much easier to just tell each other what information the two of you have directly.
Finally, I now see that until the exchange of information completes and common knowledge/agreement is actually achieved, it’s rational for even honest truth-seekers who share common priors to disagree. Therefore, two such rationalists may persistently disagree just because the amount of information they would have to exchange in order to reach agreement is too great to be practical.
Considering the agent as a whole suggests that the master’s values are the true terminal values, and the slave’s values are merely instrumental values. From this perspective, the slave seems to be just a subroutine that the master uses to carry out its wishes. Certainly in any given mind there will be numerous subroutines that are tasked with accomplishing various subgoals, and if we were to look at a subroutine in isolation, its assigned subgoal would appear to be its terminal value, but we wouldn’t consider that subgoal to be part of the mind’s true preferences. Why should we treat the slave in this model differently?
What ethical principles can we use to decide between “Shut Up and Multiply” and “Shut Up and Divide”? Why should we derive our values from our native emotional responses to seeing individual suffering, and not from the equally human paucity of response at seeing large portions of humanity suffer in aggregate? Or should we just keep our scope insensitivity, like our boredom?
And an interesting meta-question arises here as well: how much of what we think our values are, is actually the result of not thinking things through, and not realizing the implications and symmetries that exist? And if many of our values are just the result of cognitive errors or limitations, have we lived with them long enough that they’ve become an essential part of us?
By the way, I think nihilism often gets short changed around here. Given that we do not actually have at hand a solution to ontological crises in general or to the specific crisis that we face, what’s wrong with saying that the solution set may just be null? Given that evolution doesn’t constitute a particularly benevolent and farsighted designer, perhaps we may not be able to do much better than that poor spare-change collecting robot? If Eliezer is worried that actual AIs facing actual ontological crises could do worse than just crash, should we be very sanguine that for humans everything must “add up to moral normality”?
Without being a system of logic, moral philosophical reasoning likely (or at least plausibly) doesn’t have any of the nice properties that a well-constructed system of logic would have, for example, consistency, validity, soundness, or even the more basic property that considering arguments in a different order, or in a different mood, won’t cause a person to accept an entirely different set of conclusions. For all we know, somebody trying to reason about a moral concept like “fairness” may just be taking a random walk as they move from one conclusion to another based on moral arguments they encounter or think up.
There aren’t any normative facts at all, including facts about what is rational. For example, it turns out there is no one decision theory that does better than every other decision theory in every situation, and there is no obvious or widely-agreed-upon way to determine which one “wins” overall.
Just want to note that I’ve been pushing for (what I think is) a proper amount of uncertainty about “realism about rationality” for a long time. Here’s a collection of quotes from just my top-level posts, arguing against various items in your list:
-- Probability Space & Aumann Agreement
-- A Master-Slave Model of Human Preferences
-- Shut Up and Divide?
-- Ontological Crisis in Humans
-- Morality Isn’t Logical
-- Six Plausible Meta-Ethical Alternatives