This sentence of Eliezer’s is where the action is:
I’m suspicious of claims that supposedly do not require justification and yet seem to be uniquely preferred within a rather large space of possibilities.
“There are no married bachelors” gets us to nod our heads because we uniquely prefer English syntax and semantics. We pick it out of the rather large space of possible languages because it’s what everyone else is doing.
If Eliezer went around earnestly saying, “there are some married bachelors,” I would guess he had entangled himself with an environment where people go around saying such things, with a different possible language.
Eliezer insists that he could be entangled with evidence such that he believes “there are some married bachelors” is true in English as we know it. I don’t think he could; that proposition is unthinkable in good English.
I concede (a little)!
In a previous Overcoming Bias post we learned that people sometimes believe the conjunction of events R and Q is more probable than event Q alone. Thus people can believe simple and strictly illogical things, and so I shouldn’t throw around the word “unthinkable.”
If I stretch my imagination, I can just maybe imagine this sort of logical blunder with small integers.
I draw the line at P AND ~P, though: just unthinkable.