I think it’s easy to make my second point without the asymmetry. Let’s re-pose the problem so that we expect in advance not only that each book will produce strong evidence in favor of the religion it advocates, but also strong evidence that none of the other books contain strong counter-evidence or similarly undermining evidence. When you read book Z, you learn individual pieces of evidence z1, z2, …, zn. But z1, …, zn undermine your confidence that the other books contain strong arguments, thus disconfirming your belief that you’d likely find convincing evidence for Zoroastrianism in the book whether or not the religion is true. But then it starts looking like we have evidence for Zoroastrianism. However, if, as you argue, z1, …, zn only support Zoroastrianism through things we expected to see in advance of reading the book, then we shouldn’t have any evidence. So either I’m confused or we still have a problem.
The scenario, as I understand it, is based on assumption that the confidence about y = “all books contain equally strong evidence for their respective religion” is high. If y is absolutely certain, p(y) = 1, the confidence cannot be shaken by whatever is found in book Z. If, on the other hand, p(y) is not certain, then what happens depends a lot on relative strength of various pieces of evidence. But this is another (more complex and fuzzier) problem—now you expect that Z not only contains evidence for Zoroastrianism, but also evidence against the very statement of the thought experiment. Doubting y is not included in the original post, where the newly converted Zoroastrian admits that reading book A would deconvert him to atheism; he refrains from doing that only because he fears Ahura Mazda’s wrath.
I think your conclusion there trades on an ambiguity of what “evidence” refers to in your y (= “all books contain equally strong evidence for their respective religion”). The assumption y could mean either:
For each book x, x contains really compelling evidence that we’re sure would equally convince us if we were to encounter it in a normal situation (i.e., without knowing about the other books or the AI’s deviousness).
For each book x, x contains really compelling evidence even after considering and correctly reasoning about all the facts of the thought experiment.
Obviously the second interpretation is either incoherent or completely trivializes the thought experiment, since it’s an assumption about what the all-things-considered best thing to believe after reading a book is, when that’s precisely the question we’re being posed in the first place. On the other hand, the first interpretation, even if assumed with probability 1, is compatible with a given book lowering the posterior expected strength of evidence of the other books.
Fair point. The ambiguity is already included in the original formulation of the thought experiment. The first formulation is compatible with lowering the posterior expected strength of evidence of other books after reading one of them, but it is also compatible with being not convinced by the evidence at all. Assuming the first interpretation the problem is underspecified and no apparent paradox is present.
The second interpretation can have several subinterpretations:
2a) For each book x, reading x convinces ordinary human about the particular proposition argued for in x (possibly using biases and imperfections of human mind).
2b) For each book x, reading x convinces ideal Bayesian reasoner (IBR) about the particular proposition.
2a was probably closest to the meaning intended in the OP. It is a paradox only if we assume that ordinary human resoning is consistent, which we don’t assume, so there is no problem. 2b depends on what IBR exactly means. If it has no limitations on processing speed and memory the thought experiment becomes impossible, since the IBR has already considered all possible arguments and can’t be swayed by rhetorical trickery. If, on the other hand, the IBR has some physical limitations, 2b can be used to show that its thinking leads to inconsistencies, but it is not much more surprising than the same conclusion from the case 2a.
I think it’s easy to make my second point without the asymmetry. Let’s re-pose the problem so that we expect in advance not only that each book will produce strong evidence in favor of the religion it advocates, but also strong evidence that none of the other books contain strong counter-evidence or similarly undermining evidence. When you read book Z, you learn individual pieces of evidence z1, z2, …, zn. But z1, …, zn undermine your confidence that the other books contain strong arguments, thus disconfirming your belief that you’d likely find convincing evidence for Zoroastrianism in the book whether or not the religion is true. But then it starts looking like we have evidence for Zoroastrianism. However, if, as you argue, z1, …, zn only support Zoroastrianism through things we expected to see in advance of reading the book, then we shouldn’t have any evidence. So either I’m confused or we still have a problem.
The scenario, as I understand it, is based on assumption that the confidence about y = “all books contain equally strong evidence for their respective religion” is high. If y is absolutely certain, p(y) = 1, the confidence cannot be shaken by whatever is found in book Z. If, on the other hand, p(y) is not certain, then what happens depends a lot on relative strength of various pieces of evidence. But this is another (more complex and fuzzier) problem—now you expect that Z not only contains evidence for Zoroastrianism, but also evidence against the very statement of the thought experiment. Doubting y is not included in the original post, where the newly converted Zoroastrian admits that reading book A would deconvert him to atheism; he refrains from doing that only because he fears Ahura Mazda’s wrath.
I think your conclusion there trades on an ambiguity of what “evidence” refers to in your y (= “all books contain equally strong evidence for their respective religion”). The assumption y could mean either:
For each book x, x contains really compelling evidence that we’re sure would equally convince us if we were to encounter it in a normal situation (i.e., without knowing about the other books or the AI’s deviousness).
For each book x, x contains really compelling evidence even after considering and correctly reasoning about all the facts of the thought experiment.
Obviously the second interpretation is either incoherent or completely trivializes the thought experiment, since it’s an assumption about what the all-things-considered best thing to believe after reading a book is, when that’s precisely the question we’re being posed in the first place. On the other hand, the first interpretation, even if assumed with probability 1, is compatible with a given book lowering the posterior expected strength of evidence of the other books.
Fair point. The ambiguity is already included in the original formulation of the thought experiment. The first formulation is compatible with lowering the posterior expected strength of evidence of other books after reading one of them, but it is also compatible with being not convinced by the evidence at all. Assuming the first interpretation the problem is underspecified and no apparent paradox is present.
The second interpretation can have several subinterpretations:
2a) For each book x, reading x convinces ordinary human about the particular proposition argued for in x (possibly using biases and imperfections of human mind).
2b) For each book x, reading x convinces ideal Bayesian reasoner (IBR) about the particular proposition.
2a was probably closest to the meaning intended in the OP. It is a paradox only if we assume that ordinary human resoning is consistent, which we don’t assume, so there is no problem. 2b depends on what IBR exactly means. If it has no limitations on processing speed and memory the thought experiment becomes impossible, since the IBR has already considered all possible arguments and can’t be swayed by rhetorical trickery. If, on the other hand, the IBR has some physical limitations, 2b can be used to show that its thinking leads to inconsistencies, but it is not much more surprising than the same conclusion from the case 2a.