Does the chapter really count as evidence? Normally, X is evidence for Z is P(X|Z) > P(X|not Z). In this case, X = “there are compelling arguments for Z” and you already suppose that X is true whether or not Z. X is therefore not evidence for Z. Of course, after reading the chapter you learn the particular compelling arguments A(1), A(2), … But those arguments support Z only through X and since you know X is not evidence, A(n) are screened off. Put another way, you know that for each A(n) there is an equally compelling argument B(n) that cancels it out. Knowing what the argument actually says is important only if you want to independently determine its compellingness. But you have assumed that you already know this.
Consider a more realistic scenario: you have a coin and two hypotheses:
F: the coin is fair
H: the coin is biased towards heads and it comes up heads twice as frequently as tails
Now you tell your servant: “Toss the coin million times and write the results down. Then, from the record select a subsequence S which has P(S|H) = 1.58 P(S|F) and tell me.” The servant follows your instruction and says “HHTH”. Now you have learned something new (the servant could for example tell “THHH” instead), but you don’t update your odds in favour of H by factor 1.58, because it was almost certain in advance that the servant would be able to locate a subsequence of desired property whether or not H holds.
It’s not obvious to me that X screens off the individual arguments. In particular, X only asserts the existence of at least one compelling argument. If there are multiple, independent compelling arguments in the book, then this should presumably increase our confidence in Z beyond just knowing X. Or am I confused about something?
Also, the individual pieces of evidence could undercut my confidence in the strength of the other books’ arguments conditional on (knowledge of) the evidence I just learned. For example, suppose we expect all arguments from the other books to proceed from some initially plausible premise P. But the Zoroastrian book produces strong evidence against P. Then our evidence goes quite beyond X.
Then substitute “there are many deceptive compelling arguments for Z, much more than any book can contain” for definition of X. The point stands.
I believe your second point is only a specific case of arguments more compelling than expected (here due to their ability to undermine the counterarguments). This is fine—if the arguments are unexpectedly compelling, you update.
On the other hand, the problem was pretty symmetric at the beginning—all books were considered equivalent in their persuasive strength. If book Z argues well against P and all other books took P as granted without similarly undermining the leading premise of book Z the system of books would be unbalanced (reading all books would cause you to believe Z independently of reading order), which would violate the assumed symmetry. So, if you find good anti-P arguments in book Z, the odds are that your assumption about the contents of other books being based on P is incorrect or the other books contain good counterargument which cancels this out. You should be very certain that the system of books is balanced, else the problem doesn’t work.
I think it’s easy to make my second point without the asymmetry. Let’s re-pose the problem so that we expect in advance not only that each book will produce strong evidence in favor of the religion it advocates, but also strong evidence that none of the other books contain strong counter-evidence or similarly undermining evidence. When you read book Z, you learn individual pieces of evidence z1, z2, …, zn. But z1, …, zn undermine your confidence that the other books contain strong arguments, thus disconfirming your belief that you’d likely find convincing evidence for Zoroastrianism in the book whether or not the religion is true. But then it starts looking like we have evidence for Zoroastrianism. However, if, as you argue, z1, …, zn only support Zoroastrianism through things we expected to see in advance of reading the book, then we shouldn’t have any evidence. So either I’m confused or we still have a problem.
The scenario, as I understand it, is based on assumption that the confidence about y = “all books contain equally strong evidence for their respective religion” is high. If y is absolutely certain, p(y) = 1, the confidence cannot be shaken by whatever is found in book Z. If, on the other hand, p(y) is not certain, then what happens depends a lot on relative strength of various pieces of evidence. But this is another (more complex and fuzzier) problem—now you expect that Z not only contains evidence for Zoroastrianism, but also evidence against the very statement of the thought experiment. Doubting y is not included in the original post, where the newly converted Zoroastrian admits that reading book A would deconvert him to atheism; he refrains from doing that only because he fears Ahura Mazda’s wrath.
I think your conclusion there trades on an ambiguity of what “evidence” refers to in your y (= “all books contain equally strong evidence for their respective religion”). The assumption y could mean either:
For each book x, x contains really compelling evidence that we’re sure would equally convince us if we were to encounter it in a normal situation (i.e., without knowing about the other books or the AI’s deviousness).
For each book x, x contains really compelling evidence even after considering and correctly reasoning about all the facts of the thought experiment.
Obviously the second interpretation is either incoherent or completely trivializes the thought experiment, since it’s an assumption about what the all-things-considered best thing to believe after reading a book is, when that’s precisely the question we’re being posed in the first place. On the other hand, the first interpretation, even if assumed with probability 1, is compatible with a given book lowering the posterior expected strength of evidence of the other books.
Fair point. The ambiguity is already included in the original formulation of the thought experiment. The first formulation is compatible with lowering the posterior expected strength of evidence of other books after reading one of them, but it is also compatible with being not convinced by the evidence at all. Assuming the first interpretation the problem is underspecified and no apparent paradox is present.
The second interpretation can have several subinterpretations:
2a) For each book x, reading x convinces ordinary human about the particular proposition argued for in x (possibly using biases and imperfections of human mind).
2b) For each book x, reading x convinces ideal Bayesian reasoner (IBR) about the particular proposition.
2a was probably closest to the meaning intended in the OP. It is a paradox only if we assume that ordinary human resoning is consistent, which we don’t assume, so there is no problem. 2b depends on what IBR exactly means. If it has no limitations on processing speed and memory the thought experiment becomes impossible, since the IBR has already considered all possible arguments and can’t be swayed by rhetorical trickery. If, on the other hand, the IBR has some physical limitations, 2b can be used to show that its thinking leads to inconsistencies, but it is not much more surprising than the same conclusion from the case 2a.
Does the chapter really count as evidence? Normally, X is evidence for Z is P(X|Z) > P(X|not Z). In this case, X = “there are compelling arguments for Z” and you already suppose that X is true whether or not Z. X is therefore not evidence for Z. Of course, after reading the chapter you learn the particular compelling arguments A(1), A(2), … But those arguments support Z only through X and since you know X is not evidence, A(n) are screened off. Put another way, you know that for each A(n) there is an equally compelling argument B(n) that cancels it out. Knowing what the argument actually says is important only if you want to independently determine its compellingness. But you have assumed that you already know this.
Consider a more realistic scenario: you have a coin and two hypotheses:
F: the coin is fair
H: the coin is biased towards heads and it comes up heads twice as frequently as tails
Now you tell your servant: “Toss the coin million times and write the results down. Then, from the record select a subsequence S which has P(S|H) = 1.58 P(S|F) and tell me.” The servant follows your instruction and says “HHTH”. Now you have learned something new (the servant could for example tell “THHH” instead), but you don’t update your odds in favour of H by factor 1.58, because it was almost certain in advance that the servant would be able to locate a subsequence of desired property whether or not H holds.
It’s not obvious to me that X screens off the individual arguments. In particular, X only asserts the existence of at least one compelling argument. If there are multiple, independent compelling arguments in the book, then this should presumably increase our confidence in Z beyond just knowing X. Or am I confused about something?
Also, the individual pieces of evidence could undercut my confidence in the strength of the other books’ arguments conditional on (knowledge of) the evidence I just learned. For example, suppose we expect all arguments from the other books to proceed from some initially plausible premise P. But the Zoroastrian book produces strong evidence against P. Then our evidence goes quite beyond X.
Then substitute “there are many deceptive compelling arguments for Z, much more than any book can contain” for definition of X. The point stands.
I believe your second point is only a specific case of arguments more compelling than expected (here due to their ability to undermine the counterarguments). This is fine—if the arguments are unexpectedly compelling, you update.
On the other hand, the problem was pretty symmetric at the beginning—all books were considered equivalent in their persuasive strength. If book Z argues well against P and all other books took P as granted without similarly undermining the leading premise of book Z the system of books would be unbalanced (reading all books would cause you to believe Z independently of reading order), which would violate the assumed symmetry. So, if you find good anti-P arguments in book Z, the odds are that your assumption about the contents of other books being based on P is incorrect or the other books contain good counterargument which cancels this out. You should be very certain that the system of books is balanced, else the problem doesn’t work.
I think it’s easy to make my second point without the asymmetry. Let’s re-pose the problem so that we expect in advance not only that each book will produce strong evidence in favor of the religion it advocates, but also strong evidence that none of the other books contain strong counter-evidence or similarly undermining evidence. When you read book Z, you learn individual pieces of evidence z1, z2, …, zn. But z1, …, zn undermine your confidence that the other books contain strong arguments, thus disconfirming your belief that you’d likely find convincing evidence for Zoroastrianism in the book whether or not the religion is true. But then it starts looking like we have evidence for Zoroastrianism. However, if, as you argue, z1, …, zn only support Zoroastrianism through things we expected to see in advance of reading the book, then we shouldn’t have any evidence. So either I’m confused or we still have a problem.
The scenario, as I understand it, is based on assumption that the confidence about y = “all books contain equally strong evidence for their respective religion” is high. If y is absolutely certain, p(y) = 1, the confidence cannot be shaken by whatever is found in book Z. If, on the other hand, p(y) is not certain, then what happens depends a lot on relative strength of various pieces of evidence. But this is another (more complex and fuzzier) problem—now you expect that Z not only contains evidence for Zoroastrianism, but also evidence against the very statement of the thought experiment. Doubting y is not included in the original post, where the newly converted Zoroastrian admits that reading book A would deconvert him to atheism; he refrains from doing that only because he fears Ahura Mazda’s wrath.
I think your conclusion there trades on an ambiguity of what “evidence” refers to in your y (= “all books contain equally strong evidence for their respective religion”). The assumption y could mean either:
For each book x, x contains really compelling evidence that we’re sure would equally convince us if we were to encounter it in a normal situation (i.e., without knowing about the other books or the AI’s deviousness).
For each book x, x contains really compelling evidence even after considering and correctly reasoning about all the facts of the thought experiment.
Obviously the second interpretation is either incoherent or completely trivializes the thought experiment, since it’s an assumption about what the all-things-considered best thing to believe after reading a book is, when that’s precisely the question we’re being posed in the first place. On the other hand, the first interpretation, even if assumed with probability 1, is compatible with a given book lowering the posterior expected strength of evidence of the other books.
Fair point. The ambiguity is already included in the original formulation of the thought experiment. The first formulation is compatible with lowering the posterior expected strength of evidence of other books after reading one of them, but it is also compatible with being not convinced by the evidence at all. Assuming the first interpretation the problem is underspecified and no apparent paradox is present.
The second interpretation can have several subinterpretations:
2a) For each book x, reading x convinces ordinary human about the particular proposition argued for in x (possibly using biases and imperfections of human mind).
2b) For each book x, reading x convinces ideal Bayesian reasoner (IBR) about the particular proposition.
2a was probably closest to the meaning intended in the OP. It is a paradox only if we assume that ordinary human resoning is consistent, which we don’t assume, so there is no problem. 2b depends on what IBR exactly means. If it has no limitations on processing speed and memory the thought experiment becomes impossible, since the IBR has already considered all possible arguments and can’t be swayed by rhetorical trickery. If, on the other hand, the IBR has some physical limitations, 2b can be used to show that its thinking leads to inconsistencies, but it is not much more surprising than the same conclusion from the case 2a.