There are some papers that describe ways to achieve agreement in other ways, such as iterative exchange of posterior probabilities. But in such methods, the agents aren’t just moving closer to each other’s beliefs. Rather, they go through convoluted chains of deduction to infer what information the other agent must have observed, given his declarations, and then update on that new information. (The process is similar to the one needed to solve the second riddle on this page.) The two agents essentially still have to communicate I(w) and J(w) to each other, except they do so by exchanging posterior probabilities and making logical inferences from them.
Is this realistic for human rationalist wannabes? It seems wildly implausible to me that two humans can communicate all of the information they have that is relevant to the truth of some statement just by repeatedly exchanging degrees of belief about it, except in very simple situations. You need to know the other agent’s information partition exactly in order to narrow down which element of the information partition he is in from his probability declaration, and he needs to know that you know so that he can deduce what inference you’re making, in order to continue to the next step, and so on. One error in this process and the whole thing falls apart. It seems much easier to just tell each other what information the two of you have directly.
I won’t try to comment on the formal argument (my understanding that literature is mostly just what Robin Hanson has said about it), but intuitively, this seems wrong. It seems like two people trading probability estimates shouldn’t need to deduce exactly what the other has observed, they just need to make inferences along the lines of, “wow, she wasn’t swayed as much as I expected by me telling her my opinion, she must think she has some pretty good evidence.” At least that’s the inference you would make if you both knew you trust each other’s rationality. More realistically, of course, the correct inference is usually “she wasn’t swayed by me telling her my opinion, she doesn’t just trust me to be rational.”
Consider what would have to happen for two rationalists who knowingly trust each other’s rationality to have a persistent disagreement. Because of conservation of expected evidence, Alice has to think her probability estimate would on average remain the same after hearing Bob’s evidence, and Bob must think the same about hearing Alice’s evidence. That seems to suggest they both must think they have better, more relevant evidence to the question at hand. And might be perfectly reasonable for them to think that at first.
But after several rounds of sharing their probability estimates and seeing the other not budge, Alice will have to realize Bob thinks he’s better informed about the topic than she is. And Bob will have to realize the same about Alice. And if they both trust each other’s rationality, Alice will have to think, “I thought I was better informed than Bob about this, but it looks like Bob thinks he’s the one who’s better informed, so maybe I’m wrong about being better informed.” And Bob will have to have the parallel thought. Eventually, they should converge.
I won’t try to comment on the formal argument (my understanding that literature is mostly just what Robin Hanson has said about it), but intuitively, this seems wrong.
Wei Dai’s description is correct, see here for an example where the final estimate is outside the range of the initial two. And yes, the Aumann agreement theorem does not say what nearly everyone (including Eliezer) seems to intuitively think it says.
And yes, the Aumann agreement theorem does not say what nearly everyone (including Eliezer) seems to intuitively think it says.
Wonder if a list of such things can be constructed. Algorithmic information theory is an example where Eliezer drew the wrong implications from the math and unfortunately much of LessWrong inherited that. Group selection (multi-level selection) might be another example, but less clear cut, as that requires computational modeling and not just interpretation of mathematics. I’m sure there are more and better examples.
I won’t try to comment on the formal argument (my understanding that literature is mostly just what Robin Hanson has said about it), but intuitively, this seems wrong. It seems like two people trading probability estimates shouldn’t need to deduce exactly what the other has observed, they just need to make inferences along the lines of, “wow, she wasn’t swayed as much as I expected by me telling her my opinion, she must think she has some pretty good evidence.” At least that’s the inference you would make if you both knew you trust each other’s rationality. More realistically, of course, the correct inference is usually “she wasn’t swayed by me telling her my opinion, she doesn’t just trust me to be rational.”
Consider what would have to happen for two rationalists who knowingly trust each other’s rationality to have a persistent disagreement. Because of conservation of expected evidence, Alice has to think her probability estimate would on average remain the same after hearing Bob’s evidence, and Bob must think the same about hearing Alice’s evidence. That seems to suggest they both must think they have better, more relevant evidence to the question at hand. And might be perfectly reasonable for them to think that at first.
But after several rounds of sharing their probability estimates and seeing the other not budge, Alice will have to realize Bob thinks he’s better informed about the topic than she is. And Bob will have to realize the same about Alice. And if they both trust each other’s rationality, Alice will have to think, “I thought I was better informed than Bob about this, but it looks like Bob thinks he’s the one who’s better informed, so maybe I’m wrong about being better informed.” And Bob will have to have the parallel thought. Eventually, they should converge.
Wei Dai’s description is correct, see here for an example where the final estimate is outside the range of the initial two. And yes, the Aumann agreement theorem does not say what nearly everyone (including Eliezer) seems to intuitively think it says.
Wonder if a list of such things can be constructed. Algorithmic information theory is an example where Eliezer drew the wrong implications from the math and unfortunately much of LessWrong inherited that. Group selection (multi-level selection) might be another example, but less clear cut, as that requires computational modeling and not just interpretation of mathematics. I’m sure there are more and better examples.