The proposition that we should be able to reason about priors of other agents is surely not contentious. The proposition that if I learn something new about my creation, I should update on that information, is also surely not contentious, although there might be some disagreement what that update looks like.
The form is the interesting thing here. By arguing for the common prior assumption, RH is giving an argument in favor of a form of modest epistemology, which Eliezer has recently written so much against.
In the case of genetics, if I learned that I’m genetically predisposed to being optimistic, then I would update my beliefs the same way I would update them if I had performed a calibration and found my estimates consistently too high. That is unless I’ve performed calibrations in the past and know myself to be well calibrated. In that case the genetic predisposition isn’t giving me any new information—I’ve already corrected for it. This, again, surely isn’t contentious.
In Eliezer’s view, because there are no universally compelling arguments and recursive justification has to hit bottom, you don’t give up your prior just because you see that there was bias in the process which created it—nothing can be totally justified by an infinite chain of unbiased steps anyway. This means, concretely, that you shouldn’t automatically take the “outside view” on the beliefs you have which others are most likely to consider crazy; their disbelief is little evidence, if you have a strong inside view reason why you can know better than them.
In RH’s view, honest truth-seeking agents with common priors should not knowingly disagree (citation: are disagreements honest?). Since a failure of the common prior assumption entails disagreement about the origin of priors (thanks to the pre-rationality condition), and RH thinks disagreement about the origin of priors should rarely be relevant for disagreements about humans, RH thinks honest truth-seeking humans should not knowingly disagree.
I take it RH thinks some averaging should happen somewhere as a result of that, though I am not entirely sure. This would contradict Eliezer’s view.
Although I have no idea what this has to do with “species average”. Yes, I have no reason to believe that my priors are better than everybody else’s, but I also have no reason to believe that the “species average” is better than my current prior (there is also the problem that “species” is an arbitrarily chosen category).
The wording in the paper makes me think RH was intending “species” as a line beyond which the argument might fail, not one he’s necessarily going to draw (IE, he might concede that his argument can’t support a common prior assumption with aliens, but he might not concede that).
I think he does take this to be reason to believe the species average is better than your current prior, to the extent they differ.
But aside from that, I struggle to understand what, in simple terms, is being disagreed about here.
I see several large disagreements.
Is the pre-rationality condition a true constraint on rationality? RH finds it plausible; WD does not. I am conflicted.
If the pre-rationality argument makes sense for common probabilities, does it then make sense for utilities? WD thinks so; RH thinks not.
Does pre-rationality imply a rational creator? WD thinks so; RH thinks not.
Does the pre-prior formalism make sense at all? Can rationality conditions stated with use of pre-priors have any force for ordinary agents who do not reason using pre-priors? I think not, though I think there is perhaps something to be salvaged out of it.
Does the common prior assumption make sense in practice?
Should honest truth-seeking humans knowingly disagree?
Does the common prior assumption make sense in practice?
I don’t know what “make sense” means. When I said “in simple terms”, I meant that I want to avoid that sort of vagueness. The disagreement should be empirical. It seems that we need to simulate an environment with a group of bayesians with different priors, then somehow construct another group of bayesians that satisfy the pre-rationality condition, and then the claim should be that the second group outperforms the first group in accuracy. But I don’t think I saw such claims in the paper explicitly. So I continue to be confused, what exactly the disagreement is about.
Should honest truth-seeking humans knowingly disagree?
Big question, I’m not going to make big claims here, though my intuition tends to say “yes”. Also, “should” is a bad word, I’m assuming that you’re referring to accuracy (as in my previous paragraph), but I’d like to see these things stated explicitly.
you don’t give up your prior just because you see that there was bias in the process which created it
Of course not. But you do modify it. What is RH suggesting?
Is the pre-rationality condition a true constraint on rationality? RH finds it plausible; WD does not. I am conflicted.
“True” is a bad word, I have no idea what it means.
If the pre-rationality argument makes sense for common probabilities, does it then make sense for utilities? WD thinks so; RH thinks not.
RH gives a reasonable argument here, and I don’t see much of a reason why we would do this to utilities in the first place.
Does pre-rationality imply a rational creator? WD thinks so; RH thinks not.
I see literally nothing in the paper to suggest anything about this. I don’t know what WD is talking about.
I don’t know what “make sense” means. When I said “in simple terms”, I meant that I want to avoid that sort of vagueness. The disagreement should be empirical. It seems that we need to simulate an environment with a group of bayesians with different priors, then somehow construct another group of bayesians that satisfy the pre-rationality condition, and then the claim should be that the second group outperforms the first group in accuracy. But I don’t think I saw such claims in the paper explicitly. So I continue to be confused, what exactly the disagreement is about.
Ah, well, I think the “should honest truth-seeking humans knowingly disagree” is the practical form of this for RH.
(Or, even more practical, practices around such disagreements. Should a persistent disagreement be a sign of dishonesty? (RH says yes.) Should we drop beliefs which others persistently disagree with? Et cetera.)
Big question, I’m not going to make big claims here, though my intuition tends to say “yes”.
Then (according to RH), you have to deal with RH’s arguments to the contrary. Specifically, his paper is claiming that you have to have some origin disputes about other people’s priors.
Although that’s not why I’m grappling with his argument. I’m not sure whether rational truth-seekers should persistently disagree, but I’m very curious about some things going on in RH’s argument.
Also, “should” is a bad word, I’m assuming that you’re referring to accuracy (as in my previous paragraph), but I’d like to see these things stated explicitly.
I think “should” is a good word to sometimes taboo (IE taboo quickly if there seems to be any problem), but I don’t see that it needs to be an always-taboo word.
“True” is a bad word, I have no idea what it means.
I’m also pretty unclear on what it could possibly mean here, but nonetheless think it is worth debating. Not only is there the usual problem of spelling out what it means for something to be a constraint on rationality, but now there’s an extra weird thing going on with setting up pre-priors which aren’t probabilities which you use to make any decisions.
I see literally nothing in the paper to suggest anything about this. I don’t know what WD is talking about.
I think I know what WD is talking about, but I agree it isn’t what RH is really trying to say.
the usual problem of spelling out what it means for something to be a constraint on rationality
Is that a problem? What’s wrong with “believing true things”, or, more precisely, “winning bets”? (obviously, these need to be prefixed with “usually” and “across many possible universes”). If I’m being naive and these don’t work, then I’d love to hear about it.
But if they do work, then I really want to see how the idea of pre-rationality is supposed to help me believe more true things and win more bets. I legitimately don’t understand how it would.
Should honest truth-seeking humans knowingly disagree?
My intuition says “yes” in large part due to the word “humans”. I’m not certain whether two perfect bayesians should disagree, for some unrealistic sense of “perfect”, but even if they shouldn’t, it is uncertain that this would also apply to more limited agents.
Is that a problem? What’s wrong with “believing true things”, or, more precisely, “winning bets”?
Winning bets is not literally the same thing as believing true things, nor is it the same thing as having accurate beliefs, or being rational. I think Dutch-book arguments are … not exactly mistaken, but misleading, for this reason. It is not true that the only reason to have probabilistically coherent beliefs is to avoid reliably losing bets. If that were the case, we could throw rationality out the window whenever bets aren’t involved. I think betting is both a helpful illustrative thought experiment (dutch books illustrate irrationality) and a helpful tool for practicing rationality, but not synonymous with rationality.
“Believing true things” is problematic for several reasons. First, that is apparently entirely focused on epistemic rationality, excluding instrumental rationality. Second, although there are many practical cases where it isn’t a problem, there is a question of what “true” means, especially for high-level beliefs about things like tables and chairs which are more like conceptual clusters rather than objective realities. Third, even setting those aside, it is hard to see how we can get from “believing true things” to Bayes’ Law and other rules of probabilistic reasoning. I would argue that a solid connection between “believe true things” and rationality constraints of classical logic can be made, but probabilistic reasoning requires an additional insight about what kind of thing can be a rationality constraint: you don’t just have beliefs, you have degrees of belief. We can say things about why degrees of belief might be better or worse, but to do so requires a notion of quality of belief which goes beyond truth alone; you are not maximizing the expected amount of truth or anything like that.
Another possible answer, which you didn’t name but which might have been named, would be “rationality is about winning”. Something important is meant by this, but the idea is still vague—it helps point toward things that do look like potential rationality constraints and away from things which can’t serve as rationality constraints, but it is not the end of the story of what we might possibly mean by calling something a constraint of rationality.
My intuition says “yes” in large part due to the word “humans”. I’m not certain whether two perfect bayesians should disagree, for some unrealistic sense of “perfect”, but even if they shouldn’t, it is uncertain that this would also apply to more limited agents.
Most of my probability mass is on you being right here, but I find RH’s arguments to the contrary intriguing. It’s not so much that I’m engaging with them in the expectation that I’ll change my mind about whether honest truth seeking humans can knowingly disagree. (Actually I think I should have said “can” all along rather than “should” now that I think about it more!) I do, however, expect something about the structure of those disagreements can be understood more thoroughly. If ideal Bayesians always agree, that could mean understanding the ways that Bayesian assumptions break down for humans. If ideal Bayesians need not agree, it might mean understanding that better.
I really want to see how the idea of pre-rationality is supposed to help me believe more true things and win more bets. I legitimately don’t understand how it would.
I think I can understand this one, to some extent. Supposing that some version of pre-rationality does work out, and if I, hypothetically, understood pre-rationality extremely well (better than RH’s paper explains it)… I would expect more insights into at least one of the following:
agents reasoning about their own prior (what is the structure of the reasoning? to what extent can an agent approve of, or not approve of, its own prior? are there things which can make an agent decide its own prior is bad? what must an agent believe about the process which created its prior? What should an agent do if it discovers that the process which created its prior was biased, or systematically not truth-seeking, or otherwise ‘problematic’?)
common knowledge of beliefs (is it realistic for beliefs to be common knowledge? when? are there more things to say about the structure of common knowledge, which help reconcile the usual assumption that an agent knows its own prior with the paradoxes of self-reference which prevent agents from knowing themselves so well?)
what it means for an agent to have a prior (how to we designate a special belief-state to call the prior, for realistic agents? can we do so at all in the face of logical uncertainty? is it better to just think in terms of a sequence of belief states, with some being relatively prior to others? can we make good models of agents who are becoming rational as they are learning, such that they lack an initial perfectly rational prior?)
reaching agreement with other agents (by an Aumann-like process or otherwise; by bringing in origin disputes or otherwise)
reasoning about one’s own origins (especially in the sense of justification structures; endorsing or not endorsing the way one’s beliefs were constructed or the way those beliefs became what they are more generally).
Winning bets is not literally the same thing as believing true things, nor is it the same thing as having accurate beliefs, or being rational.
They are not the same, but that’s ok. You asked about constraints on, not definitions of rationality. This may not be an exhaustive list, but if someone has an idea about rationality that translates neither into winning some hypothetical bets, nor into having even slightly more accurate beliefs about anything, then I can confidently say that I’m not interested.
(Of course this is not to say that an idea that has no such applications has literally zero value)
Supposing that some version of pre-rationality does work out, and if I, hypothetically, understood pre-rationality extremely well (better than RH’s paper explains it)… I would expect more insights into at least one of the following: <...>
I completely agree that if RH was right, and if you understood him well, then you would receive multiple benefits, most of which could translate into winning hypothetical bets, and into having more accurate beliefs about many things. But that’s just the usual effect of learning, and not because you would satisfy the pre-rationality condition.
I continue to not understand in what precise way the agent that satisfies the pre-rationality condition is (claimed to be) superior to the agent that doesn’t. To be fair, this could be a hard question, and even if we don’t immediately see the benefit, that doesn’t mean that there is no benefit. But still, I’m quite suspicious. In my view this is the single most important question, and it’s weird to me that I don’t see it explicitly addressed.
The form is the interesting thing here. By arguing for the common prior assumption, RH is giving an argument in favor of a form of modest epistemology, which Eliezer has recently written so much against.
In Eliezer’s view, because there are no universally compelling arguments and recursive justification has to hit bottom, you don’t give up your prior just because you see that there was bias in the process which created it—nothing can be totally justified by an infinite chain of unbiased steps anyway. This means, concretely, that you shouldn’t automatically take the “outside view” on the beliefs you have which others are most likely to consider crazy; their disbelief is little evidence, if you have a strong inside view reason why you can know better than them.
In RH’s view, honest truth-seeking agents with common priors should not knowingly disagree (citation: are disagreements honest?). Since a failure of the common prior assumption entails disagreement about the origin of priors (thanks to the pre-rationality condition), and RH thinks disagreement about the origin of priors should rarely be relevant for disagreements about humans, RH thinks honest truth-seeking humans should not knowingly disagree.
I take it RH thinks some averaging should happen somewhere as a result of that, though I am not entirely sure. This would contradict Eliezer’s view.
The wording in the paper makes me think RH was intending “species” as a line beyond which the argument might fail, not one he’s necessarily going to draw (IE, he might concede that his argument can’t support a common prior assumption with aliens, but he might not concede that).
I think he does take this to be reason to believe the species average is better than your current prior, to the extent they differ.
I see several large disagreements.
Is the pre-rationality condition a true constraint on rationality? RH finds it plausible; WD does not. I am conflicted.
If the pre-rationality argument makes sense for common probabilities, does it then make sense for utilities? WD thinks so; RH thinks not.
Does pre-rationality imply a rational creator? WD thinks so; RH thinks not.
Does the pre-prior formalism make sense at all? Can rationality conditions stated with use of pre-priors have any force for ordinary agents who do not reason using pre-priors? I think not, though I think there is perhaps something to be salvaged out of it.
Does the common prior assumption make sense in practice?
Should honest truth-seeking humans knowingly disagree?
I don’t know what “make sense” means. When I said “in simple terms”, I meant that I want to avoid that sort of vagueness. The disagreement should be empirical. It seems that we need to simulate an environment with a group of bayesians with different priors, then somehow construct another group of bayesians that satisfy the pre-rationality condition, and then the claim should be that the second group outperforms the first group in accuracy. But I don’t think I saw such claims in the paper explicitly. So I continue to be confused, what exactly the disagreement is about.
Big question, I’m not going to make big claims here, though my intuition tends to say “yes”. Also, “should” is a bad word, I’m assuming that you’re referring to accuracy (as in my previous paragraph), but I’d like to see these things stated explicitly.
Of course not. But you do modify it. What is RH suggesting?
“True” is a bad word, I have no idea what it means.
RH gives a reasonable argument here, and I don’t see much of a reason why we would do this to utilities in the first place.
I see literally nothing in the paper to suggest anything about this. I don’t know what WD is talking about.
Ah, well, I think the “should honest truth-seeking humans knowingly disagree” is the practical form of this for RH.
(Or, even more practical, practices around such disagreements. Should a persistent disagreement be a sign of dishonesty? (RH says yes.) Should we drop beliefs which others persistently disagree with? Et cetera.)
Then (according to RH), you have to deal with RH’s arguments to the contrary. Specifically, his paper is claiming that you have to have some origin disputes about other people’s priors.
Although that’s not why I’m grappling with his argument. I’m not sure whether rational truth-seekers should persistently disagree, but I’m very curious about some things going on in RH’s argument.
I think “should” is a good word to sometimes taboo (IE taboo quickly if there seems to be any problem), but I don’t see that it needs to be an always-taboo word.
I’m also pretty unclear on what it could possibly mean here, but nonetheless think it is worth debating. Not only is there the usual problem of spelling out what it means for something to be a constraint on rationality, but now there’s an extra weird thing going on with setting up pre-priors which aren’t probabilities which you use to make any decisions.
I think I know what WD is talking about, but I agree it isn’t what RH is really trying to say.
Is that a problem? What’s wrong with “believing true things”, or, more precisely, “winning bets”? (obviously, these need to be prefixed with “usually” and “across many possible universes”). If I’m being naive and these don’t work, then I’d love to hear about it.
But if they do work, then I really want to see how the idea of pre-rationality is supposed to help me believe more true things and win more bets. I legitimately don’t understand how it would.
My intuition says “yes” in large part due to the word “humans”. I’m not certain whether two perfect bayesians should disagree, for some unrealistic sense of “perfect”, but even if they shouldn’t, it is uncertain that this would also apply to more limited agents.
Winning bets is not literally the same thing as believing true things, nor is it the same thing as having accurate beliefs, or being rational. I think Dutch-book arguments are … not exactly mistaken, but misleading, for this reason. It is not true that the only reason to have probabilistically coherent beliefs is to avoid reliably losing bets. If that were the case, we could throw rationality out the window whenever bets aren’t involved. I think betting is both a helpful illustrative thought experiment (dutch books illustrate irrationality) and a helpful tool for practicing rationality, but not synonymous with rationality.
“Believing true things” is problematic for several reasons. First, that is apparently entirely focused on epistemic rationality, excluding instrumental rationality. Second, although there are many practical cases where it isn’t a problem, there is a question of what “true” means, especially for high-level beliefs about things like tables and chairs which are more like conceptual clusters rather than objective realities. Third, even setting those aside, it is hard to see how we can get from “believing true things” to Bayes’ Law and other rules of probabilistic reasoning. I would argue that a solid connection between “believe true things” and rationality constraints of classical logic can be made, but probabilistic reasoning requires an additional insight about what kind of thing can be a rationality constraint: you don’t just have beliefs, you have degrees of belief. We can say things about why degrees of belief might be better or worse, but to do so requires a notion of quality of belief which goes beyond truth alone; you are not maximizing the expected amount of truth or anything like that.
Another possible answer, which you didn’t name but which might have been named, would be “rationality is about winning”. Something important is meant by this, but the idea is still vague—it helps point toward things that do look like potential rationality constraints and away from things which can’t serve as rationality constraints, but it is not the end of the story of what we might possibly mean by calling something a constraint of rationality.
Most of my probability mass is on you being right here, but I find RH’s arguments to the contrary intriguing. It’s not so much that I’m engaging with them in the expectation that I’ll change my mind about whether honest truth seeking humans can knowingly disagree. (Actually I think I should have said “can” all along rather than “should” now that I think about it more!) I do, however, expect something about the structure of those disagreements can be understood more thoroughly. If ideal Bayesians always agree, that could mean understanding the ways that Bayesian assumptions break down for humans. If ideal Bayesians need not agree, it might mean understanding that better.
I think I can understand this one, to some extent. Supposing that some version of pre-rationality does work out, and if I, hypothetically, understood pre-rationality extremely well (better than RH’s paper explains it)… I would expect more insights into at least one of the following:
agents reasoning about their own prior (what is the structure of the reasoning? to what extent can an agent approve of, or not approve of, its own prior? are there things which can make an agent decide its own prior is bad? what must an agent believe about the process which created its prior? What should an agent do if it discovers that the process which created its prior was biased, or systematically not truth-seeking, or otherwise ‘problematic’?)
common knowledge of beliefs (is it realistic for beliefs to be common knowledge? when? are there more things to say about the structure of common knowledge, which help reconcile the usual assumption that an agent knows its own prior with the paradoxes of self-reference which prevent agents from knowing themselves so well?)
what it means for an agent to have a prior (how to we designate a special belief-state to call the prior, for realistic agents? can we do so at all in the face of logical uncertainty? is it better to just think in terms of a sequence of belief states, with some being relatively prior to others? can we make good models of agents who are becoming rational as they are learning, such that they lack an initial perfectly rational prior?)
reaching agreement with other agents (by an Aumann-like process or otherwise; by bringing in origin disputes or otherwise)
reasoning about one’s own origins (especially in the sense of justification structures; endorsing or not endorsing the way one’s beliefs were constructed or the way those beliefs became what they are more generally).
They are not the same, but that’s ok. You asked about constraints on, not definitions of rationality. This may not be an exhaustive list, but if someone has an idea about rationality that translates neither into winning some hypothetical bets, nor into having even slightly more accurate beliefs about anything, then I can confidently say that I’m not interested.
(Of course this is not to say that an idea that has no such applications has literally zero value)
I completely agree that if RH was right, and if you understood him well, then you would receive multiple benefits, most of which could translate into winning hypothetical bets, and into having more accurate beliefs about many things. But that’s just the usual effect of learning, and not because you would satisfy the pre-rationality condition.
I continue to not understand in what precise way the agent that satisfies the pre-rationality condition is (claimed to be) superior to the agent that doesn’t. To be fair, this could be a hard question, and even if we don’t immediately see the benefit, that doesn’t mean that there is no benefit. But still, I’m quite suspicious. In my view this is the single most important question, and it’s weird to me that I don’t see it explicitly addressed.