I’m inclined to agree with your actual point here, but it might help to be clearer on the distinction between “a group of idealized, albeit bounded, rationalists” as opposed to “a group of painfully biased actual humans who are trying to be rational”, i.e., us.
Most of the potential conflicts between your four forms of rationality apply only to the latter case—which is not to say we should ignore them, quite the opposite in fact. So, to avoid distractions about how hypothetical true rationalists should always agree and whatnot, it may be helpful to make explicit that what you’re proposing is a kludge to work around systematic human irrationality, not a universal principle of rationality.
In conventional decision/game theory, there is often conflict between individual and group rationality even if we assume idealized (non-altruistic) individuals. Eliezer and others have been working on more advanced decision/game theories which may be able to avoid these conflicts, but that’s still fairly speculative at this point. If we put that work aside, I think my point about over- and under-confidence hurting individual rationality, but possibly helping group rationality (by lessening the public goods problem in knowledge production), is a general one.
There is one paragraph in my post that is not about rationality in general, but only meant to apply to humans, but I made that pretty clear, I think:
If you’re overconfident in an idea, then you would tend to be more interested
in working out its applications. Conversely, if you’re underconfident in it (i.e., are
excessively skeptical), you would tend to work harder to try to find its flaws.
For ideal rational agents with converging confidences, you could still get a spread of activities (not confidence levels) in a community, because if an angle (excessive skepticism for example) is not being explored enough, the potential payoff for working on it increases even if your confidence remains unchanged. But you seem to want to change activities by changing confidence levels, that is, hacking human irrationality.
Well said. I agree with Wei’s point with the same clarifications you supply here. Looking at any potential desirability of individual calibration among otherwise ideal rationalists may be an interesting question in itself but it is a different point and it is important not to blur the lines between ‘applicable to all of humanity’ and ‘universal principle of rationality’. When things are presented as universal I have to go around disagreeing with their claims even when I totally agree with the principle.
I think it’s not a case of blurring the line, but instead there’s probably a substantive disagreement between us about whether one of my points applies generally to rational agents or just to humans. Would you or SoullessAutomaton please explain why you don’t think it applies generally?
Sorry for the late reply; I don’t have much time for LW these days, sadly.
Based on some of your comments, perhaps I’m operating under a different definition of group vs. individual rationality? If uncoordinated individuals making locally optimal choices would lead to a suboptimal global outcome, and this is generally known to the group, then they must act to rationally solve the coordination problem, not merely fall back to non-coordination. A bunch of people unanimously playing D in the prisoner’s dilemma are clearly not, in any coherent sense, rationally maximizing individual outcomes. Thus I don’t really see such a scenario as presenting a group vs. individual conflict, but rather a practical problem of coordinated action. Certainly, solving such problems applies to any rational agent, not just humans.
The part about giving undue weight to unlikely ideas—which seems to comprise about half the post—by mis-calibrating confidence levels to motivate behavior seems to be strictly human-oriented. Lacking the presence of human cognitive biases, the decision to examine low-confidence ideas is just another coordination issue with no special features; in fact it’s an unusually tractable one, as a passable solution exists (random choice, as per CannibalSmith’s comment, which was also my immediate thought) even with the presumption that coordination is not only expensive but essentially impossible!
Overall, any largely symmetric, fault-tolerant coordination problem that can be trivially resolved by a quasi-Kantian maxim of “always take the action that would work out best if everyone took that action” is a “problem” only insofar as humans are unreliable and will probably screw up; thus any proposed solution is necessarily non-general.
The situation is much stickier in other cases; for instance, if coordination costs are comparable to the gains from coordination, or if it’s not clear that every individual has a reasonable expectation of preferring the group-optimal outcome, or if the optimal actions are asymmetric in ways not locally obvious, or if the optimal group action isn’t amenable to a partition/parallelize/recombine algorithm. But none of those are the case in your example! Perhaps that sort of thing is what Eliezer et al. are working on, but (due to aforementioned time constraints) I’ve not kept up with LW, so you’ll have to forgive me if this is all old hat.
At any rate, tl;dr version: wedrifid’s “Anything an irrational agent can do due to an epistemic flaw a rational agent can do because it is the best thing for it to do.” and the associated comment thread pretty much covers what I had in mind when I left the earlier comment. Hope that clarifies matters.
I get that impression too now that you have said that there was only one small part applied to humans specifically. I will see if Soulless can (and wants to) express his position succinctly first. It is a significant topic and I know that for me to cover it adequately I would need to spend considerable time (and words) to express it and Soulless may have an answer cached.
I’m inclined to agree with your actual point here, but it might help to be clearer on the distinction between “a group of idealized, albeit bounded, rationalists” as opposed to “a group of painfully biased actual humans who are trying to be rational”, i.e., us.
Most of the potential conflicts between your four forms of rationality apply only to the latter case—which is not to say we should ignore them, quite the opposite in fact. So, to avoid distractions about how hypothetical true rationalists should always agree and whatnot, it may be helpful to make explicit that what you’re proposing is a kludge to work around systematic human irrationality, not a universal principle of rationality.
In conventional decision/game theory, there is often conflict between individual and group rationality even if we assume idealized (non-altruistic) individuals. Eliezer and others have been working on more advanced decision/game theories which may be able to avoid these conflicts, but that’s still fairly speculative at this point. If we put that work aside, I think my point about over- and under-confidence hurting individual rationality, but possibly helping group rationality (by lessening the public goods problem in knowledge production), is a general one.
There is one paragraph in my post that is not about rationality in general, but only meant to apply to humans, but I made that pretty clear, I think:
Then why the appeal to human biases? Here:
For ideal rational agents with converging confidences, you could still get a spread of activities (not confidence levels) in a community, because if an angle (excessive skepticism for example) is not being explored enough, the potential payoff for working on it increases even if your confidence remains unchanged. But you seem to want to change activities by changing confidence levels, that is, hacking human irrationality.
Well said. I agree with Wei’s point with the same clarifications you supply here. Looking at any potential desirability of individual calibration among otherwise ideal rationalists may be an interesting question in itself but it is a different point and it is important not to blur the lines between ‘applicable to all of humanity’ and ‘universal principle of rationality’. When things are presented as universal I have to go around disagreeing with their claims even when I totally agree with the principle.
I think it’s not a case of blurring the line, but instead there’s probably a substantive disagreement between us about whether one of my points applies generally to rational agents or just to humans. Would you or SoullessAutomaton please explain why you don’t think it applies generally?
Sorry for the late reply; I don’t have much time for LW these days, sadly.
Based on some of your comments, perhaps I’m operating under a different definition of group vs. individual rationality? If uncoordinated individuals making locally optimal choices would lead to a suboptimal global outcome, and this is generally known to the group, then they must act to rationally solve the coordination problem, not merely fall back to non-coordination. A bunch of people unanimously playing D in the prisoner’s dilemma are clearly not, in any coherent sense, rationally maximizing individual outcomes. Thus I don’t really see such a scenario as presenting a group vs. individual conflict, but rather a practical problem of coordinated action. Certainly, solving such problems applies to any rational agent, not just humans.
The part about giving undue weight to unlikely ideas—which seems to comprise about half the post—by mis-calibrating confidence levels to motivate behavior seems to be strictly human-oriented. Lacking the presence of human cognitive biases, the decision to examine low-confidence ideas is just another coordination issue with no special features; in fact it’s an unusually tractable one, as a passable solution exists (random choice, as per CannibalSmith’s comment, which was also my immediate thought) even with the presumption that coordination is not only expensive but essentially impossible!
Overall, any largely symmetric, fault-tolerant coordination problem that can be trivially resolved by a quasi-Kantian maxim of “always take the action that would work out best if everyone took that action” is a “problem” only insofar as humans are unreliable and will probably screw up; thus any proposed solution is necessarily non-general.
The situation is much stickier in other cases; for instance, if coordination costs are comparable to the gains from coordination, or if it’s not clear that every individual has a reasonable expectation of preferring the group-optimal outcome, or if the optimal actions are asymmetric in ways not locally obvious, or if the optimal group action isn’t amenable to a partition/parallelize/recombine algorithm. But none of those are the case in your example! Perhaps that sort of thing is what Eliezer et al. are working on, but (due to aforementioned time constraints) I’ve not kept up with LW, so you’ll have to forgive me if this is all old hat.
At any rate, tl;dr version: wedrifid’s “Anything an irrational agent can do due to an epistemic flaw a rational agent can do because it is the best thing for it to do.” and the associated comment thread pretty much covers what I had in mind when I left the earlier comment. Hope that clarifies matters.
I get that impression too now that you have said that there was only one small part applied to humans specifically. I will see if Soulless can (and wants to) express his position succinctly first. It is a significant topic and I know that for me to cover it adequately I would need to spend considerable time (and words) to express it and Soulless may have an answer cached.