It’s explicitly opposition to core Sequences content, which Eliezer felt was important enough to write a whole additional philosophical dialogue about after the main Sequences were done. Eliezer’s response when informed about it was:
is somebody trolling? Have they never read anything I’ve written in my entire life? Do they have no sense, even, of irony?
That doesn’t seem like Eliezer agrees with you that someone got this wrong by accident, that seems like Eliezer agrees with me that someone identifying as a Rationalist has to be trying to get core things wrong to end up saying something like that.
I don’t think this follows. I do not see how degree of wrongness implies intent. Eliezer’s comment rhetorically suggests intent (“trolling”) as a way of highlighting how wrong the person is; he is free to correct me if I am wrong, but I am pretty sure that is not an actual suggestion of intent, only a rhetorical one.
I would say moreover, that this is the sort of mistake that occurs, over and over, by default, with no intent necessary. I might even say that it is avoiding, not committing, this sort of mistake, that requires intent. Because this sort of mistake is just sort of what people fall into by default, and avoiding it requires active effort.
Is it contrary to everything Eliezer’s ever written? Sure! But reading the entirety of the Sequences, calling yourself a “rationalist”, does not in any way obviate the need to do the actual work of better group epistemology, of noticing such mistakes (and the path to them) and correcting/avoiding them.
I think we can only infer intent like you’re talking about if the person in question is, actually, y’know, thinking about what they’re doing. But I think people are really, like, acting on autopilot a pretty big fraction of the time; not autopiloting takes effort, and doing that work may be what a “rationalist” is supposed to do, it’s still not the default. All I think we can infer from this is a failure to do the work to shift out of autopilot and think. Bad group epistemology via laziness rather than via intent strikes me as the more likely explanation.
This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.
Once is happenstance. Twice is coincidence. Three times is enemy action.
I can imagine, after reading the sequences, continuing to have the epistemic modesty bias in my own thoughts, but I don’t see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.
Behavior is better explained as strategy than as error, if the behaviors add up to push the world in some direction (along a dimension that’s “distant” from the behavior, like how “make a box with food appear at my door” is “distant” from “wiggle my fingers on my keyboard”). If a pattern of correlated error is the sort of pattern that doesn’t easily push the world in a direction, then that pattern might be evidence against intent. For example, the conjunction fallacy will produce a pattern of wrong probability estimates with a distinct character, but it seems unlikely to push the world in some specific direction (beyond whatever happens when you have incoherent probabilities). (Maybe this argument is fuzzy on the edges, like if someone keeps trying to show you information and you keep ignoring it, you’re sort of “pushing the world in a direction” when compared to what’s “supposed to happen”, i.e. that you update; which suggests intent, although it’s “reactive” rather than “proactive”, whatever that means. I at least claim that your argument is too general, proves too much, and would be more clear if it were narrower.)
The effective direction the epistemic modesty / argument from authority bias pushes things, is away from shared narrative as something that dynamically adjusts to new information, and towards shared narrative as a way to identify and coordinate who’s receiving instructions from whom.
People frequently make “mistakes” as a form of submission, and it shouldn’t be surprising that other types of systematic error function as a means of domination, i.e. of submission enforcement.
If someone makes correlated errors, they are better explained as part of a strategy.
That does seem right to me.
It seems like very often correlated errors are the result of a mistaken, upstream crux. They’re making one mistake, which is flowing into a bunch of specific instances.
This at least has to be another hypothesis, along with “this is a conscious or unconscious strategy to get what they want.”
This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.
I mean, there is a word for correlated errors, and that word is “bias”; so you seem to be essentially claiming that people are unbiased? I’m guessing that’s probably not what you’re trying to claim, but that is what I am concluding? Regardless, I’m saying people are biased towards this mistake.
Or really, what I’m saying it’s the same sort of phenomenon that Eliezer discusses here. So it could indeed be construed as a strategy as you say; but it would not be a strategy on the part of the conscious agent, but rather a strategy on the part of the “corrupted hardware” itself. Or something like that—sorry, that’s not a great way of putting it, but I don’t really have a better one, and I hope that conveys what I’m getting at.
Like, I think you’re assuming too much awareness/agency of people. A person who makes correlated errors, and is aware of what they are doing, is executing a deliberate strategy. But lots of people who make correlated errors are just biased, or the errors are part of a built-in strategy they’re executing, not deliberately, but by default without thinking about it, that requires effort not to execute.
We should expect someone calling themself a rationalist to be better, obviously, but, IDK, sometimes things go bad?
I can imagine, after reading the sequences, continuing to have this bias in my own thoughts, but I don’t see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.
I mean people don’t necessarily fully internalize everything they read, and in some people the “hold on what am I doing?” can be weak? <shrug>
I mean I certainly don’t want to rule out deliberate malice like you’re talking about, but neither do I think this one snippet is enough to strongly conclude it.
In most cases it seems intentional but not deliberate. People will resist pressure to change the pattern, or find new ways to execute it if the specific way they were engaged in this bias is effectively discouraged, but don’t consciously represent to themselves their intent to do it or engage in explicit means-ends reasoning about it.
Yeah, that sounds about right to me. I’m not saying that you should assume such people are harmless or anything! Just that, like, you might want to try giving them a kick first—“hey, constant vigilance, remember?” :P—and see how they respond before giving up and treating them as hostile.
It’s explicitly opposition to core Sequences content, which Eliezer felt was important enough to write a whole additional philosophical dialogue about after the main Sequences were done. Eliezer’s response when informed about it was:
That doesn’t seem like Eliezer agrees with you that someone got this wrong by accident, that seems like Eliezer agrees with me that someone identifying as a Rationalist has to be trying to get core things wrong to end up saying something like that.
I don’t think this follows. I do not see how degree of wrongness implies intent. Eliezer’s comment rhetorically suggests intent (“trolling”) as a way of highlighting how wrong the person is; he is free to correct me if I am wrong, but I am pretty sure that is not an actual suggestion of intent, only a rhetorical one.
I would say moreover, that this is the sort of mistake that occurs, over and over, by default, with no intent necessary. I might even say that it is avoiding, not committing, this sort of mistake, that requires intent. Because this sort of mistake is just sort of what people fall into by default, and avoiding it requires active effort.
Is it contrary to everything Eliezer’s ever written? Sure! But reading the entirety of the Sequences, calling yourself a “rationalist”, does not in any way obviate the need to do the actual work of better group epistemology, of noticing such mistakes (and the path to them) and correcting/avoiding them.
I think we can only infer intent like you’re talking about if the person in question is, actually, y’know, thinking about what they’re doing. But I think people are really, like, acting on autopilot a pretty big fraction of the time; not autopiloting takes effort, and doing that work may be what a “rationalist” is supposed to do, it’s still not the default. All I think we can infer from this is a failure to do the work to shift out of autopilot and think. Bad group epistemology via laziness rather than via intent strikes me as the more likely explanation.
This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.
I can imagine, after reading the sequences, continuing to have the epistemic modesty bias in my own thoughts, but I don’t see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.
Behavior is better explained as strategy than as error, if the behaviors add up to push the world in some direction (along a dimension that’s “distant” from the behavior, like how “make a box with food appear at my door” is “distant” from “wiggle my fingers on my keyboard”). If a pattern of correlated error is the sort of pattern that doesn’t easily push the world in a direction, then that pattern might be evidence against intent. For example, the conjunction fallacy will produce a pattern of wrong probability estimates with a distinct character, but it seems unlikely to push the world in some specific direction (beyond whatever happens when you have incoherent probabilities). (Maybe this argument is fuzzy on the edges, like if someone keeps trying to show you information and you keep ignoring it, you’re sort of “pushing the world in a direction” when compared to what’s “supposed to happen”, i.e. that you update; which suggests intent, although it’s “reactive” rather than “proactive”, whatever that means. I at least claim that your argument is too general, proves too much, and would be more clear if it were narrower.)
The effective direction the epistemic modesty / argument from authority bias pushes things, is away from shared narrative as something that dynamically adjusts to new information, and towards shared narrative as a way to identify and coordinate who’s receiving instructions from whom.
People frequently make “mistakes” as a form of submission, and it shouldn’t be surprising that other types of systematic error function as a means of domination, i.e. of submission enforcement.
(I indeed find this a more clear+compelling argument and appreciate you trying to make this known.)
That does seem right to me.
It seems like very often correlated errors are the result of a mistaken, upstream crux. They’re making one mistake, which is flowing into a bunch of specific instances.
This at least has to be another hypothesis, along with “this is a conscious or unconscious strategy to get what they want.”
I mean, there is a word for correlated errors, and that word is “bias”; so you seem to be essentially claiming that people are unbiased? I’m guessing that’s probably not what you’re trying to claim, but that is what I am concluding? Regardless, I’m saying people are biased towards this mistake.
Or really, what I’m saying it’s the same sort of phenomenon that Eliezer discusses here. So it could indeed be construed as a strategy as you say; but it would not be a strategy on the part of the conscious agent, but rather a strategy on the part of the “corrupted hardware” itself. Or something like that—sorry, that’s not a great way of putting it, but I don’t really have a better one, and I hope that conveys what I’m getting at.
Like, I think you’re assuming too much awareness/agency of people. A person who makes correlated errors, and is aware of what they are doing, is executing a deliberate strategy. But lots of people who make correlated errors are just biased, or the errors are part of a built-in strategy they’re executing, not deliberately, but by default without thinking about it, that requires effort not to execute.
We should expect someone calling themself a rationalist to be better, obviously, but, IDK, sometimes things go bad?
I mean people don’t necessarily fully internalize everything they read, and in some people the “hold on what am I doing?” can be weak? <shrug>
I mean I certainly don’t want to rule out deliberate malice like you’re talking about, but neither do I think this one snippet is enough to strongly conclude it.
In most cases it seems intentional but not deliberate. People will resist pressure to change the pattern, or find new ways to execute it if the specific way they were engaged in this bias is effectively discouraged, but don’t consciously represent to themselves their intent to do it or engage in explicit means-ends reasoning about it.
Yeah, that sounds about right to me. I’m not saying that you should assume such people are harmless or anything! Just that, like, you might want to try giving them a kick first—“hey, constant vigilance, remember?” :P—and see how they respond before giving up and treating them as hostile.