I’m glad you agree that the behavior Jessica describes is explicitly opposed to the content of the Sequences, and that you clearly care a lot about this. I don’t think anyone can reasonably claim you didn’t try hard to get people to behave better, or could reasonably blame you for the fact that many people persistently try to do the opposite of what you say, in the name of Rationality.
I do think it would be a very good idea for you to investigate why & how the institutions you helped build and are still actively participating in are optimizing against your explicitly stated intentions. Anna’s endorsement of this post seems like reasonably strong confirmation that organizations nominally committed to your agenda are actually opposed to it, unless you’re actually checking. And MIRI/CFAR donors seem to for the most part think that you’re aware of and endorse those orgs’ activities.
When Jessica and another recent MIRI employee asked a few years ago for some of your time to explain why they’d left, your response was:
My guess is that I could talk over Signal voice for 30 minutes or in person for 15 minutes on the 15th, with an upcoming other commitment providing a definite cutoff point and also meaning that it wouldn’t otherwise be an uninterrupted day. That’s not enough time to persuade each other things, but I suspect that neither would be a week, and hopefully it’s enough that you can convey to me any information you want me to know and don’t want to write down. Again, for framing, this is a sort of thing I basically don’t do anymore due to stamina limitations—Nate talks to people, I talk to Nate.
You responded a little bit by email, but didn’t seem very interested in what was going on (you didn’t ask the kinds of followup questions that would establish common knowledge about agreement or disagreement), so your interlocutors didn’t perceive a real opportunity to inform you of these dynamics at that time.
Anna’s endorsement of this post seems like reasonably strong confirmation that organizations nominally committed to your agenda are actually opposed to it,
Presumably Eliezer’s agenda is much broader than “make sure nobody tries to socially enforce deferral to high-status figures in an ungrounded way” though I do think this is part of his goals.
The above seems to me like it tries to equivocate between “this is confirmation that at least some people don’t act in full agreement with your agenda, despite being nominally committed to it” and “this is confirmation that people are actively working against your agenda”. These two really don’t strike me as the same, and I really don’t like how this comment seems like it tries to equivocate between the two.
Of course, the claim that some chunk of the community/organizations Eliezer created are working actively against some agenda that Eliezer tried to set for them is plausible. But calling the above a “strong confirmation” of this fact strikes me as a very substantial stretch.
It’s explicitly opposition to core Sequences content, which Eliezer felt was important enough to write a whole additional philosophical dialogue about after the main Sequences were done. Eliezer’s response when informed about it was:
is somebody trolling? Have they never read anything I’ve written in my entire life? Do they have no sense, even, of irony?
That doesn’t seem like Eliezer agrees with you that someone got this wrong by accident, that seems like Eliezer agrees with me that someone identifying as a Rationalist has to be trying to get core things wrong to end up saying something like that.
I don’t think this follows. I do not see how degree of wrongness implies intent. Eliezer’s comment rhetorically suggests intent (“trolling”) as a way of highlighting how wrong the person is; he is free to correct me if I am wrong, but I am pretty sure that is not an actual suggestion of intent, only a rhetorical one.
I would say moreover, that this is the sort of mistake that occurs, over and over, by default, with no intent necessary. I might even say that it is avoiding, not committing, this sort of mistake, that requires intent. Because this sort of mistake is just sort of what people fall into by default, and avoiding it requires active effort.
Is it contrary to everything Eliezer’s ever written? Sure! But reading the entirety of the Sequences, calling yourself a “rationalist”, does not in any way obviate the need to do the actual work of better group epistemology, of noticing such mistakes (and the path to them) and correcting/avoiding them.
I think we can only infer intent like you’re talking about if the person in question is, actually, y’know, thinking about what they’re doing. But I think people are really, like, acting on autopilot a pretty big fraction of the time; not autopiloting takes effort, and doing that work may be what a “rationalist” is supposed to do, it’s still not the default. All I think we can infer from this is a failure to do the work to shift out of autopilot and think. Bad group epistemology via laziness rather than via intent strikes me as the more likely explanation.
This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.
Once is happenstance. Twice is coincidence. Three times is enemy action.
I can imagine, after reading the sequences, continuing to have the epistemic modesty bias in my own thoughts, but I don’t see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.
Behavior is better explained as strategy than as error, if the behaviors add up to push the world in some direction (along a dimension that’s “distant” from the behavior, like how “make a box with food appear at my door” is “distant” from “wiggle my fingers on my keyboard”). If a pattern of correlated error is the sort of pattern that doesn’t easily push the world in a direction, then that pattern might be evidence against intent. For example, the conjunction fallacy will produce a pattern of wrong probability estimates with a distinct character, but it seems unlikely to push the world in some specific direction (beyond whatever happens when you have incoherent probabilities). (Maybe this argument is fuzzy on the edges, like if someone keeps trying to show you information and you keep ignoring it, you’re sort of “pushing the world in a direction” when compared to what’s “supposed to happen”, i.e. that you update; which suggests intent, although it’s “reactive” rather than “proactive”, whatever that means. I at least claim that your argument is too general, proves too much, and would be more clear if it were narrower.)
The effective direction the epistemic modesty / argument from authority bias pushes things, is away from shared narrative as something that dynamically adjusts to new information, and towards shared narrative as a way to identify and coordinate who’s receiving instructions from whom.
People frequently make “mistakes” as a form of submission, and it shouldn’t be surprising that other types of systematic error function as a means of domination, i.e. of submission enforcement.
If someone makes correlated errors, they are better explained as part of a strategy.
That does seem right to me.
It seems like very often correlated errors are the result of a mistaken, upstream crux. They’re making one mistake, which is flowing into a bunch of specific instances.
This at least has to be another hypothesis, along with “this is a conscious or unconscious strategy to get what they want.”
This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.
I mean, there is a word for correlated errors, and that word is “bias”; so you seem to be essentially claiming that people are unbiased? I’m guessing that’s probably not what you’re trying to claim, but that is what I am concluding? Regardless, I’m saying people are biased towards this mistake.
Or really, what I’m saying it’s the same sort of phenomenon that Eliezer discusses here. So it could indeed be construed as a strategy as you say; but it would not be a strategy on the part of the conscious agent, but rather a strategy on the part of the “corrupted hardware” itself. Or something like that—sorry, that’s not a great way of putting it, but I don’t really have a better one, and I hope that conveys what I’m getting at.
Like, I think you’re assuming too much awareness/agency of people. A person who makes correlated errors, and is aware of what they are doing, is executing a deliberate strategy. But lots of people who make correlated errors are just biased, or the errors are part of a built-in strategy they’re executing, not deliberately, but by default without thinking about it, that requires effort not to execute.
We should expect someone calling themself a rationalist to be better, obviously, but, IDK, sometimes things go bad?
I can imagine, after reading the sequences, continuing to have this bias in my own thoughts, but I don’t see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.
I mean people don’t necessarily fully internalize everything they read, and in some people the “hold on what am I doing?” can be weak? <shrug>
I mean I certainly don’t want to rule out deliberate malice like you’re talking about, but neither do I think this one snippet is enough to strongly conclude it.
In most cases it seems intentional but not deliberate. People will resist pressure to change the pattern, or find new ways to execute it if the specific way they were engaged in this bias is effectively discouraged, but don’t consciously represent to themselves their intent to do it or engage in explicit means-ends reasoning about it.
Yeah, that sounds about right to me. I’m not saying that you should assume such people are harmless or anything! Just that, like, you might want to try giving them a kick first—“hey, constant vigilance, remember?” :P—and see how they respond before giving up and treating them as hostile.
I’m glad you agree that the behavior Jessica describes is explicitly opposed to the content of the Sequences, and that you clearly care a lot about this. I don’t think anyone can reasonably claim you didn’t try hard to get people to behave better, or could reasonably blame you for the fact that many people persistently try to do the opposite of what you say, in the name of Rationality.
I do think it would be a very good idea for you to investigate why & how the institutions you helped build and are still actively participating in are optimizing against your explicitly stated intentions. Anna’s endorsement of this post seems like reasonably strong confirmation that organizations nominally committed to your agenda are actually opposed to it, unless you’re actually checking. And MIRI/CFAR donors seem to for the most part think that you’re aware of and endorse those orgs’ activities.
When Jessica and another recent MIRI employee asked a few years ago for some of your time to explain why they’d left, your response was:
You responded a little bit by email, but didn’t seem very interested in what was going on (you didn’t ask the kinds of followup questions that would establish common knowledge about agreement or disagreement), so your interlocutors didn’t perceive a real opportunity to inform you of these dynamics at that time.
Presumably Eliezer’s agenda is much broader than “make sure nobody tries to socially enforce deferral to high-status figures in an ungrounded way” though I do think this is part of his goals.
The above seems to me like it tries to equivocate between “this is confirmation that at least some people don’t act in full agreement with your agenda, despite being nominally committed to it” and “this is confirmation that people are actively working against your agenda”. These two really don’t strike me as the same, and I really don’t like how this comment seems like it tries to equivocate between the two.
Of course, the claim that some chunk of the community/organizations Eliezer created are working actively against some agenda that Eliezer tried to set for them is plausible. But calling the above a “strong confirmation” of this fact strikes me as a very substantial stretch.
It’s explicitly opposition to core Sequences content, which Eliezer felt was important enough to write a whole additional philosophical dialogue about after the main Sequences were done. Eliezer’s response when informed about it was:
That doesn’t seem like Eliezer agrees with you that someone got this wrong by accident, that seems like Eliezer agrees with me that someone identifying as a Rationalist has to be trying to get core things wrong to end up saying something like that.
I don’t think this follows. I do not see how degree of wrongness implies intent. Eliezer’s comment rhetorically suggests intent (“trolling”) as a way of highlighting how wrong the person is; he is free to correct me if I am wrong, but I am pretty sure that is not an actual suggestion of intent, only a rhetorical one.
I would say moreover, that this is the sort of mistake that occurs, over and over, by default, with no intent necessary. I might even say that it is avoiding, not committing, this sort of mistake, that requires intent. Because this sort of mistake is just sort of what people fall into by default, and avoiding it requires active effort.
Is it contrary to everything Eliezer’s ever written? Sure! But reading the entirety of the Sequences, calling yourself a “rationalist”, does not in any way obviate the need to do the actual work of better group epistemology, of noticing such mistakes (and the path to them) and correcting/avoiding them.
I think we can only infer intent like you’re talking about if the person in question is, actually, y’know, thinking about what they’re doing. But I think people are really, like, acting on autopilot a pretty big fraction of the time; not autopiloting takes effort, and doing that work may be what a “rationalist” is supposed to do, it’s still not the default. All I think we can infer from this is a failure to do the work to shift out of autopilot and think. Bad group epistemology via laziness rather than via intent strikes me as the more likely explanation.
This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.
I can imagine, after reading the sequences, continuing to have the epistemic modesty bias in my own thoughts, but I don’t see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.
Behavior is better explained as strategy than as error, if the behaviors add up to push the world in some direction (along a dimension that’s “distant” from the behavior, like how “make a box with food appear at my door” is “distant” from “wiggle my fingers on my keyboard”). If a pattern of correlated error is the sort of pattern that doesn’t easily push the world in a direction, then that pattern might be evidence against intent. For example, the conjunction fallacy will produce a pattern of wrong probability estimates with a distinct character, but it seems unlikely to push the world in some specific direction (beyond whatever happens when you have incoherent probabilities). (Maybe this argument is fuzzy on the edges, like if someone keeps trying to show you information and you keep ignoring it, you’re sort of “pushing the world in a direction” when compared to what’s “supposed to happen”, i.e. that you update; which suggests intent, although it’s “reactive” rather than “proactive”, whatever that means. I at least claim that your argument is too general, proves too much, and would be more clear if it were narrower.)
The effective direction the epistemic modesty / argument from authority bias pushes things, is away from shared narrative as something that dynamically adjusts to new information, and towards shared narrative as a way to identify and coordinate who’s receiving instructions from whom.
People frequently make “mistakes” as a form of submission, and it shouldn’t be surprising that other types of systematic error function as a means of domination, i.e. of submission enforcement.
(I indeed find this a more clear+compelling argument and appreciate you trying to make this known.)
That does seem right to me.
It seems like very often correlated errors are the result of a mistaken, upstream crux. They’re making one mistake, which is flowing into a bunch of specific instances.
This at least has to be another hypothesis, along with “this is a conscious or unconscious strategy to get what they want.”
I mean, there is a word for correlated errors, and that word is “bias”; so you seem to be essentially claiming that people are unbiased? I’m guessing that’s probably not what you’re trying to claim, but that is what I am concluding? Regardless, I’m saying people are biased towards this mistake.
Or really, what I’m saying it’s the same sort of phenomenon that Eliezer discusses here. So it could indeed be construed as a strategy as you say; but it would not be a strategy on the part of the conscious agent, but rather a strategy on the part of the “corrupted hardware” itself. Or something like that—sorry, that’s not a great way of putting it, but I don’t really have a better one, and I hope that conveys what I’m getting at.
Like, I think you’re assuming too much awareness/agency of people. A person who makes correlated errors, and is aware of what they are doing, is executing a deliberate strategy. But lots of people who make correlated errors are just biased, or the errors are part of a built-in strategy they’re executing, not deliberately, but by default without thinking about it, that requires effort not to execute.
We should expect someone calling themself a rationalist to be better, obviously, but, IDK, sometimes things go bad?
I mean people don’t necessarily fully internalize everything they read, and in some people the “hold on what am I doing?” can be weak? <shrug>
I mean I certainly don’t want to rule out deliberate malice like you’re talking about, but neither do I think this one snippet is enough to strongly conclude it.
In most cases it seems intentional but not deliberate. People will resist pressure to change the pattern, or find new ways to execute it if the specific way they were engaged in this bias is effectively discouraged, but don’t consciously represent to themselves their intent to do it or engage in explicit means-ends reasoning about it.
Yeah, that sounds about right to me. I’m not saying that you should assume such people are harmless or anything! Just that, like, you might want to try giving them a kick first—“hey, constant vigilance, remember?” :P—and see how they respond before giving up and treating them as hostile.