I think that negative low-level associations really matter if you’re trying to be a mass movement and scale, like a political movement.
Many of the world’s smartest, most competent, and most influential people are ideologues. This probably includes whoever ends up developing and controlling advanced technologies. It would be nice to be able to avoid such people dismissing our ideas out of hand. You may not find them impressive or expect them to make intellectual progress on rationality, but for such progress to matter, the ideas have to be taken seriously outside LW at some point. I guess I don’t understand the case against caution in this area, so long as the cost is only having to avoid some peripheral topics instead of adopting or promoting false beliefs.
Rather than debating the case for or against caution, I think the most interesting question is how to arrange a peaceful schism. Team Shared Maps That Reflect The Territory and Team Seek Power For The Greater Good obviously do not belong in the same “movement” or “community.” It’s understandable that Team Power doesn’t want to be associated with Team Shared Maps because they’re afraid we’ll say things that will get them in trouble. (We totally will.) But for their part of the bargain, Team Power needs to not fraudulently market their beacon as “the rationality community” and thereby confuse innocents who came looking for shared maps.
I think of my team as being “Team Shared Maps That Reflect The Territory But With a Few Blank Spots, Subject to Cautious Private Discussion, Where Depicting the Territory Would Have Caused the Maps to be Burned”. I don’t think calling it “Team Seek Power For The Greater Good” is a fair characterization both because the Team is scrupulous not to draw fake stuff on the map and because the Team does not seek power for itself but rather seeks for it to be possible for true ideas to have influence regardless of what persons are associated with the true ideas.
That’s fair. Maybe our crux is about to what extent “don’t draw fake stuff on the map” is a actually a serious constraint? When standing trial for a crime you didn’t commit, it’s not exactly comforting to be told that the prosecutor never lies, but “merely” reveals Shared Maps That Reflect The Territory But With a Few Blank Spots Where Depicting the Territory Would Have Caused the Defendant to Be Acquitted. It’s good that the prosecutor never lies! But it’s important that the prosecutor is known as the prosecutor, rather than claiming to be the judge. Same thing with a so-called “rationalist” community.
I don’t think anyone understands the phrase “rationalist community” as implying a claim that its members don’t sometimes allow practical considerations to affect which topics they remain silent on. I don’t advocate that people leave out good points merely for being inconvenient to the case they’re making, optimizing for the audience to believe some claim regardless of the truth of that claim, as suggested by the prosecutor analogy. I advocate that people leave out good points for being relatively unimportant and predictably causing (part of) the audience to be harmfully irrational. I.e., if you saw someone else than the defendant commit the murder, then say that, but don’t start talking about how ugly the judge’s children are even if you think the ugliness of the judge’s children slightly helped inspire the real murderer. We can disagree about which discussions are more like talking about whether you saw someone else commit the murder and which discussions are more like talking about how ugly the judge’s children are.
I guess I feel like we’re at an event for the physics institute and someone’s being nerdy/awkward in the corner, and there’s a question of whether or not we should let that person be or whether we should publicly tell them off / kick them out. I feel like the best people there are a bit nerdy and overly analytical, and that’s fine, and deciding to publicly tell them off is over the top and will make all the physicists more uptight and self-aware.
To pick a very concrete problem we’ve worked on: the AI alignment problem is totally taken seriously by very important people who are also aware that LW is weird, but Eliezer goes on the Sam Harris podcast and Bostrom is invited by the UK government to advise and so on and Karnofsky’s got a billion dollars and focusing to a large part on the AI problem. We’re not being defined by this odd stuff, and I think we don’t need to feel like we are. I expect as we find similar concrete problems or proposals, we’ll continue to be taken very seriously and have major success.
As I see it, we’ve had this success partly because many of us have been scrupulous about not being needlessly offensive. (Bostrom is a good example here.) The rationalist brand is already weak (e.g. search Twitter for relevant terms), and if LessWrong had actually tried to have forthright discussions of every interesting topic, that might well have been fatal.
Many of the world’s smartest, most competent, and most influential people are ideologues. This probably includes whoever ends up developing and controlling advanced technologies. It would be nice to be able to avoid such people dismissing our ideas out of hand. You may not find them impressive or expect them to make intellectual progress on rationality, but for such progress to matter, the ideas have to be taken seriously outside LW at some point. I guess I don’t understand the case against caution in this area, so long as the cost is only having to avoid some peripheral topics instead of adopting or promoting false beliefs.
Rather than debating the case for or against caution, I think the most interesting question is how to arrange a peaceful schism. Team Shared Maps That Reflect The Territory and Team Seek Power For The Greater Good obviously do not belong in the same “movement” or “community.” It’s understandable that Team Power doesn’t want to be associated with Team Shared Maps because they’re afraid we’ll say things that will get them in trouble. (We totally will.) But for their part of the bargain, Team Power needs to not fraudulently market their beacon as “the rationality community” and thereby confuse innocents who came looking for shared maps.
I think of my team as being “Team Shared Maps That Reflect The Territory But With a Few Blank Spots, Subject to Cautious Private Discussion, Where Depicting the Territory Would Have Caused the Maps to be Burned”. I don’t think calling it “Team Seek Power For The Greater Good” is a fair characterization both because the Team is scrupulous not to draw fake stuff on the map and because the Team does not seek power for itself but rather seeks for it to be possible for true ideas to have influence regardless of what persons are associated with the true ideas.
That’s fair. Maybe our crux is about to what extent “don’t draw fake stuff on the map” is a actually a serious constraint? When standing trial for a crime you didn’t commit, it’s not exactly comforting to be told that the prosecutor never lies, but “merely” reveals Shared Maps That Reflect The Territory But With a Few Blank Spots Where Depicting the Territory Would Have Caused the Defendant to Be Acquitted. It’s good that the prosecutor never lies! But it’s important that the prosecutor is known as the prosecutor, rather than claiming to be the judge. Same thing with a so-called “rationalist” community.
I don’t think anyone understands the phrase “rationalist community” as implying a claim that its members don’t sometimes allow practical considerations to affect which topics they remain silent on. I don’t advocate that people leave out good points merely for being inconvenient to the case they’re making, optimizing for the audience to believe some claim regardless of the truth of that claim, as suggested by the prosecutor analogy. I advocate that people leave out good points for being relatively unimportant and predictably causing (part of) the audience to be harmfully irrational. I.e., if you saw someone else than the defendant commit the murder, then say that, but don’t start talking about how ugly the judge’s children are even if you think the ugliness of the judge’s children slightly helped inspire the real murderer. We can disagree about which discussions are more like talking about whether you saw someone else commit the murder and which discussions are more like talking about how ugly the judge’s children are.
I guess I feel like we’re at an event for the physics institute and someone’s being nerdy/awkward in the corner, and there’s a question of whether or not we should let that person be or whether we should publicly tell them off / kick them out. I feel like the best people there are a bit nerdy and overly analytical, and that’s fine, and deciding to publicly tell them off is over the top and will make all the physicists more uptight and self-aware.
To pick a very concrete problem we’ve worked on: the AI alignment problem is totally taken seriously by very important people who are also aware that LW is weird, but Eliezer goes on the Sam Harris podcast and Bostrom is invited by the UK government to advise and so on and Karnofsky’s got a billion dollars and focusing to a large part on the AI problem. We’re not being defined by this odd stuff, and I think we don’t need to feel like we are. I expect as we find similar concrete problems or proposals, we’ll continue to be taken very seriously and have major success.
As I see it, we’ve had this success partly because many of us have been scrupulous about not being needlessly offensive. (Bostrom is a good example here.) The rationalist brand is already weak (e.g. search Twitter for relevant terms), and if LessWrong had actually tried to have forthright discussions of every interesting topic, that might well have been fatal.