But the thing we’re interested in is instrumental rationality, not epistemic rationality.
Ironically, this sentence is epistemically true but instrumentally very dangerous.
See, to accurately assess which parts of epistemic rationality one should sacrifice for instrumental improvements requires a whole lot of epistemic rationality. And once you’ve made that sacrifice and lost some epistemic rationality, your capacity to make such trade-offs wisely in the future is severely impaired. But if you just focus on epistemic rationality, you can get quite a lot of winning as a side effect.
To bring it back to our example: it’s very dangerous to convince yourself that Jesus died for your sins just because you notice Christians have more friends. To do so you need to understand why believing in Jesus correlates with having friends. If you have a strong enough understanding of friendship and social structures for that, you can easily make friends and build a community without Jesus.
But if you install Jesus on your system you’re now left vulnerable to a lot of instrumentally bad things, with no guarantee that you’ll actually get the friends and community you wanted.
Assuming that the instrumental utility of religion can be separated from the religious parts is an old misconception. If all you need is a bit of sociological knowledge, shouldn’t it be possible to just engineer a cult of reason? Well, as it turns out, people have been trying for centuries, and it’s never really stuck. For one thing, there are, in startup terms, network effects. I’m not saying you should think of St. Paul as the Zuckerberg of Rome, but I’ve been to one of those churches where they dropped all the wacky supernatural stuff and I’d rather go to a meetup for GNU Social power users.
For another thing, it’s interesting that Eliezer Yudkowsky, who seems to be primarily interested in intellectual matters that relate to entities that are, while constrained by the rules of the universe, effectively all-knowing and all-powerful, and who cultivated interest in the mundane stuff out of the desire to get more people interested in said intellectual matters, seems to have gotten unusually far with the cult-of-reason project, at least so far.
Of course, if we think of LW as the seed of what could become a new religion (or at least a new philosophical scene, as world-spanning empires sometimes generate when they’re coming off a golden age—and didn’t Socrates have a thing or two to say about raising the sanity waterline?), this discussion would have to look a lot different, and ideally would be carried out in a smoke-filled room somewhere. You don’t want everyone in your society believing whatever nonsense will help them out with their social climbing, for reasons which I hope are obvious. (On the other hand, if we think of LW as the seed of what could become a new religion, its unusual antipathy to other religions—I haven’t seen anyone deploy the murder-Gandhi argument to explain why people shouldn’t do drugs or make tulpas—is an indisputable adaptive necessity. So there’s that.)
If, on the other hand, we think of LW as some people who are interested in instrumental rationality, the case has to be made that there’s at least fruit we can reach without becoming giraffes in grinding epistemic rationality. But most of us are shut-ins who read textbooks for fun, so how likely should we think it is that our keys are under the streetlight?
its unusual antipathy to other religions—I haven’t seen anyone deploy the murder-Gandhi argument to explain why people shouldn’t do drugs or make tulpas
The murder-Gandhi argument against drugs is so common it has a name, “addiction.” Rationalists appear to me to have a perfectly rational level of concern about addiction (which means being less concerned about certain drugs, such as MDMA, and more concerned about other drugs, such as alcohol).
I am puzzled about how making tulpas could interfere with one’s ability to decide not to make any more tulpas.
The only explanation I caught wind of for the parking lot incident was that it had something to do with tulpamancy gone wrong. And I recall SSC attributing irreversible mental effects to hallucinogens and noting that a lot of the early proponents of hallucinogens ended up somewhat wacky.
But maybe it really does all work out such that the sorts of things that are popular in upper-middle-class urban twenty-something circles just aren’t anything to worry about, and the sorts of things that are unpopular in them (or worse, popular elsewhere) just are. What a coincidence!
Is your goal to have a small community of friends or to take over the world? The tightest-knit religions are the smaller and weirder ones, so if you want stronger social bonds you should join Scientology and not the Catholic church.
Or, you know, you can just go to a LessWrong meetup. I’ve been to one yesterday: we had cake, and wine, and we did a double crux discussion about rationality and self-improvement. I dare say that we’re getting at least half as much community benefit as the average church-goer, all for a modest investment of effort and without sacrificing our sanity.
If someone doesn’t have a social life because don’t leave their house, they should leave their house. The religious shut-ins who read the Bible for fun aren’t getting much social benefit either.
One day I will have to write a longer text about this, but shortly: it is a false dilemma to see “small and tight-knit community” and “taking over the world” as mutually exclusive. Catholic church is not a small community, but it contains many small communities. It is an “eukaryotic” community, containing both the tight-knit subgroups and the masses of lukewarm believers, which together contribute to its long-term survival.
I would like to see the rationalist community to become “eukaryotic” in a similar way. In certain ways it already happens: we have people who work at MIRI and CFAR, we have people who participate at local meetups, we have people who debate online. This diversity is strength, not weakness: if you only have one mode of participation, then people who are unable to participate in that one specific way, are lost to the community.
The tricky part is keeping it all together. Preventing the tight-knit groups from excommunicating everyone else as “not real members”, but also preventing the lukewarm members from making it all about social interaction and abandoning the original purpose, because both of those are natural human tendencies.
I imagine this could be tricky to research even if people wouldn’t try to obfuscate the reality (which they of course will). It would be difficult to distinguish “these two people conspired together” from “they are two extremely smart people, living in the same city, of course they are likely to have met each other”.
For example, in a small country with maybe five elite high schools, elite people of the same age have high probability to have been high-school classmates. If they later take over the world together, it would make a good story to claim that they already conspired to do that during the high school. Even if the real idea only came 20 years later, no one would believe it after some journalist finds out that actually they are former classmates.
So the information is likely to be skewed in both ways: not seeing connections where they are, and seeing meaningful connections in mere coincidences.
Small groups have a bigger problem: they won’t be very well documented. As far as I know, the only major source on the Junto is Ben Franklin’s autobiography, which I’ve already read.
Large groups, of course, have an entirely different problem: if they get an appreciable amount of power, conspiracy theorists will probably find out, and put out reams of garbage on them. I haven’t started trying to look into the history of the Freemasons yet because I’m not sure about the difficulty of telling garbage from useful history.
Ironically, this sentence is epistemically true but instrumentally very dangerous.
See, to accurately assess which parts of epistemic rationality one should sacrifice for instrumental improvements requires a whole lot of epistemic rationality. And once you’ve made that sacrifice and lost some epistemic rationality, your capacity to make such trade-offs wisely in the future is severely impaired. But if you just focus on epistemic rationality, you can get quite a lot of winning as a side effect.
To bring it back to our example: it’s very dangerous to convince yourself that Jesus died for your sins just because you notice Christians have more friends. To do so you need to understand why believing in Jesus correlates with having friends. If you have a strong enough understanding of friendship and social structures for that, you can easily make friends and build a community without Jesus.
But if you install Jesus on your system you’re now left vulnerable to a lot of instrumentally bad things, with no guarantee that you’ll actually get the friends and community you wanted.
Assuming that the instrumental utility of religion can be separated from the religious parts is an old misconception. If all you need is a bit of sociological knowledge, shouldn’t it be possible to just engineer a cult of reason? Well, as it turns out, people have been trying for centuries, and it’s never really stuck. For one thing, there are, in startup terms, network effects. I’m not saying you should think of St. Paul as the Zuckerberg of Rome, but I’ve been to one of those churches where they dropped all the wacky supernatural stuff and I’d rather go to a meetup for GNU Social power users.
For another thing, it’s interesting that Eliezer Yudkowsky, who seems to be primarily interested in intellectual matters that relate to entities that are, while constrained by the rules of the universe, effectively all-knowing and all-powerful, and who cultivated interest in the mundane stuff out of the desire to get more people interested in said intellectual matters, seems to have gotten unusually far with the cult-of-reason project, at least so far.
Of course, if we think of LW as the seed of what could become a new religion (or at least a new philosophical scene, as world-spanning empires sometimes generate when they’re coming off a golden age—and didn’t Socrates have a thing or two to say about raising the sanity waterline?), this discussion would have to look a lot different, and ideally would be carried out in a smoke-filled room somewhere. You don’t want everyone in your society believing whatever nonsense will help them out with their social climbing, for reasons which I hope are obvious. (On the other hand, if we think of LW as the seed of what could become a new religion, its unusual antipathy to other religions—I haven’t seen anyone deploy the murder-Gandhi argument to explain why people shouldn’t do drugs or make tulpas—is an indisputable adaptive necessity. So there’s that.)
If, on the other hand, we think of LW as some people who are interested in instrumental rationality, the case has to be made that there’s at least fruit we can reach without becoming giraffes in grinding epistemic rationality. But most of us are shut-ins who read textbooks for fun, so how likely should we think it is that our keys are under the streetlight?
The murder-Gandhi argument against drugs is so common it has a name, “addiction.” Rationalists appear to me to have a perfectly rational level of concern about addiction (which means being less concerned about certain drugs, such as MDMA, and more concerned about other drugs, such as alcohol).
I am puzzled about how making tulpas could interfere with one’s ability to decide not to make any more tulpas.
The only explanation I caught wind of for the parking lot incident was that it had something to do with tulpamancy gone wrong. And I recall SSC attributing irreversible mental effects to hallucinogens and noting that a lot of the early proponents of hallucinogens ended up somewhat wacky.
But maybe it really does all work out such that the sorts of things that are popular in upper-middle-class urban twenty-something circles just aren’t anything to worry about, and the sorts of things that are unpopular in them (or worse, popular elsewhere) just are. What a coincidence!
Is your goal to have a small community of friends or to take over the world? The tightest-knit religions are the smaller and weirder ones, so if you want stronger social bonds you should join Scientology and not the Catholic church.
Or, you know, you can just go to a LessWrong meetup. I’ve been to one yesterday: we had cake, and wine, and we did a double crux discussion about rationality and self-improvement. I dare say that we’re getting at least half as much community benefit as the average church-goer, all for a modest investment of effort and without sacrificing our sanity.
If someone doesn’t have a social life because don’t leave their house, they should leave their house. The religious shut-ins who read the Bible for fun aren’t getting much social benefit either.
Rationality is a bad religion, but if you understand religions well enough you probably don’t need one.
One day I will have to write a longer text about this, but shortly: it is a false dilemma to see “small and tight-knit community” and “taking over the world” as mutually exclusive. Catholic church is not a small community, but it contains many small communities. It is an “eukaryotic” community, containing both the tight-knit subgroups and the masses of lukewarm believers, which together contribute to its long-term survival.
I would like to see the rationalist community to become “eukaryotic” in a similar way. In certain ways it already happens: we have people who work at MIRI and CFAR, we have people who participate at local meetups, we have people who debate online. This diversity is strength, not weakness: if you only have one mode of participation, then people who are unable to participate in that one specific way, are lost to the community.
The tricky part is keeping it all together. Preventing the tight-knit groups from excommunicating everyone else as “not real members”, but also preventing the lukewarm members from making it all about social interaction and abandoning the original purpose, because both of those are natural human tendencies.
One thing I’d like to see is more research into the effects of… if not secret societies, then at least societies of some sort.
For example, is it just a coincidence that Thiel and Musk, arguably the two most interesting public figures in the tech scene, are both Paypal Mafia?
Another good example is the Junto.
I imagine this could be tricky to research even if people wouldn’t try to obfuscate the reality (which they of course will). It would be difficult to distinguish “these two people conspired together” from “they are two extremely smart people, living in the same city, of course they are likely to have met each other”.
For example, in a small country with maybe five elite high schools, elite people of the same age have high probability to have been high-school classmates. If they later take over the world together, it would make a good story to claim that they already conspired to do that during the high school. Even if the real idea only came 20 years later, no one would believe it after some journalist finds out that actually they are former classmates.
So the information is likely to be skewed in both ways: not seeing connections where they are, and seeing meaningful connections in mere coincidences.
Small groups have a bigger problem: they won’t be very well documented. As far as I know, the only major source on the Junto is Ben Franklin’s autobiography, which I’ve already read.
Large groups, of course, have an entirely different problem: if they get an appreciable amount of power, conspiracy theorists will probably find out, and put out reams of garbage on them. I haven’t started trying to look into the history of the Freemasons yet because I’m not sure about the difficulty of telling garbage from useful history.