the more seriously you took it, the more horrifying it became.
Eh. Up to a point. And then if you take it more seriously than that, it becomes less horrifying again.
Arguments for why it’s scary are the decision theory equivalent to someone describing how scary knives are, and how to make your own sharp knives, but never mentioning any knife safety tips.
“Sharp knives,” in this metaphor, is the recognition that other people might try to manipulate us, and the decision theory of why they’d do it and how it would work. “Knife safety” is our own ability to use decision theory to not get manipulated.
The reason that I think Roko’s basilisk is a net harmful idea is because there are a lot of people who are way more motivated to learn/talk about “cool” topics like sharp scary knives or ideas that sound dangerous, who are not nearly as motivated to learn about “boring” topics like knife safety or mathy decision theory. So for people who happen to allocate their attention this way (maybe you’re an edgy young adult, or maybe you’re just a naturally anxious person, or maybe you’re an edgy anxious person), it might just make them more anxious or otherwise mislead them.
Eh. Up to a point. And then if you take it more seriously than that, it becomes less horrifying again.
Arguments for why it’s scary are the decision theory equivalent to someone describing how scary knives are, and how to make your own sharp knives, but never mentioning any knife safety tips.
“Sharp knives,” in this metaphor, is the recognition that other people might try to manipulate us, and the decision theory of why they’d do it and how it would work. “Knife safety” is our own ability to use decision theory to not get manipulated.
The reason that I think Roko’s basilisk is a net harmful idea is because there are a lot of people who are way more motivated to learn/talk about “cool” topics like sharp scary knives or ideas that sound dangerous, who are not nearly as motivated to learn about “boring” topics like knife safety or mathy decision theory. So for people who happen to allocate their attention this way (maybe you’re an edgy young adult, or maybe you’re just a naturally anxious person, or maybe you’re an edgy anxious person), it might just make them more anxious or otherwise mislead them.