Meanwhile Rationality A-Z is just super long. I think anyone who’s a longterm member of LessWrong or the alignment community should read the whole thing sooner or later – it covers a lot of different subtle errors and philosophical confusions that are likely to come up (both in AI alignment and in other difficult challenges)
My current guess is that the meme “every alignment person needs to read the Sequences / Rationality A-Z” is net harmful. They seem to have been valuable for some people but I think many people can contribute to reducing AI x-risk without reading them. I think the current AI risk community overrates them because they are selected strongly to have liked them.
Some anecodtal evidence in favor of my view:
To the extent you think I’m promising for reducing AI x-risk and have good epistemics, I haven’t read most of the Sequences. (I have liked some of Eliezer’s other writing, like Intelligence Explosion Microeconomics.)
I’ve been moving some of my most talented friends toward work on reducing AI x-risk and similarly have found that while I think all have great epistemics, there’s mixed reception to rationalist-style writing. e.g. one is trialing at a top alignment org and doesn’t like HPMOR, while another likes HPMOR, ACX, etc.
but I think many people can contribute to reducing AI x-risk without reading them.
I think the tough thing here is it’s very hard to evaluate who, if anyone, is making any useful contributions. After all, no one has successfully aligned a superintelligence to date. Maybe it’s all way-off track. All else equal, I trust people who’ve read the Sequences to be better judges of whether we’re making progress in the absence of proper end-to-end feedback than those who haven’t.
Caveat: I am not someone who could plausibly claim to have made any potential contribution myself. :P
I think it’s plausible that it is either harmful to perpetuate “every alignment person needs to read the Sequences / Rationality A-Z” or maybe even inefficient.
For example, to the extent that alignment needs more really good machine learning engineers, it’s possible they might benefit less from the sequences than a conceptual alignment researcher.
However, relying on anecdotal evidence seems potentially unnecessary. We might be able to use polls, or otherwise systemically investigate the relationship between interest/engagement with the sequences and various paths to contribution with AI. A prediction market might also work for information aggregation.
I’d bet that all else equal, engagement with the sequences is beneficial but that this might be less pronounced among those growing up in academically inclined cultures.
My current guess is that the meme “every alignment person needs to read the Sequences / Rationality A-Z” is net harmful. They seem to have been valuable for some people but I think many people can contribute to reducing AI x-risk without reading them. I think the current AI risk community overrates them because they are selected strongly to have liked them.
Some anecodtal evidence in favor of my view:
To the extent you think I’m promising for reducing AI x-risk and have good epistemics, I haven’t read most of the Sequences. (I have liked some of Eliezer’s other writing, like Intelligence Explosion Microeconomics.)
I’ve been moving some of my most talented friends toward work on reducing AI x-risk and similarly have found that while I think all have great epistemics, there’s mixed reception to rationalist-style writing. e.g. one is trialing at a top alignment org and doesn’t like HPMOR, while another likes HPMOR, ACX, etc.
I think the tough thing here is it’s very hard to evaluate who, if anyone, is making any useful contributions. After all, no one has successfully aligned a superintelligence to date. Maybe it’s all way-off track. All else equal, I trust people who’ve read the Sequences to be better judges of whether we’re making progress in the absence of proper end-to-end feedback than those who haven’t.
Caveat: I am not someone who could plausibly claim to have made any potential contribution myself. :P
I think it’s plausible that it is either harmful to perpetuate “every alignment person needs to read the Sequences / Rationality A-Z” or maybe even inefficient.
For example, to the extent that alignment needs more really good machine learning engineers, it’s possible they might benefit less from the sequences than a conceptual alignment researcher.
However, relying on anecdotal evidence seems potentially unnecessary. We might be able to use polls, or otherwise systemically investigate the relationship between interest/engagement with the sequences and various paths to contribution with AI. A prediction market might also work for information aggregation.
I’d bet that all else equal, engagement with the sequences is beneficial but that this might be less pronounced among those growing up in academically inclined cultures.