Turntrout and JDP had an important insight in the discord, which I want to talk about: A lot of AI doom content is fundamentally written like good fanfic, and a major influx of people concerned about AI doom came from HPMOR and Friendship is Optimal. More generally, ratfic is basically the foundation of a lot of AI doom content, and how people believe in AI is going to kill us all, and while I’ll give it credit for being more coherent and generally exploring things that the original fic doesn’t, there is no reason for the amount of credence given to a lot of the assumptions in AI doom, especially once we realize that a lot of them probably come from fanfiction stories, not reality.
This is an important point, because it explains why there’s so many epistemic flaws in a lot of LW content on AI doom, especially around deceptive alignment: They’re fundamentally writing fanfiction, and forgot that there is basically no-little connection between how a fictional story plays out on AI and how our real outcomes of AI safety will turn out.
I think the most important implication of this belief is that it’s fundamentally okay to hold the view that classic AI risk almost certainly doesn’t exist, and importantly I think this is why I’m so confident in my predictions, since the AI doom thesis is held up by essentially fictional stories, which is not any guide to reality at all.
Yann Lecun once said that a lot of AI doom scenarios are essentially science fiction, and this is non-trivially right, once we realize who is preaching it and how they came to believe it, I suspect the majority came from HPMOR and FiO fanfics. More generally, I think it’s a red flag that how LW came into existence was basically through fanfiction, and while people like John Wentworth and Chris Olah/Neel Nanda are thankfully not nearly as reliant on fanfiction as a lot of LWers are, they are still a minority (though thankfully improving).
This is not intended to serve as a replacement for either my object level cases against doom, or anyone else’s case, but instead as a unifying explanation of why so much LW content on AI is essentially worthless, as they rely on ratfic far too much.
Since many AI doom scenarios sound like science fiction, let me ask this:
Could the SkyNet take-over in Terminator have happened if SkyNet had been open source?
To answer the question, the answer is maybe??? It very much depends on the details, here.
I find issues with the current way of talking about AI and existential risk.
My high level summary is that the question of AI doom is a really good meme, an interesting and compelling fictional story. It contains high stakes (end of the world), it contains good and evil (the ones for and against) and it contains magic (super intelligence). We have a hard time resisting this narrative because it contains these classic elements of an interesting story.
More generally, ratfic is basically the foundation of a lot of AI doom content, and how people believe in AI is going to kill us all, and while I’ll give it credit for being more coherent and generally exploring things that the original fic doesn’t, there is no reason for the amount of credence given to a lot of the assumptions in AI doom, especially once we realize that a lot of them probably come from fanfiction stories, not reality.
Noting for the record that this seems pretty clearly false to me.
I may weaken this, but my point is that a lot of people in LW probably came here through HPMOR and FiO, and with the ability for anyone to write a post and it getting karma, I think it’s likely that people who came through that route and had basically no structure akin to science to guide them away from unpromising paths likely allowed for low standards of discussion to be created.
I do buy that your social circle isn’t relying on fanfiction for your research. I am worried that a lot of the people on LW, especially the non-experts are implicitly relying on ratfic or science-fiction models as reasons to be worried on AI.
Turntrout and JDP had an important insight in the discord, which I want to talk about: A lot of AI doom content is fundamentally written like good fanfic, and a major influx of people concerned about AI doom came from HPMOR and Friendship is Optimal. More generally, ratfic is basically the foundation of a lot of AI doom content, and how people believe in AI is going to kill us all, and while I’ll give it credit for being more coherent and generally exploring things that the original fic doesn’t, there is no reason for the amount of credence given to a lot of the assumptions in AI doom, especially once we realize that a lot of them probably come from fanfiction stories, not reality.
This is an important point, because it explains why there’s so many epistemic flaws in a lot of LW content on AI doom, especially around deceptive alignment: They’re fundamentally writing fanfiction, and forgot that there is basically no-little connection between how a fictional story plays out on AI and how our real outcomes of AI safety will turn out.
I think the most important implication of this belief is that it’s fundamentally okay to hold the view that classic AI risk almost certainly doesn’t exist, and importantly I think this is why I’m so confident in my predictions, since the AI doom thesis is held up by essentially fictional stories, which is not any guide to reality at all.
Yann Lecun once said that a lot of AI doom scenarios are essentially science fiction, and this is non-trivially right, once we realize who is preaching it and how they came to believe it, I suspect the majority came from HPMOR and FiO fanfics. More generally, I think it’s a red flag that how LW came into existence was basically through fanfiction, and while people like John Wentworth and Chris Olah/Neel Nanda are thankfully not nearly as reliant on fanfiction as a lot of LWers are, they are still a minority (though thankfully improving).
This is not intended to serve as a replacement for either my object level cases against doom, or anyone else’s case, but instead as a unifying explanation of why so much LW content on AI is essentially worthless, as they rely on ratfic far too much.
https://twitter.com/ylecun/status/1718743423404908545
To answer the question, the answer is maybe??? It very much depends on the details, here.
https://twitter.com/ArYoMo/status/1693221455180288151
Noting for the record that this seems pretty clearly false to me.
I may weaken this, but my point is that a lot of people in LW probably came here through HPMOR and FiO, and with the ability for anyone to write a post and it getting karma, I think it’s likely that people who came through that route and had basically no structure akin to science to guide them away from unpromising paths likely allowed for low standards of discussion to be created.
I do buy that your social circle isn’t relying on fanfiction for your research. I am worried that a lot of the people on LW, especially the non-experts are implicitly relying on ratfic or science-fiction models as reasons to be worried on AI.
I have specifically committed not to read HPMOR for this reason, and do not read much fiction in general, as a datapoint from a “doomer”.
I’m okay with that, but I wasn’t wanting to have that drastic of an effect on people. I more wanted to point out something that is overlooked.