In any case, it’s a story, not a prediction, and I’d defend it as plausible in that context. Any story has a thousand assumptions and events that, in sequence, reduce the probability to infinitesimal.
Yeah, I don’t actually disagree. It’s just that, if someone asks “how could an AI actually be dangerous? It’s just on a computer” and I respond by “here look at this cool story someone wrote which answers that question”, they might go “Aha, you think it will be developed on a laptop. This is clearly nonsense, therefore I now dismiss your case entirely”. I think you wanna bend over backwards to not make misleading statements if you argue for the dangers-from-ai-is-a-real-thing side.
You’re of course correct that any scenario with this level of detail is necessarily extremely unlikely, but I think that will be more obvious for other details like how exactly the AI reasons than it is for the above. I don’t see anyone going “aha, the AI reasoned that X→Y→Z which is clearly implausible because it’s specific, therefore I won’t take this seriously”.
If I had written this, I would add a disclaimer rather than change the title. The disclaimer could also explain that “paperclips” is a stand-in for any utility function that maximizes for just a particular physical thing.
Yeah, I don’t actually disagree. It’s just that, if someone asks “how could an AI actually be dangerous? It’s just on a computer” and I respond by “here look at this cool story someone wrote which answers that question”, they might go “Aha, you think it will be developed on a laptop. This is clearly nonsense, therefore I now dismiss your case entirely”. I think you wanna bend over backwards to not make misleading statements if you argue for the dangers-from-ai-is-a-real-thing side.
You’re of course correct that any scenario with this level of detail is necessarily extremely unlikely, but I think that will be more obvious for other details like how exactly the AI reasons than it is for the above. I don’t see anyone going “aha, the AI reasoned that X→Y→Z which is clearly implausible because it’s specific, therefore I won’t take this seriously”.
If I had written this, I would add a disclaimer rather than change the title. The disclaimer could also explain that “paperclips” is a stand-in for any utility function that maximizes for just a particular physical thing.
That’s a good point, I’ll write up a brief explanation/disclaimer and put it in as a footnote.