It’s probably worth noting that Yudkowsky did not really make the argument for AI risk in his article. He says that AI will literally kill everyone on Earth, and he gives an example of how it might do so, but he doesn’t present a compelling argument for why it would.[0] He does not even mention orthogonality or instrumental convergence. I find it hard to blame these various internet figures who were unconvinced about AI risk upon reading the article.
[0] He does quote “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”
The way I took it the article was meant to bring people to the table regarding AI risk so there was a tradeoff between keeping the message simple and clear and relaying the best arguments. Even though orthogonality and instrumental convergence are important theories, in this context he probably didn’t want to risk the average reader being put off by technical sounding jargon and losing interest. There could be an entire website in a similar vein to LessWrong about conveying difficult messages to a culture not attuned to the technical aspects involved.
It’s probably worth noting that Yudkowsky did not really make the argument for AI risk in his article. He says that AI will literally kill everyone on Earth, and he gives an example of how it might do so, but he doesn’t present a compelling argument for why it would.[0] He does not even mention orthogonality or instrumental convergence. I find it hard to blame these various internet figures who were unconvinced about AI risk upon reading the article.
[0] He does quote “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”
The way I took it the article was meant to bring people to the table regarding AI risk so there was a tradeoff between keeping the message simple and clear and relaying the best arguments. Even though orthogonality and instrumental convergence are important theories, in this context he probably didn’t want to risk the average reader being put off by technical sounding jargon and losing interest. There could be an entire website in a similar vein to LessWrong about conveying difficult messages to a culture not attuned to the technical aspects involved.