There’s a lot going on in this essay, but the big point would appear to be: to create advanced AI is to materialize an Unknown Unknown, and why on earth would you expect that to be something you can even understand, let alone something that is sympathetic to you or “aligned” with you?
Then I made a PDF of the article and fed it to Claude Opus and to Google’s Gemini-powered NotebookLM, and both AIs seemed to get the gist immediately, as well as understanding the article’s detailed structure. There is a deep irony in hearing NotebookLM’s pod-people restating the essay’s points in their own words, and agreeing that its warnings make sense.
There’s a lot going on in this essay, but the big point would appear to be: to create advanced AI is to materialize an Unknown Unknown, and why on earth would you expect that to be something you can even understand, let alone something that is sympathetic to you or “aligned” with you?
Then I made a PDF of the article and fed it to Claude Opus and to Google’s Gemini-powered NotebookLM, and both AIs seemed to get the gist immediately, as well as understanding the article’s detailed structure. There is a deep irony in hearing NotebookLM’s pod-people restating the essay’s points in their own words, and agreeing that its warnings make sense.