Sorry I took awhile to find this. I think the best short answer to this question is Scott Alexander’s Superintelligence FAQ.
The arguments involved here deal with a fair number of areas where human intuitions just aren’t very good (i.e. we’re used to dealing with linear systems, not exponential ones). So if Scott’s post isn’t persuasive to you, the next best answer I have is unfortunately “read the sequences.”
Sorry I took awhile to find this. I think the best short answer to this question is Scott Alexander’s Superintelligence FAQ.
The arguments involved here deal with a fair number of areas where human intuitions just aren’t very good (i.e. we’re used to dealing with linear systems, not exponential ones). So if Scott’s post isn’t persuasive to you, the next best answer I have is unfortunately “read the sequences.”