Wouldn’t it be surprising that the intelligence threshold for building UFAI and FAI turn out to be the same?
Probably not, since you use knowledge to build things, not just intelligence, and knowledge accumulates. For example, humans can build (invent) both windmills and space shuttles, and lower intelligence would probably bar humans from building (inventing) windmills.
Again, that whole section makes more sense if we assume a limited time horizon. But I’m curious: you don’t think it’s likely that there is some distribution of intelligence at which humanity would have invented windmills but then stagnated before reaching space shuttles? It seems to me that building windmills require only some physical/mechanical intuitions plus trial and error, whereas space shuttles need higher mathematics and computers, which can’t be understood (much less invented) below a certain IQ.
I wrote that comment before reading the clarification about there being an intended context of a time horizon. I agree that it’d take longer to develop FAI than to develop an UFAI, all else equal. (Of course, all else is not equal, and you did write about intelligence threshold and not time.)
I think there is significant speedup in the time it took us to build space shuttles gained through outliers in intelligence, compared to the time it would take to do that for a species of uniform ability that was only barely smart enough to invent windmills, but I expect they’d get there, barring defeaters (to use Chalmers’ term), if they manage to stumble upon a science-generating social institution, and avoid collapse of their civilization for long enough to accumulate the necessary knowledge. People that are not that smart can do science, and learn science, as they can learn language or any other skill, it just takes them much longer to learn it or find useful abstract insights, and each individual can master less. We (as a civilization) don’t use people of low ability for this purpose only because there is a much more efficient alternative.
Probably not, since you use knowledge to build things, not just intelligence, and knowledge accumulates. For example, humans can build (invent) both windmills and space shuttles, and lower intelligence would probably bar humans from building (inventing) windmills.
Again, that whole section makes more sense if we assume a limited time horizon. But I’m curious: you don’t think it’s likely that there is some distribution of intelligence at which humanity would have invented windmills but then stagnated before reaching space shuttles? It seems to me that building windmills require only some physical/mechanical intuitions plus trial and error, whereas space shuttles need higher mathematics and computers, which can’t be understood (much less invented) below a certain IQ.
I wrote that comment before reading the clarification about there being an intended context of a time horizon. I agree that it’d take longer to develop FAI than to develop an UFAI, all else equal. (Of course, all else is not equal, and you did write about intelligence threshold and not time.)
I think there is significant speedup in the time it took us to build space shuttles gained through outliers in intelligence, compared to the time it would take to do that for a species of uniform ability that was only barely smart enough to invent windmills, but I expect they’d get there, barring defeaters (to use Chalmers’ term), if they manage to stumble upon a science-generating social institution, and avoid collapse of their civilization for long enough to accumulate the necessary knowledge. People that are not that smart can do science, and learn science, as they can learn language or any other skill, it just takes them much longer to learn it or find useful abstract insights, and each individual can master less. We (as a civilization) don’t use people of low ability for this purpose only because there is a much more efficient alternative.