Personally, most of my intuition here comes from looking at differences within the existing human distribution.
For instance, consider a medieval lord facing a technologist making gunpowder. That’s not even a large tech lead, but it’s already enough that the less-knowledgeable human just has absolutely no idea what’s going on. Or, consider this example about some protesters vs a lobbyist. (Note that the first example is more about knowledge, the second more about “intelligence” in a sense closer to “AGI” than to IQ; I expect AGI to exceed top humans along both of those dimensions.)
Bear in mind that there’s a filter bubble here—people who go to college and then work in the sort of places where everyone has a degree typically hang out with ~zero people who are on the low end of human intelligence/knowledge/willpower. Every now and then there will be some survey that finds most people can’t solve simple problems of type X, and the people who don’t really hang out with anyone on the low end of the intelligence/knowledge/willpower curve are amazed that the average person manages to get by at all. And “college degree” isn’t even selecting people who are all that far on the high side of the curve. There’s a quote I’ve heard attributed to Murray Gell-Mann (although I can’t vouch for its authenticity); supposedly he said to a roomful of physics grad students “You are to the rest of the world as the rest of the world is to fish.”. And… yeah, that just seems basically true.
Personally, most of my intuition here comes from looking at differences within the existing human distribution.
For instance, consider a medieval lord facing a technologist making gunpowder. That’s not even a large tech lead, but it’s already enough that the less-knowledgeable human just has absolutely no idea what’s going on. Or, consider this example about some protesters vs a lobbyist. (Note that the first example is more about knowledge, the second more about “intelligence” in a sense closer to “AGI” than to IQ; I expect AGI to exceed top humans along both of those dimensions.)
Bear in mind that there’s a filter bubble here—people who go to college and then work in the sort of places where everyone has a degree typically hang out with ~zero people who are on the low end of human intelligence/knowledge/willpower. Every now and then there will be some survey that finds most people can’t solve simple problems of type X, and the people who don’t really hang out with anyone on the low end of the intelligence/knowledge/willpower curve are amazed that the average person manages to get by at all. And “college degree” isn’t even selecting people who are all that far on the high side of the curve. There’s a quote I’ve heard attributed to Murray Gell-Mann (although I can’t vouch for its authenticity); supposedly he said to a roomful of physics grad students “You are to the rest of the world as the rest of the world is to fish.”. And… yeah, that just seems basically true.