I respond with arguments like, “In the one real example of intelligence being developed we have to look at, continuous application of natural selection in fact found Homo sapiens sapiens, and the capability-gain curves of the ecosystem for various measurables were in fact sharply kinked by this new species (e.g., using machines, we sharply outperform other animals on well-established metrics such as “airspeed”, “altitude”, and “cargo carrying capacity”).”
Their response in turn is generally some variant of “well, natural selection wasn’t optimizing very intelligently” or “maybe humans weren’t all that sharply above evolutionary trends” or “maybe the power that let humans beat the rest of the ecosystem was simply the invention of culture, and nothing embedded in our own already-existing culture can beat us” or suchlike.
Rather than arguing further here, I’ll just say that failing to believe the hard problem exists is one surefire way to avoid tackling it.
It sounds like you don’t want to argue this point further here, but I would like to point something very simple out that I think your argument here glosses over.
Humanity is a species, not an individual. It wasn’t the case that a single animal arose among all the others, and out-competed everyone else. Instead, it was a large set of entities that collectively out-competed all the other animals. And I think this distinction is quite important to make.
If you think that an analogy to human evolution is critical to understanding our epistemic situation, it appears to me that the evolutionary analogy should force you to draw the opposite conclusion from the one you have drawn here (relative to credible people who disagree).
In my understanding of our situation, the conclusion to draw from human evolution is that a single species can acquire a host of very powerful technologies, and tower above everyone else, in a relatively short period of time. That is, we should predict that, in the future, a collection of AIs could eventually out-match humanity.
But you’re not arguing that thesis! (At least, as I understand your argument) You’re arguing that the evolutionary analogy shows that a single individual can outcompete everyone else. And I don’t know where that idea is coming from.
It sounds like you don’t want to argue this point further here, but I would like to point something very simple out that I think your argument here glosses over.
Humanity is a species, not an individual. It wasn’t the case that a single animal arose among all the others, and out-competed everyone else. Instead, it was a large set of entities that collectively out-competed all the other animals. And I think this distinction is quite important to make.
If you think that an analogy to human evolution is critical to understanding our epistemic situation, it appears to me that the evolutionary analogy should force you to draw the opposite conclusion from the one you have drawn here (relative to credible people who disagree).
In my understanding of our situation, the conclusion to draw from human evolution is that a single species can acquire a host of very powerful technologies, and tower above everyone else, in a relatively short period of time. That is, we should predict that, in the future, a collection of AIs could eventually out-match humanity.
But you’re not arguing that thesis! (At least, as I understand your argument) You’re arguing that the evolutionary analogy shows that a single individual can outcompete everyone else. And I don’t know where that idea is coming from.