I think the basic problem here is an undissolved question: what is ‘intelligence’? Humans, being human, tend to imagine a superintelligence as a highly augmented human intelligence, so the natural assumption is that regardless of the ‘level’ of intelligence, skills will cluster roughly the way they do in human minds, i.e. having the ability to take over the world implies a high posterior probability of having the ability to understand human goals.
The problem with this assumption is that mind-design space is large (<--understatement), and the prior probability of a superintelligence randomly ending up with ability clusters analogous to human ability clusters is infinitesimal. Granted, the probability of this happening given a superintelligence designed by humans is significantly higher, but still not very high. (I don’t actually have enough technical knowledge to estimate this precisely, but just by eyeballing it I’d put it under 5%.)
In fact, autistic people are an example of non-human-standard ability clusters, and even that’s only by a tiny amount in the scale of mind-design-space.
As for an elevator pitch of this concept, something like “just because evolution happened design our brains to be really good at modeling human goal systems, doesn’t mean all intelligences are good at it, regardless of how good they might be at destroying the planet”.
the prior probability of a superintelligence randomly ending up with ability clusters analogous to human ability clusters is infinitesimal. Granted, the probability of this happening given a superintelligence designed by humans is significantly higher, but still not very high. (I don’t actually have enough technical knowledge to estimate this precisely, but just by eyeballing it I’d put it under 5%.)
Possibly the question is to what extent is human intelligence a bunch of hardcoded domain-specific algorithms as opposed to universal intelligence. I would have thought that understanding human goals might not be very different from other AI problems. Build a really powerful inference system, and if you feed it a training set of cars driving, it learns to drive, feed it data of human behaviour, and it learns to predict human behaviour, and probably to understand goals. Now its possible that the amount of general intelligence needed to develop advanced nanotech is less then the intelligence needed to understand human goals and the only reason why this seems counter intuitive is because evolution has optimised our brains for social cognition, but this does not seem obviously true to me.
I think the basic problem here is an undissolved question: what is ‘intelligence’? Humans, being human, tend to imagine a superintelligence as a highly augmented human intelligence, so the natural assumption is that regardless of the ‘level’ of intelligence, skills will cluster roughly the way they do in human minds, i.e. having the ability to take over the world implies a high posterior probability of having the ability to understand human goals.
The problem with this assumption is that mind-design space is large (<--understatement), and the prior probability of a superintelligence randomly ending up with ability clusters analogous to human ability clusters is infinitesimal. Granted, the probability of this happening given a superintelligence designed by humans is significantly higher, but still not very high. (I don’t actually have enough technical knowledge to estimate this precisely, but just by eyeballing it I’d put it under 5%.)
In fact, autistic people are an example of non-human-standard ability clusters, and even that’s only by a tiny amount in the scale of mind-design-space.
As for an elevator pitch of this concept, something like “just because evolution happened design our brains to be really good at modeling human goal systems, doesn’t mean all intelligences are good at it, regardless of how good they might be at destroying the planet”.
What is this process of random design? Actual Ai design is done by humans trying to emulate human abilities.
Possibly the question is to what extent is human intelligence a bunch of hardcoded domain-specific algorithms as opposed to universal intelligence. I would have thought that understanding human goals might not be very different from other AI problems. Build a really powerful inference system, and if you feed it a training set of cars driving, it learns to drive, feed it data of human behaviour, and it learns to predict human behaviour, and probably to understand goals. Now its possible that the amount of general intelligence needed to develop advanced nanotech is less then the intelligence needed to understand human goals and the only reason why this seems counter intuitive is because evolution has optimised our brains for social cognition, but this does not seem obviously true to me.