Humans are capable of winning at chess without the terminal goal of doing so. Nor were humans designed by evolution specifically for chess. Why should we expect a general superintelligence to have intelligence than generalizes less easily than a human’s does?
Biological evolution is not the full picture here. Humans were programmed to be capable of winning at chess, and to care to do so, by cultural evolution, education, and environmental feedback in the form of incentives given by other people challenging them to play.
I don’t know how this works. But I do not dispute the danger of neuromorphic AIs, as you know from a comment elsewhere.
Do you suggest that from the expected behavior of neuromorphic AIs it is possible to draw conclusions about the behavior of what you call a ‘seed AI’? Would such a seed AI, as would be the case with neuromorphic AIs, be constantly programmed by environmental feedback?
You keep coming back to this ‘logically incoherent goals’ and ‘vague goals’ idea. Honestly, I don’t have the slightest idea what you mean by those things.
What I mean is that if you program a perfect scientist but give this perfect scientist a hypothesis that does not make any predictions, then it will not be able to unfold its power.
Conditioned on a self-modifying AGI...the hypothetical we’re trying to assess.
I believe that I already wrote that I do not dispute that the idea you seem to have in mind is a risk by definition. If such an AI is likely, then we are likely going extinct if we fail at making it care about human values.
You haven’t given arguments suggesting that here.
I feel uncomfortable to say this, but I do not see that the burden of proof is on me to show that it takes deliberate and intentional effort to make an AI exhibit those drives, as long that is not part of your very definition. I find the current argument in favor of AI drives to be thoroughly unconvincing.
Be careful to note, to yourself and others, when you switch between the claims ‘a superintelligence is too hard to make’ and ‘if we made a superintelligence it would probably be safe’.
The former has always been one of the arguments in favor of the latter in the posts I wrote on my blog.
Biological evolution is not the full picture here. Humans were programmed to be capable of winning at chess, and to care to do so, by cultural evolution, education, and environmental feedback in the form of incentives given by other people challenging them to play.
I don’t know how this works. But I do not dispute the danger of neuromorphic AIs, as you know from a comment elsewhere.
Do you suggest that from the expected behavior of neuromorphic AIs it is possible to draw conclusions about the behavior of what you call a ‘seed AI’? Would such a seed AI, as would be the case with neuromorphic AIs, be constantly programmed by environmental feedback?
What I mean is that if you program a perfect scientist but give this perfect scientist a hypothesis that does not make any predictions, then it will not be able to unfold its power.
I believe that I already wrote that I do not dispute that the idea you seem to have in mind is a risk by definition. If such an AI is likely, then we are likely going extinct if we fail at making it care about human values.
I feel uncomfortable to say this, but I do not see that the burden of proof is on me to show that it takes deliberate and intentional effort to make an AI exhibit those drives, as long that is not part of your very definition. I find the current argument in favor of AI drives to be thoroughly unconvincing.
The former has always been one of the arguments in favor of the latter in the posts I wrote on my blog.