My sense is that explicit technological modifications of humans are already heavily concerned with the question “Will we still be human after this modification” which at least gestures at the problems you identify. It is exact the lack of this type of concern in the AI field that motivates SIAI’s Friendliness activism. But the sorts of technological advances you are pointing towards seem more likely to arise in part from medical researcher methodologies, which seem more concerned with potential negative psychological and sociological effects than some other forms of technological research.
In short, if every AI researcher was already worried about safety to the extent that medical researchers seem to be worried, then there would be no need for SIAI to exist—all AI researchers worrying about the Friendliness problem is what winning looks like for SIAI. Since medical researchers are already worried about these types of problems, an SIAI-equivalent is not necessary. Consider all the different medical ethics councils—which are much more powerful than their institutional equivalents in AI research.
My sense is that explicit technological modifications of humans are already heavily concerned with the question “Will we still be human after this modification” which at least gestures at the problems you identify. It is exact the lack of this type of concern in the AI field that motivates SIAI’s Friendliness activism. But the sorts of technological advances you are pointing towards seem more likely to arise in part from medical researcher methodologies, which seem more concerned with potential negative psychological and sociological effects than some other forms of technological research.
In short, if every AI researcher was already worried about safety to the extent that medical researchers seem to be worried, then there would be no need for SIAI to exist—all AI researchers worrying about the Friendliness problem is what winning looks like for SIAI. Since medical researchers are already worried about these types of problems, an SIAI-equivalent is not necessary. Consider all the different medical ethics councils—which are much more powerful than their institutional equivalents in AI research.