An AGI that’s complicit with the phasing out of humanity (presumably as humans merge with it, or an off-shoot of it, e.g., uploading), to the point that “not much would remian that’s recognizably human” would seem to be at odds with its coded imperative to remain “friendly.” At the very least, I think this concern highlights the trickiness of formalizing a definition for “friendliness,” which AFAIK anyone has yet to do.
An AGI that’s complicit with the phasing out of humanity (presumably as humans merge with it, or an off-shoot of it, e.g., uploading), to the point that “not much would remian that’s recognizably human” would seem to be at odds with its coded imperative to remain “friendly.” At the very least, I think this concern highlights the trickiness of formalizing a definition for “friendliness,” which AFAIK anyone has yet to do.