AGI that’s complicit with the phasing out of humanity [...] would seem to be at odds with its coded imperative to remain “friendly.”
With the CEV definition of Friendliness, it would be Friendly iff that’s what humans wanted (in the CEV technical sense). My vision includes that being what humans will want—if I’m wrong about that, a CEV-designed AI wouldn’t take us in that direction.
I think the problem of whether what would result would really be the descendants of humanity is directly analogous to the problem of personal identity—if the average atom in the human body has a half-life (of remaining in the body) of two weeks, how can we say we’re the same person over time? Evolving patterns. I don’t think we really understand either problem too well.
AGI that’s complicit with the phasing out of humanity [...] would seem to be at odds with its coded imperative to remain “friendly.”
With the CEV definition of Friendliness, it would be Friendly iff that’s what humans wanted (in the CEV technical sense). My vision includes that being what humans will want—if I’m wrong about that, a CEV-designed AI wouldn’t take us in that direction.
I think the problem of whether what would result would really be the descendants of humanity is directly analogous to the problem of personal identity—if the average atom in the human body has a half-life (of remaining in the body) of two weeks, how can we say we’re the same person over time? Evolving patterns. I don’t think we really understand either problem too well.