That building an intellegent agent that qualifies as “ethical,” even of it is SUPER ethical, may not be the same thing as building an intelligent agent that is compatible with humans or their values.
More plainly stated, just because your AI has a self-consitent, justifiable ethics system, doesnt mean that it likes humans, or even cares about wiping them out.
Having an AI that is ethical isn’t enough. It has to actually care about humans and their values. Even if it has rules in place like not aggressing, attacking, or killing humans, it may still be able to cause humanity to go extinct indirectly.
That building an intellegent agent that qualifies as “ethical,” even of it is SUPER ethical, may not be the same thing as building an intelligent agent that is compatible with humans or their values.
More plainly stated, just because your AI has a self-consitent, justifiable ethics system, doesnt mean that it likes humans, or even cares about wiping them out.
Having an AI that is ethical isn’t enough. It has to actually care about humans and their values. Even if it has rules in place like not aggressing, attacking, or killing humans, it may still be able to cause humanity to go extinct indirectly.