Well, if Prawn knew that they could just tell us and we would be convinced, ending this argument.
More generally … maybe some sort of social contract theory? It might be stable with enough roughly-equal agents, anyway. Prawn has said it would have to be deducible from the axioms of rationality, implying something that’s rational for (almost?) every goal.
Why would Clippy want to hit the top of the Kohlberg Hierarchy?
Well, if Prawn knew that they could just tell us
“The way people sometimes realise their values are wrong...only more efficiently, because its super intelligent. Well, I’ll concede that with care you might be able to design a clippy, by very carefully boxing off its values from its ability to update. But why worry? Neither nature nor our haphazard stabs at AI are likely to hit on such a design. Intelligence requires the ability to update, to reflect, and to reflect on what is important. Judgements of importance are based on values. So it is important to have the right way of judging importance, the right values. So an intelligent agent would judge it important to have the right values.”
Well, if Prawn knew that they could just tell us and we would be convinced, ending this argument.
More generally … maybe some sort of social contract theory? It might be stable with enough roughly-equal agents, anyway. Prawn has said it would have to be deducible from the axioms of rationality, implying something that’s rational for (almost?) every goal.
“The way people sometimes realise their values are wrong...only more efficiently, because its super intelligent. Well, I’ll concede that with care you might be able to design a clippy, by very carefully boxing off its values from its ability to update. But why worry? Neither nature nor our haphazard stabs at AI are likely to hit on such a design. Intelligence requires the ability to update, to reflect, and to reflect on what is important. Judgements of importance are based on values. So it is important to have the right way of judging importance, the right values. So an intelligent agent would judge it important to have the right values.”