It’s uncontrovesial that rational agents need to update, and that AIs need to self-modify. The claim that values are in either case insulated from updates is the extraordinary one. The Cipper theory tells you that you could build something like that if you were crazy enough. Since Clippers are contrived, nothing can be inferred from them about
typical agents. People are messy, and can accidentally update their values when trying to do something else, For instance, LukeProg updated to “atheist” after studying Christian apologetics for the opposite reason.
Yes, value drift is the typical state for minds in our experience.
Building a committed Clipper that cannot accidentally update its values when trying to do something else is only possible after the problem of value drift has been solved. A system that experiences value drift isn’t a reliable Clipper, isn’t a reliable good-thing-doer, isn’t reliable at all.
It’s uncontrovesial that rational agents need to update, and that AIs need to self-modify. The claim that values are in either case insulated from updates is the extraordinary one.
I never claimed that it was controversial, nor that AIs didn’t need to self-modify, nor that values are exempt.
I’m claiming that updates and self modification do not imply a change of behavior towards behavior desired by humans.
I can build a small toy program to illustrate, if that would help.
I am not suggesting that human ethics is coincidentally universal ethics.
I am suggesting that if neither moral realism nor relativism is initially discarded, one can eventually arrive at a compromise position where rational agents in a particular context arrive at a non arbitrary ethics which is appropriate to that context.
It’s uncontrovesial that rational agents need to update, and that AIs need to self-modify. The claim that values are in either case insulated from updates is the extraordinary one. The Cipper theory tells you that you could build something like that if you were crazy enough. Since Clippers are contrived, nothing can be inferred from them about typical agents. People are messy, and can accidentally update their values when trying to do something else, For instance, LukeProg updated to “atheist” after studying Christian apologetics for the opposite reason.
Yes, value drift is the typical state for minds in our experience.
Building a committed Clipper that cannot accidentally update its values when trying to do something else is only possible after the problem of value drift has been solved. A system that experiences value drift isn’t a reliable Clipper, isn’t a reliable good-thing-doer, isn’t reliable at all.
Next.
I never claimed that it was controversial, nor that AIs didn’t need to self-modify, nor that values are exempt.
I’m claiming that updates and self modification do not imply a change of behavior towards behavior desired by humans.
I can build a small toy program to illustrate, if that would help.
I am not suggesting that human ethics is coincidentally universal ethics. I am suggesting that if neither moral realism nor relativism is initially discarded, one can eventually arrive at a compromise position where rational agents in a particular context arrive at a non arbitrary ethics which is appropriate to that context.