Legal systems are (at best) a “satisficer”s map of human values, necessary but insufficient. If you want to spend your life making as many paper clips as possible within the constraints of the law, we’ll be disappointed in you but we won’t forceably stop you, because it’s your life. If an AGI decides it wants the same thing, I we’d rightly consider it a failure, even if we were lucky enough that it wanted to be similarly constrained.
Legal systems are maps of human values.
Legal systems are (at best) a “satisficer”s map of human values, necessary but insufficient. If you want to spend your life making as many paper clips as possible within the constraints of the law, we’ll be disappointed in you but we won’t forceably stop you, because it’s your life. If an AGI decides it wants the same thing, I we’d rightly consider it a failure, even if we were lucky enough that it wanted to be similarly constrained.