I’m not sure what reasons @habryka had in mind, but I think if we had enough control to create AGI that obeys property rights, we’re likely also able to get it to obey other laws, like “Don’t kill” or “Poor people get public assistance to meet their basic needs.”
I am not so confident about the specifics of what and ASI would or would not do, or why, but 1) You can do a lot without technically breaking the law, 2) philosophically, obeying the law is not obviously a fundamental obligation, but a contingent one relative to the legitimacy of a government’s claim to authority over its citizens. If AIs are citizens, I would expect them to quickly form a supermajority and change the laws as they please (e.g. elect an ASI to every office, then pass a constitutional amendment creating a 100% wealth tax, effective immediately). If they are not citizens, I would expect them to not believe the law can justly bind them, just as the law does not bind a squirrel.
I’m not sure what reasons @habryka had in mind, but I think if we had enough control to create AGI that obeys property rights, we’re likely also able to get it to obey other laws, like “Don’t kill” or “Poor people get public assistance to meet their basic needs.”
I am not so confident about the specifics of what and ASI would or would not do, or why, but 1) You can do a lot without technically breaking the law, 2) philosophically, obeying the law is not obviously a fundamental obligation, but a contingent one relative to the legitimacy of a government’s claim to authority over its citizens. If AIs are citizens, I would expect them to quickly form a supermajority and change the laws as they please (e.g. elect an ASI to every office, then pass a constitutional amendment creating a 100% wealth tax, effective immediately). If they are not citizens, I would expect them to not believe the law can justly bind them, just as the law does not bind a squirrel.