You’re mischaracterizing the quote that your post replies to. EY claims that he is attempting to comprehend morality as a logical, not a physical thing, and he’s trying to convince readers to do the same. You’re evidently thinking of morality as a physical thing, something essentially derived from the observation of brains. You’re restating the position his post responds to, without strengthening it.
The argument in that post seems incoherent to me. In the conventional natural sciences, what is moral is the subject matter of biology. This employs game theory and evolutionary theory (i.e. logic), but also considers the laws of physics and the local state of the universe to explain existing moral systems.
For instance, consider the question of whether it is wrong to drive on the left-hand side of the road. That isn’t logic, it depends on the local state of the universe. Two advanced superintelligences which had evolved independently could easily find themselves in disagreement over such issues.
This is an example of spontaneous symmetry breaking. It is one of the factors which explains how arbitrarily-advanced agents can still disagree on what the right thing to do is.
You’re mischaracterizing the quote that your post replies to. EY claims that he is attempting to comprehend morality as a logical, not a physical thing, and he’s trying to convince readers to do the same. You’re evidently thinking of morality as a physical thing, something essentially derived from the observation of brains. You’re restating the position his post responds to, without strengthening it.
The argument in that post seems incoherent to me. In the conventional natural sciences, what is moral is the subject matter of biology. This employs game theory and evolutionary theory (i.e. logic), but also considers the laws of physics and the local state of the universe to explain existing moral systems.
For instance, consider the question of whether it is wrong to drive on the left-hand side of the road. That isn’t logic, it depends on the local state of the universe. Two advanced superintelligences which had evolved independently could easily find themselves in disagreement over such issues.
This is an example of spontaneous symmetry breaking. It is one of the factors which explains how arbitrarily-advanced agents can still disagree on what the right thing to do is.