Again you may object that this is circular reasoning and I am assuming ought right in the statement 1. But it would be like saying that I am assuming to have two apples. Sure, I am assuming that. And what is the problem exactly?
The difference from Sean Carroll’s point of view (logical argumentation) is that not every scientifically competent agent will find this notion of “ought” compelling. (Really, only chess-playing programs would, if “ought” is taken in a terminal-value sense.) Whereas, such an agent’s scientific competence would lead it to find compelling the axiom that you have two apples.
And I think that Sam Harris would agree with that, so far as it goes. But he would deny that this keeps him from reducing “ought” statements to purely scientific “is” statements, because he’s taking the dialectical-argumentation point of view, not the logical-argumentation one. At any rate, Harris understands that a superintelligent AI might not be bothered by a universe consisting purely of extreme suffering. This was clear from his conversation with Eliezer Yudkowsky.
The difference from Sean Carroll’s point of view (logical argumentation) is that not every scientifically competent agent will find this notion of “ought” compelling. (Really, only chess-playing programs would, if “ought” is taken in a terminal-value sense.) Whereas, such an agent’s scientific competence would lead it to find compelling the axiom that you have two apples.
And I think that Sam Harris would agree with that, so far as it goes. But he would deny that this keeps him from reducing “ought” statements to purely scientific “is” statements, because he’s taking the dialectical-argumentation point of view, not the logical-argumentation one. At any rate, Harris understands that a superintelligent AI might not be bothered by a universe consisting purely of extreme suffering. This was clear from his conversation with Eliezer Yudkowsky.