Good point, I see what you mean. I think we could have 2 distinct concepts of “ethics” and 2 corresponding orthogonality theses:
Concept “ethics1” requires ethics to be motivational. Some set of rules can only be the true ethics if, necessarily, everyone who knows them is motivated to follow them. (I think moral internalist probably use this concept?)
Concept “ethics2” doesn’t require some set of rules to be motivational to be the correct ethics.
The orthogonality thesis for 1 is what I mentioned: Since there are (probably) no rules that necessarily motivate everyone who knows them, the AI would not find the true ethical theory.
The orthogonality thesis for 2 is what you mention: Even if the AI finds it, it would not necessarily be motivated by it.
Good point, I see what you mean. I think we could have 2 distinct concepts of “ethics” and 2 corresponding orthogonality theses:
Concept “ethics1” requires ethics to be motivational. Some set of rules can only be the true ethics if, necessarily, everyone who knows them is motivated to follow them. (I think moral internalist probably use this concept?)
Concept “ethics2” doesn’t require some set of rules to be motivational to be the correct ethics.
The orthogonality thesis for 1 is what I mentioned: Since there are (probably) no rules that necessarily motivate everyone who knows them, the AI would not find the true ethical theory.
The orthogonality thesis for 2 is what you mention: Even if the AI finds it, it would not necessarily be motivated by it.
Exactly!