I think a lot of confusion here comes from people lumping together ultimate and intermediate goals in their definitions of morality. Ultimate goals are parts of your utility function: what you really want. As you said, you can’t derive these rationally; they’re just there. Intermediate goals, on the other hand, are mental shortcuts, things that you want as a proxy for some deeper desire. An example would be the goal that violent criminals get thrown in jail or otherwise separated from society; the ultimate goal that this serves is our desire to avoid things like being savagely beaten by random ne’er-do-wells when we go to the 7⁄11 to buy delicious melon bread. But if there were a more effective, or cheaper, or more humane way to prevent violent crime, rationality can help you figure out that you should prefer it.
Rationality can and should define your intermediate goals, but can’t define your ultimate goals. But when most people talk about morality, they make no distinction between these. As soon as you do distinguish between these, the question tends to dissolve. Just look at all the flak that Sam Harris is getting for saying that science can answer moral questions. What he’s really saying is that science can help us determine our utility functions and figure out how to optimize them. The criticism he gets would probably evaporate if he would taboo “morality” for a little while, but he gets way more media attention by talking this way.
I think a lot of confusion here comes from people lumping together ultimate and intermediate goals in their definitions of morality. Ultimate goals are parts of your utility function: what you really want. As you said, you can’t derive these rationally; they’re just there. Intermediate goals, on the other hand, are mental shortcuts, things that you want as a proxy for some deeper desire. An example would be the goal that violent criminals get thrown in jail or otherwise separated from society; the ultimate goal that this serves is our desire to avoid things like being savagely beaten by random ne’er-do-wells when we go to the 7⁄11 to buy delicious melon bread. But if there were a more effective, or cheaper, or more humane way to prevent violent crime, rationality can help you figure out that you should prefer it.
Rationality can and should define your intermediate goals, but can’t define your ultimate goals. But when most people talk about morality, they make no distinction between these. As soon as you do distinguish between these, the question tends to dissolve. Just look at all the flak that Sam Harris is getting for saying that science can answer moral questions. What he’s really saying is that science can help us determine our utility functions and figure out how to optimize them. The criticism he gets would probably evaporate if he would taboo “morality” for a little while, but he gets way more media attention by talking this way.