Eliezer, wishes aren’t horses; strongly wanting to be able to tell the difference doesn’t by itself give us evidence to distinguish. Note that legal punishments often distinguish between conscious intention and all other internal causation; so apparently that is the distinction the law considers most relevant, and/or easiest to determine. “Optimize” invites too many knee-jerk complaints that we won’t exactly optimize anything.
Eliezer, wishes aren’t horses; strongly wanting to be able to tell the difference doesn’t by itself give us evidence to distinguish. Note that legal punishments often distinguish between conscious intention and all other internal causation; so apparently that is the distinction the law considers most relevant, and/or easiest to determine. “Optimize” invites too many knee-jerk complaints that we won’t exactly optimize anything.