Of course, the Humean theory may be false and so Bostrom wisely avoids it in his defence of the orthogonality thesis.
I had the opposite reaction. The Humean theory of motivation is correct, and I see no reason to avoid tying the orthogonality thesis to it. To me, Bostrom’s distancing of the orthogonality thesis from Humean motivation seemed like splitting hairs. Since how strong a given motivation is can only be measured relative to other motivations, Bostrom’s point that an agent could have very strong motivations not arising from beliefs and that these could then overwhelm the motivating beliefs, is essentially equivalent to saying that there might be motivating beliefs, but only weakly-motivating beliefs; in other words, that the Humean theory could be false but close enough to true that it doesn’t matter. The point that there might be motivating beliefs but that these are disjoint from instrumental beliefs and thus an agent would not have a motivation to acquire the correct motivating beliefs, seems compatible with Humean motivation up to differing definitions of ambiguous words. If you have to taboo “belief” and “value”, then “the set of thoughts that predict events and the effects of potential actions is disjoint from the set of thoughts that motivate performing actions predicted by other thoughts to have some particular effects” seems like a plausible interpretation of the Humean theory, and makes it no longer sound so different from Bostrom’s point.
From John Danaher’s review:
I had the opposite reaction. The Humean theory of motivation is correct, and I see no reason to avoid tying the orthogonality thesis to it. To me, Bostrom’s distancing of the orthogonality thesis from Humean motivation seemed like splitting hairs. Since how strong a given motivation is can only be measured relative to other motivations, Bostrom’s point that an agent could have very strong motivations not arising from beliefs and that these could then overwhelm the motivating beliefs, is essentially equivalent to saying that there might be motivating beliefs, but only weakly-motivating beliefs; in other words, that the Humean theory could be false but close enough to true that it doesn’t matter. The point that there might be motivating beliefs but that these are disjoint from instrumental beliefs and thus an agent would not have a motivation to acquire the correct motivating beliefs, seems compatible with Humean motivation up to differing definitions of ambiguous words. If you have to taboo “belief” and “value”, then “the set of thoughts that predict events and the effects of potential actions is disjoint from the set of thoughts that motivate performing actions predicted by other thoughts to have some particular effects” seems like a plausible interpretation of the Humean theory, and makes it no longer sound so different from Bostrom’s point.