Here’s my guess at one type of situation Eliezer might be
thinking of when calling proposition B false: It is
rational (let us stipulate) for a paperclip maximizer to
turn all the matter in the solar system into computronium in
order to compute ways to maximize paperclips, but “should”
does not apply to paperclip maximizers.
EDIT: If I were picking nits, I would say, “‘Should’ does apply to paperclip maximizers—it is rational for X to make paperclips but it should not do so—however, paperclip maximizers don’t care and so it is pointless to talk about what they should do.” But the overall intent of the statement is correct—I disagree with its intent in neither anticipation nor morals—and in such cases I usually just say “Correct”. In this case I suppose that wasn’t the best policy, but it is my usual policy.
Interesting, what about either of the following:
A) If X should do A, then it is rational for X to do A.
B) If it is rational for X to do A, then X should do A.
From what I understand of what Eliezer’s position:
False
False.
(If this isn’t the case then Eliezer’s ‘should’ is even more annoying than how I now understand it.)
Yep, both false.
So, just to dwell on this for a moment, there exist X and A such that (1) it is rational for X to do A and (2) X should not do A.
How do you reconcile this with “rationalists should win”? (I think I know what your response will be, but I want to make sure.)
Here’s my guess at one type of situation Eliezer might be thinking of when calling proposition B false: It is rational (let us stipulate) for a paperclip maximizer to turn all the matter in the solar system into computronium in order to compute ways to maximize paperclips, but “should” does not apply to paperclip maximizers.
Correct.
EDIT: If I were picking nits, I would say, “‘Should’ does apply to paperclip maximizers—it is rational for X to make paperclips but it should not do so—however, paperclip maximizers don’t care and so it is pointless to talk about what they should do.” But the overall intent of the statement is correct—I disagree with its intent in neither anticipation nor morals—and in such cases I usually just say “Correct”. In this case I suppose that wasn’t the best policy, but it is my usual policy.
Of course, Kant distinguished between two different meanings of “should”: the hypothetical and the categorical.
If you want to be a better Go player, you should study the games of Honinbo Shusaku.
You should pull the baby off the rail track.
This seems useful here...
False. Be consistent.