Suppose Sia’s only goal is to commit suicide, and she’s given the opportunity to kill herself straightaway. Then, it certainly won’t be rational for her to pursue self-preservation.
It seems you found one terminal goal which doesn’t give rise to the instrumental subgoal of self-preservation. Are there others, or does basically every terminal goal benefit from instrumental self-preservation except for suicide?
(I skipped around a bit and didn’t read your full post, so maybe you explain this already and I missed it.)
There are infinitely many desires like that, in fact (that’s what proposition 2 shows).
More generally, take any self-preservation contingency plan, A, and any other contingency plan, B. If we start out uncertain about what Sia wants, then we should think her desires are just as likely to make A more rational than B as they are to make B more rational than A. (That’s what proposition 3 shows.)
That’s rough and subject to a bunch of caveats, of course. I try to go through all of those caveats carefully in the draft.
Related question: Doesn’t goal preservation typically imply self preservation? If I want to preserve my goal, and then I perish, I’ve failed because now my goal has been reassigned from X to nil.
A quick prefatory note on how I’m thinking about ‘goals’ (I don’t think it’s relevant, but I’m not sure): as I’m modelling things, Sia’s desires/goals are given by a function from ways the world could be (colloquially, ‘worlds’) to real numbers, D, with the interpretation that D(W) is how well satisfied Sia’s desires are if W turns out to be the way the world actually is. By ‘the world’, I mean to include all of history, from the beginning to the end of time, and I mean to encompass every region of space. I assume that this function can be well-defined even for worlds in which Sia never existed or dies quickly. Humans can want to never have been born, and they can want to die. So I’m assuming that Sia can also have those kinds of desires, in principle. So her goal can be achieved even if she’s not around.
When I talk about ‘goal preservation’, I was talking about Sia not wanting to change her desires. I think you’re right that that’s different from Sia wanting to retain her desires. If she dies, then she hasn’t retained her desires, but neither has she changed them. The effect I found was that Sia is somewhat more likely to not want her desires changed.
Love to see an orthodoxy challenged!
It seems you found one terminal goal which doesn’t give rise to the instrumental subgoal of self-preservation. Are there others, or does basically every terminal goal benefit from instrumental self-preservation except for suicide?
(I skipped around a bit and didn’t read your full post, so maybe you explain this already and I missed it.)
There are infinitely many desires like that, in fact (that’s what proposition 2 shows).
More generally, take any self-preservation contingency plan, A, and any other contingency plan, B. If we start out uncertain about what Sia wants, then we should think her desires are just as likely to make A more rational than B as they are to make B more rational than A. (That’s what proposition 3 shows.)
That’s rough and subject to a bunch of caveats, of course. I try to go through all of those caveats carefully in the draft.
Interesting… still taking that in.
Related question: Doesn’t goal preservation typically imply self preservation? If I want to preserve my goal, and then I perish, I’ve failed because now my goal has been reassigned from X to nil.
A quick prefatory note on how I’m thinking about ‘goals’ (I don’t think it’s relevant, but I’m not sure): as I’m modelling things, Sia’s desires/goals are given by a function from ways the world could be (colloquially, ‘worlds’) to real numbers, D, with the interpretation that D(W) is how well satisfied Sia’s desires are if W turns out to be the way the world actually is. By ‘the world’, I mean to include all of history, from the beginning to the end of time, and I mean to encompass every region of space. I assume that this function can be well-defined even for worlds in which Sia never existed or dies quickly. Humans can want to never have been born, and they can want to die. So I’m assuming that Sia can also have those kinds of desires, in principle. So her goal can be achieved even if she’s not around.
When I talk about ‘goal preservation’, I was talking about Sia not wanting to change her desires. I think you’re right that that’s different from Sia wanting to retain her desires. If she dies, then she hasn’t retained her desires, but neither has she changed them. The effect I found was that Sia is somewhat more likely to not want her desires changed.