What would constitute a non-arbitrary limit? A utility function just describes the behavior of a physical being. It is not surprising that a physical being has limits in its behavior—that follows from the fact that physical law imposes limits on what a thing does. This is why e.g. transhumanist hope for real immortality is absurd. Even if you could find a way around entropy, you will never change the fact that you are following physical laws. The only way you will exist forever is if current physical laws extrapolated from your current situation imply that you will exist forever. There is no reason to think this is the case, and extremely strong reasons to think that it is not.
In the same way, everything I do is implied by physical laws, including the fact that I express a preference for one thing rather than another. It may be that a good conman will be able to persuade some people to order their preferences in a way that gets them to commit suicide (e.g. by accepting the lifespan offer), but they will be very unlikely to be able to persuade me to order my preferences that way. This is “arbitrary” only in the sense that in order for this to be true, there have to be physical differences between me and the people who would be persuaded that in theory neither of us is responsible for. I don’t have a problem with that. I still don’t plan to commit suicide.
Then we agree. I too have limits that are defined by my physiology. This is why I think that I couldn’t stand 50 years of torture but I could stand 3^^^3 dust speck diluted among 3^^^3 lives.
“arbitrary limit”
What would constitute a non-arbitrary limit? A utility function just describes the behavior of a physical being. It is not surprising that a physical being has limits in its behavior—that follows from the fact that physical law imposes limits on what a thing does. This is why e.g. transhumanist hope for real immortality is absurd. Even if you could find a way around entropy, you will never change the fact that you are following physical laws. The only way you will exist forever is if current physical laws extrapolated from your current situation imply that you will exist forever. There is no reason to think this is the case, and extremely strong reasons to think that it is not.
In the same way, everything I do is implied by physical laws, including the fact that I express a preference for one thing rather than another. It may be that a good conman will be able to persuade some people to order their preferences in a way that gets them to commit suicide (e.g. by accepting the lifespan offer), but they will be very unlikely to be able to persuade me to order my preferences that way. This is “arbitrary” only in the sense that in order for this to be true, there have to be physical differences between me and the people who would be persuaded that in theory neither of us is responsible for. I don’t have a problem with that. I still don’t plan to commit suicide.
Then we agree. I too have limits that are defined by my physiology. This is why I think that I couldn’t stand 50 years of torture but I could stand 3^^^3 dust speck diluted among 3^^^3 lives.