You would give yourself what you like. Maybe you like danger. People voluntarily parachute and mountain-climb. If the unsafe thing you get is what you want, where is the problem?
But if you do care about your wishes being fulfilled safely, then safety will be one of the things that you want, and so you will get it.
So long as your preferences are coherent, stable, and self-consistent then you should be fine. If you care about something that’s relevant to the wish then it will be incorporated into the wish. If you don’t care about something then it may not be incorporated into the wish, but you shouldn’t mind that: because it’s something you don’t care about.
Unfortunately, people’s preferences often aren’t coherent and stable. For instance an alcoholic may throw away a bottle of wine because they don’t want to be tempted by it. Right now, they don’t want their future selves to drink it. And yet they know that their future selves might have different priorities.
Is this the sort of thing you were concerned about?
“So long as your preferences are coherent, stable, and self-consistent then you should be fine.”
Yes, absolutely.
And yes, the fact that my preferences are not coherent, stable, and self-consistent is probably the sort of thing I was concerned about… though it was years ago.
A genie who gives me what I would give myself is far from being a safe fulfiller of a wish.
Because?
Because I am not guaranteed to only give myself things that are safe.
You would give yourself what you like. Maybe you like danger. People voluntarily parachute and mountain-climb. If the unsafe thing you get is what you want, where is the problem?
Sure, if all I care about is whether I get what I want, and I don’t care about whether my wishes are fulfilled safely, then there’s no problem.
But if you do care about your wishes being fulfilled safely, then safety will be one of the things that you want, and so you will get it.
So long as your preferences are coherent, stable, and self-consistent then you should be fine. If you care about something that’s relevant to the wish then it will be incorporated into the wish. If you don’t care about something then it may not be incorporated into the wish, but you shouldn’t mind that: because it’s something you don’t care about.
Unfortunately, people’s preferences often aren’t coherent and stable. For instance an alcoholic may throw away a bottle of wine because they don’t want to be tempted by it. Right now, they don’t want their future selves to drink it. And yet they know that their future selves might have different priorities.
Is this the sort of thing you were concerned about?
Yes, absolutely.
And yes, the fact that my preferences are not coherent, stable, and self-consistent is probably the sort of thing I was concerned about… though it was years ago.