″...if you would prefer not to become orgasmium, then why should you?”
I’d prefer not to become orgasmium, because I value my consciousness and humanity, my ability to think, decide, and interact. However, it’s unclear to me what exactly preference is, other than the traversal of pathways we’ve constructed, whether we’re aware of them or not, leading to pleasure, or away from pain. To drastically oversimplify, those values exist in me as a result of beliefs I’ve constructed, linking the lack of those things to an identity that I don’t want, which in turn is eventually linked to an emotional state of sadness and loss that I’d like to avoid. There’s also probably a link to a certain identity that I do want, which leads to a certain sense of pride and rightness, which leads me to a positive emotional state.
Eliezer, you said there was nothing higher to override your preference to increase intelligence gradually. But what about the preferences that led you to that one? What was it about the gradual increase of intelligence, and your beliefs about what that means, that compelled you to prefer it? Isn’t that deeper motivation closer to your actual volition? How far down can we chase this? What is the terminal value of fun, if not orgasmium?
Or is “fun” in this context the pursuit specifically of those preferences that we’re consciously aware of as goals?
″...if you would prefer not to become orgasmium, then why should you?”
I’d prefer not to become orgasmium, because I value my consciousness and humanity, my ability to think, decide, and interact. However, it’s unclear to me what exactly preference is, other than the traversal of pathways we’ve constructed, whether we’re aware of them or not, leading to pleasure, or away from pain. To drastically oversimplify, those values exist in me as a result of beliefs I’ve constructed, linking the lack of those things to an identity that I don’t want, which in turn is eventually linked to an emotional state of sadness and loss that I’d like to avoid. There’s also probably a link to a certain identity that I do want, which leads to a certain sense of pride and rightness, which leads me to a positive emotional state.
Eliezer, you said there was nothing higher to override your preference to increase intelligence gradually. But what about the preferences that led you to that one? What was it about the gradual increase of intelligence, and your beliefs about what that means, that compelled you to prefer it? Isn’t that deeper motivation closer to your actual volition? How far down can we chase this? What is the terminal value of fun, if not orgasmium?
Or is “fun” in this context the pursuit specifically of those preferences that we’re consciously aware of as goals?