Also, the expected number of paperclips (what you describe) is not equal to the expected experienced number of paperclips (what would be the relevant weighting for my post).
If you make robots that maximize your proposed “subjective experience” (proportional to mass) and I make robots that maximize some totally different “subjective experience” (how about proportional to mass squared!), all of those robots will act exactly like one would expect—the linear-experience maximizers would maximize linear-experience, the squared-experience maximizers would maximize squared-experience.
Because anything can be putt into a utility function, it’s very hard to talk about subjective experience by referencing utility functions. We want to reduce “subjective experience” to some kind of behavior that we don’t have to put into the utility function by hand.
In the Sleeping Beauty problem, we can start with an agent that selfishly values some payoff (say, candy bars), with no specific weighting on the number of copies, and no explicit terms for “subjective experience.” But then we put it in an unusual situation, and it turns out that the optimum betting strategy is the one where it gives more weight to world where there are more copies of it. That kind o behavior is what indicate to me that there’s something going on with subjective experience.
If you make robots that maximize your proposed “subjective experience” (proportional to mass) and I make robots that maximize some totally different “subjective experience” (how about proportional to mass squared!), all of those robots will act exactly like one would expect—the linear-experience maximizers would maximize linear-experience, the squared-experience maximizers would maximize squared-experience.
Because anything can be putt into a utility function, it’s very hard to talk about subjective experience by referencing utility functions. We want to reduce “subjective experience” to some kind of behavior that we don’t have to put into the utility function by hand.
In the Sleeping Beauty problem, we can start with an agent that selfishly values some payoff (say, candy bars), with no specific weighting on the number of copies, and no explicit terms for “subjective experience.” But then we put it in an unusual situation, and it turns out that the optimum betting strategy is the one where it gives more weight to world where there are more copies of it. That kind o behavior is what indicate to me that there’s something going on with subjective experience.