I suggested that, in some situations, questions like “What is your posterior probability?” might not have answers, unless they are part of decision problems like “What odds should you bet at?” or even “What should you rationally anticipate to get a brain that trusts rational anticipation?”. You didn’t comment on the suggestion, so I thought about problems you might have seen in it.
In the suggestion, the “correct” subjective probability depends on a utility function and a UDT/TDT agent’s starting probabilities, which never change. The most important way the suggestion is incomplete is that it doesn’t itself explain something we do naturally: we care about the way our “existentness” has “flowed” to us, and if we learn things about how “existentness” or “experiencedness” works, we change what we care about. So when we experiment on quantum systems, and we get experimental statistics that are more probable under a Born rule with a power of 2 than (hand-waving normalization problems) under a Born rule with a power of 4, we change our preferences, so that we care about what happens in possible future worlds in proportion to their integrated squared amplitude, and not in proportion to the integral of the fourth power. But, if there were people who consistently got experimental statistics that were more probable under a Born rule with a power of 4 (whatever that would mean), we would want them to care about possible future worlds in proportion to the integral of the fourth power of their amplitude.
This can even be done in classical decision theory. Suppose you were creating an agent to be put into a world with Ebborean physics, and you had uncertainty about whether, in the law relating world-thickness ratios (at splitting time) to “existentness” ratios, the power was 2 or 4. It would be easy to put a prior probability of 1⁄2 on each power, and then have “the agent” update from measurements of the relative thicknesses of the sides of the split worlds it (i.e. its local copy) ended up on. But this doesn’t explain why you would want to do that.
What would a UDT/TDT prior belief distribution or utility function have to look like in order to define agents that can “update” in this way, while only thinking in terms of copying and not subjective probability? Suppose you were creating an agent to be put into a world with Ebborean physics, and you had uncertainty about whether, in the relation between world thickness ratios and “existentness” ratios, the power was 2 or 4. And this time, suppose the agent was to be an updateless decision theory agent. I think a UDT agent which uses “probability” can be converted by an expected utility calculation into a behaviorally equivalent UDT agent which uses no probability. Instead of probability, the agent uses only “importances”: relative strengths of its (linearly additive) preferences about what happens in the various deterministic worlds the agent “was” copied into at the time of its creation. To make such an agent in Ebborean physics “update” on “evidence” about existentness, you could take the relative importance you assigned to influencing world-sheets, split it into two halves, and distribute each half across world-sheets in a different way. Half of the importance would be distributed in proportion to the cumulative products of the squares of the worlds’ thickness ratios at their times of splitting, and half of the importance would be distributed in proportion to the cumulative products of the fourth powers of the worlds’ thickness ratios at their times of splitting. Then, in each world-sheet, the copy of the agent in that world-sheet would make some measurements of the relative thicknesses on its side of a split, and it would use use those measurements to decide what kinds of local futures it should prioritize influencing.
But, again, this doesn’t explain why you would want to do that. (Maybe you wanted the agents to take a coordinated action at the end of time using the world-sheets they controlled, and you didn’t know which kinds of world-sheets would become good general-purpose resources for that action?)
I think there was another way my suggestion is incomplete, which has something to do with the way your definition of altruism doesn’t work without a definition of “correct” subjective probability. But I don’t remember what your definition of altruism was or why it didn’t work without subjective probability.
I still think the right way to answer the question, “What is the correct subjective probability?” might be partly to derive “Bayesian updating” as an approximation that can be used by computationally limited agents implementing an updateless or other decision theory, with a utility function defined over mathematical descriptions of worlds containing some number of copies of the agent, when the differences in utility that result from the agent’s decisions fulfill certain independence and linearity assumptions. I need to mathematically formalize those assumptions. “Subjective probability” would then be a variable used in that approximation, which would be meaningless or undefined when the assumptions failed.
I suggested that, in some situations, questions like “What is your posterior probability?” might not have answers, unless they are part of decision problems like “What odds should you bet at?” or even “What should you rationally anticipate to get a brain that trusts rational anticipation?”. You didn’t comment on the suggestion, so I thought about problems you might have seen in it.
In the suggestion, the “correct” subjective probability depends on a utility function and a UDT/TDT agent’s starting probabilities, which never change. The most important way the suggestion is incomplete is that it doesn’t itself explain something we do naturally: we care about the way our “existentness” has “flowed” to us, and if we learn things about how “existentness” or “experiencedness” works, we change what we care about. So when we experiment on quantum systems, and we get experimental statistics that are more probable under a Born rule with a power of 2 than (hand-waving normalization problems) under a Born rule with a power of 4, we change our preferences, so that we care about what happens in possible future worlds in proportion to their integrated squared amplitude, and not in proportion to the integral of the fourth power. But, if there were people who consistently got experimental statistics that were more probable under a Born rule with a power of 4 (whatever that would mean), we would want them to care about possible future worlds in proportion to the integral of the fourth power of their amplitude.
This can even be done in classical decision theory. Suppose you were creating an agent to be put into a world with Ebborean physics, and you had uncertainty about whether, in the law relating world-thickness ratios (at splitting time) to “existentness” ratios, the power was 2 or 4. It would be easy to put a prior probability of 1⁄2 on each power, and then have “the agent” update from measurements of the relative thicknesses of the sides of the split worlds it (i.e. its local copy) ended up on. But this doesn’t explain why you would want to do that.
What would a UDT/TDT prior belief distribution or utility function have to look like in order to define agents that can “update” in this way, while only thinking in terms of copying and not subjective probability? Suppose you were creating an agent to be put into a world with Ebborean physics, and you had uncertainty about whether, in the relation between world thickness ratios and “existentness” ratios, the power was 2 or 4. And this time, suppose the agent was to be an updateless decision theory agent. I think a UDT agent which uses “probability” can be converted by an expected utility calculation into a behaviorally equivalent UDT agent which uses no probability. Instead of probability, the agent uses only “importances”: relative strengths of its (linearly additive) preferences about what happens in the various deterministic worlds the agent “was” copied into at the time of its creation. To make such an agent in Ebborean physics “update” on “evidence” about existentness, you could take the relative importance you assigned to influencing world-sheets, split it into two halves, and distribute each half across world-sheets in a different way. Half of the importance would be distributed in proportion to the cumulative products of the squares of the worlds’ thickness ratios at their times of splitting, and half of the importance would be distributed in proportion to the cumulative products of the fourth powers of the worlds’ thickness ratios at their times of splitting. Then, in each world-sheet, the copy of the agent in that world-sheet would make some measurements of the relative thicknesses on its side of a split, and it would use use those measurements to decide what kinds of local futures it should prioritize influencing.
But, again, this doesn’t explain why you would want to do that. (Maybe you wanted the agents to take a coordinated action at the end of time using the world-sheets they controlled, and you didn’t know which kinds of world-sheets would become good general-purpose resources for that action?)
I think there was another way my suggestion is incomplete, which has something to do with the way your definition of altruism doesn’t work without a definition of “correct” subjective probability. But I don’t remember what your definition of altruism was or why it didn’t work without subjective probability.
I still think the right way to answer the question, “What is the correct subjective probability?” might be partly to derive “Bayesian updating” as an approximation that can be used by computationally limited agents implementing an updateless or other decision theory, with a utility function defined over mathematical descriptions of worlds containing some number of copies of the agent, when the differences in utility that result from the agent’s decisions fulfill certain independence and linearity assumptions. I need to mathematically formalize those assumptions. “Subjective probability” would then be a variable used in that approximation, which would be meaningless or undefined when the assumptions failed.