I don’t want to spend too much time trying to convince you, since I think people should mostly follow their own instincts (if they have strong instincts) when choosing what research directions to pursue. I was mainly curious if you had already looked into UDT and found it wanting for some reason. But I’ll try to answer your questions.
why is Bayesian conditionalization not a constraint on the set of beliefs you hold?
What justifies Bayesian conditionalization? Is Bayesian conditionalization so obviously correct that it should be considered an axiom?
It turns out that Bayesian updating is appropriate only under certain conditions (which in particular are not satisfied in situations with indexical uncertainty), but this is not easy to see except in the context of decision theory. See Why (and why not) Bayesian Updating?
what are the most important implications of getting the right decision theory, other than building super-AIs?
I’ve already mentioned that I found it productive to consider anthropic reasoning from a decision theoretic perspective (btw, that, not super-AIs, was in fact my original motivation for studying decision theory). So I’m not quite sure what you’re asking here...
The obviousness of Bayesian conditionalization seems beside the point. Which is that it constrains beliefs and need not be derived from the set of decisions that seem reasonable.
Your link seems to only suggest that using Bayesian conditionalization in the context of a poor decision theory doesn’t give you the results you want. Which doesn’t say much about Bayesian conditionalization. Am I missing something?
“So I’m not quite sure what you’re asking here...”
It is possible for things to be more important than an unquantified increase in productivity on anthropics. I’m also curious whether you think it has other implications.
I think the important point is that Bayesian conditionalization is a consequences of a decision theory that, naturally stated, does not invoke Bayesian conditionalization.
That being:
Consider the set of all strategies mapping situations to actions. Play the one which maximizes your expected utility from a state of no information.
I don’t want to spend too much time trying to convince you, since I think people should mostly follow their own instincts (if they have strong instincts) when choosing what research directions to pursue. I was mainly curious if you had already looked into UDT and found it wanting for some reason. But I’ll try to answer your questions.
What justifies Bayesian conditionalization? Is Bayesian conditionalization so obviously correct that it should be considered an axiom?
It turns out that Bayesian updating is appropriate only under certain conditions (which in particular are not satisfied in situations with indexical uncertainty), but this is not easy to see except in the context of decision theory. See Why (and why not) Bayesian Updating?
I’ve already mentioned that I found it productive to consider anthropic reasoning from a decision theoretic perspective (btw, that, not super-AIs, was in fact my original motivation for studying decision theory). So I’m not quite sure what you’re asking here...
The obviousness of Bayesian conditionalization seems beside the point. Which is that it constrains beliefs and need not be derived from the set of decisions that seem reasonable.
Your link seems to only suggest that using Bayesian conditionalization in the context of a poor decision theory doesn’t give you the results you want. Which doesn’t say much about Bayesian conditionalization. Am I missing something?
“So I’m not quite sure what you’re asking here...”
It is possible for things to be more important than an unquantified increase in productivity on anthropics. I’m also curious whether you think it has other implications.
I think the important point is that Bayesian conditionalization is a consequences of a decision theory that, naturally stated, does not invoke Bayesian conditionalization.
That being:
Consider the set of all strategies mapping situations to actions. Play the one which maximizes your expected utility from a state of no information.
Bayesian conditionalization can be derived from Dutch book arguments, which are (hypothetical) decisions...