What is your definition of philosophy for this article? Why is it a failing of a highly intelligent mind that it can’t “do philosophy”? Why would a Bayesian EU maximizer necessarily be unable to tell that a computable prior is wrong? When is Bayesian updating the wrong thing to do? What should I have learned from your link to Updateless Decision Theory that causes me to suspect that EU maximizing with Bayesian updating on a universal prior is wrong? Doesn’t rationality require identification of one’s goals, therefore inheriting the full complexity of value of oneself? What would count as an example of a metaphilosophical insight?
What should I have learned from your link to Updateless Decision Theory that causes me to suspect that EU maximizing with Bayesian updating on a universal prior is wrong?
From what I can glean from the UDT descriptions it seems that UDT defines ‘updating’ to include things that I would prefer to describe as ‘naive updating’, ‘updating wrong’ or ‘updating the wrong thing’.
Doesn’t rationality require identification of one’s goals, therefore inheriting the full complexity of value of oneself?
Seconded. We can certainly imagine an amoral agent that responds to rational argument — say, a paperclipper that can be convinced to one-box on Newcomb’s problem. This gives rise to the illusion that rationality is somehow universal.
But in what sense is an EU-maximizer with a TM-based universal prior “wrong”? If it loses money when betting on a unary encoding of the Busy Beaver sequence, maybe we should conclude that making money isn’t its goal.
If someone knows a way to extract goals from an arbitrary agent in a way that might reveal the agent to be irrational, I would like to hear it.
For instrumental rationality, yes; for epistemic rationality, no. If the reason the EU-maximizer loses money is because it believes that the encoding will be different than it actually is, then it is irrational.
At least as I understand his point about rationality being objective, I assume he means that “given a set of goals and possible decisions, the most effective decision is determined.”
I don’t really understand why this doesn’t apply to morality as such, unless they aren’t similar in the way he implies.
Wei Dai distinguishes between objective rationality and subjectively obective morality:
in ethics we find it hard to do better than to identify “morality” with a huge blob of computation which is particular to human minds, but it appears that in decision theory “rationality” isn’t similarly dependent on complex details unique to humanity.
Morality does seem to be more complex than decision theory. But aren’t they both subjectively objective? Just as we can define the class of moral agents as those which respond to moral arguments, we can define the (larger) class of rational agents as those which respond to arguments about rationality.
I’m familiar with that. As I understand it, EY does not say that Bayesian updating is suboptimal. If anything, he says the opposite, that standard rationality gives you the right answer.
Could you be more specific about where you believe that article claims Bayesian updating is the wrong thing to do?
Does “extremal counterfactual mugging” where $100 is replaced by self-destruction of agent and $10000 is replaced by creation of 100 agent’s copies (outside of agent’s light cone) requires same answer as counterfactual mugging?
I don’t believe it’s particularly controversial. There is a question of whether humans retain preference about counterfactual worlds, but decision-theoretically, not-updating in the usual sense is strictly superior, because you get to make decisions you otherwise wouldn’t be able to.
Okay, then let me try to trace back to the point where we disagree.
As I understand it:
1) Timeless Decision Theory tells you what to do, given your beliefs. Any belief updating would happen before you apply TDT then, so I don’t understand how TDT would err in terms of doing a Bayesian update (and that update is wrong to do) -- the error is independent of TDT, as TDT (see below) shields your actions from making losing decisions on the basis of such a “bad” update.
2) TDT can be stated as, “When calculating EU for the outcomes of an action, you must instead weight each outcome’s utility by the probability that this action would have led to it if your decision procedure were such that it outputs this action.”
So, on counterfactual mugging, even if it reasons that it’s not in the winning world, it reasons that its decision theory leads to the highest (TDT-calculated) EU by setting its action to a policy of paying out on losing, as then it can add the utility of the winning side into its EU.
Or does EY agree that TDT fails on CM? (I couldn’t tell from the CM article.)
3) Edit: And even if this is a case of Bayesian updating failing, does that generalize to dropping it altogether?
I just answered due to a strong gut feeling. It’s some time since I read that article.
But there’s always a way to set up a particular situation (at least regarding thought-experiments) where the optimal strategy is by definition of the rules not to update on evidence. If I remember right it didn’t matter if Omega left and so it couldn’t remove money anymore, because it was capable of perfect prediction and only does bestow those who do not update, i.e. are irrational agents under most other ‘real-life’ circumstances.
Anyway, I just tried to point you to to something because nobody replied to that particular question yet. I even had to look up what ‘TDT’ stands for :-)
What is your definition of philosophy for this article?
Why is it a failing of a highly intelligent mind that it can’t “do philosophy”?
Why would a Bayesian EU maximizer necessarily be unable to tell that a computable prior is wrong?
When is Bayesian updating the wrong thing to do?
What should I have learned from your link to Updateless Decision Theory that causes me to suspect that EU maximizing with Bayesian updating on a universal prior is wrong?
Doesn’t rationality require identification of one’s goals, therefore inheriting the full complexity of value of oneself?
What would count as an example of a metaphilosophical insight?
From what I can glean from the UDT descriptions it seems that UDT defines ‘updating’ to include things that I would prefer to describe as ‘naive updating’, ‘updating wrong’ or ‘updating the wrong thing’.
Pray tell, what is the right thing to update?
Seconded. We can certainly imagine an amoral agent that responds to rational argument — say, a paperclipper that can be convinced to one-box on Newcomb’s problem. This gives rise to the illusion that rationality is somehow universal.
But in what sense is an EU-maximizer with a TM-based universal prior “wrong”? If it loses money when betting on a unary encoding of the Busy Beaver sequence, maybe we should conclude that making money isn’t its goal.
If someone knows a way to extract goals from an arbitrary agent in a way that might reveal the agent to be irrational, I would like to hear it.
For instrumental rationality, yes; for epistemic rationality, no. If the reason the EU-maximizer loses money is because it believes that the encoding will be different than it actually is, then it is irrational.
At least as I understand his point about rationality being objective, I assume he means that “given a set of goals and possible decisions, the most effective decision is determined.”
I don’t really understand why this doesn’t apply to morality as such, unless they aren’t similar in the way he implies.
Wei Dai distinguishes between objective rationality and subjectively obective morality:
Morality does seem to be more complex than decision theory. But aren’t they both subjectively objective? Just as we can define the class of moral agents as those which respond to moral arguments, we can define the (larger) class of rational agents as those which respond to arguments about rationality.
I think, but I’m not sure… http://lesswrong.com/lw/nc/newcombs_problem_and_regret_of_rationality/
I’m familiar with that. As I understand it, EY does not say that Bayesian updating is suboptimal. If anything, he says the opposite, that standard rationality gives you the right answer.
Could you be more specific about where you believe that article claims Bayesian updating is the wrong thing to do?
Bayesian updating is the wrong thing to do in counterfactual mugging, and the reason TDT goes wrong on that problem is that it updates.
Does “extremal counterfactual mugging” where $100 is replaced by self-destruction of agent and $10000 is replaced by creation of 100 agent’s copies (outside of agent’s light cone) requires same answer as counterfactual mugging?
And this is an uncontroversial view here, which one can safely assert as a premise, as Wei_Dai did here?
I don’t believe it’s particularly controversial. There is a question of whether humans retain preference about counterfactual worlds, but decision-theoretically, not-updating in the usual sense is strictly superior, because you get to make decisions you otherwise wouldn’t be able to.
Okay, then let me try to trace back to the point where we disagree.
As I understand it:
1) Timeless Decision Theory tells you what to do, given your beliefs. Any belief updating would happen before you apply TDT then, so I don’t understand how TDT would err in terms of doing a Bayesian update (and that update is wrong to do) -- the error is independent of TDT, as TDT (see below) shields your actions from making losing decisions on the basis of such a “bad” update.
2) TDT can be stated as, “When calculating EU for the outcomes of an action, you must instead weight each outcome’s utility by the probability that this action would have led to it if your decision procedure were such that it outputs this action.”
So, on counterfactual mugging, even if it reasons that it’s not in the winning world, it reasons that its decision theory leads to the highest (TDT-calculated) EU by setting its action to a policy of paying out on losing, as then it can add the utility of the winning side into its EU.
Or does EY agree that TDT fails on CM? (I couldn’t tell from the CM article.)
3) Edit: And even if this is a case of Bayesian updating failing, does that generalize to dropping it altogether?
I just answered due to a strong gut feeling. It’s some time since I read that article.
But there’s always a way to set up a particular situation (at least regarding thought-experiments) where the optimal strategy is by definition of the rules not to update on evidence. If I remember right it didn’t matter if Omega left and so it couldn’t remove money anymore, because it was capable of perfect prediction and only does bestow those who do not update, i.e. are irrational agents under most other ‘real-life’ circumstances.
Anyway, I just tried to point you to to something because nobody replied to that particular question yet. I even had to look up what ‘TDT’ stands for :-)
Sorry for bothering.