If we use some variant of UDT, the same line of reasoning is experienced by many other minds and we should reason as if we have causal power over all these minds.
As I understand UDT, this isn’t right. UDT 1.1 chooses an input-output mapping that maximizes expected utility. Even assuming that all people who read LW run UDT 1.1, this choice still only determines the input-output behavior of a couple of programs (humans). The outputs of programs that don’t depend on our outputs because those programs aren’t running UDT are held constant. Therefore, if you formalized this problem, UDT’s output could be “stockpile food” even if [every human doing that] would lead to a disaster.
I think “pretend as if everyone runs UDT” was neither intentioned by Wei Dei nor is it a good idea. Differently put, UDT agents don’t cooperate in a one-shot prisoner’s dilemma if they play vs. CDT agents.
Also: if a couple of people stockpile food, but most people don’t, that seems like a preferable outcome to everyone doing nothing (provided stockpiling food is worth doing). It means some get to prepare, and the food market isn’t significantly affected. So this particular situation actually doesn’t seem to be isomorphic to the prisoner’s dilemma (if modeled via game theory).
I agree with avturchin, it’s an appropriate thought to be having. UDT-like reasoning is actually fairly common in populations that have not been tainted with CDT rationality (IE, normal people) (usually it is written off by cdt rationalists as moralising or collectivism). This line of thinking doesn’t require exact equivalence, the fact that there are many other people telling many other communities to prep is enough that all of those communities should consider the aggregate effects of that reasoning process. They are all capable of saying “what if everyone else did this as well? Wouldn’t it be bad? Should we really do it?”
They are all capable of saying “what if everyone else did this as well? Wouldn’t it be bad? Should we really do it?”
This doesn’t seem very similar to actual UDT reasoning though. It seems like a perfectly consistent outcome if “normal people” reason like this and conclude that they should refrain from hoarding food, and UDT agents do hoard food because they calculate a low logical correlation between themselves and “normal people”.
I think that cooperating only with those who are provably UDT-agents would make the whole UDT-idea weaker. However, in our case people don’t need to know the word “UDT” to understand that by buying food they are limiting other’s chances to buy it.
I don’t think there is a UDT-idea that prescribes cooperating with non-UDT agents. UDT is sufficiently formalized that we know what happens if a UDT agent plays a prisoner’s dilemma with a CDT agent and both parties know each other’s algorithm/code: they both defect.
If you want to cooperate out of altruism, I think the solution is to model the game differently. The outputs that go into the game theory model should be whatever your utility function says, not your well-being. So if you value the other person’s well-being as much as yours, then you don’t have a prisoner’s dilemma because cooperate/defect is a better outcome for you than defect/defect.
by buying food they are limiting other’s chances to buy it.
But they’re only doing that if there will, in fact, be a supply shortage. That was my initial point – it depends on how many other people will stockpile food.
What worries me here is that while playing, say, prisoner dilemma, an agent needs to perform an act of communication with another prisoner to learn her decision theory, which kills all the problem: if we can communicate, we can have some coordination strategy. In one shot prisoner’s dilemma we don’t know if the other side UDT or CDT agent, and other side also don’t know this about us. So the both are using similar lines of reasoning trying to guess if other agent is CDT or UDT. This similar reasoning itself could be a subject of UDT on meta-level, as we both would win more, if we assume that the other agent is UDT-agent.
As I understand UDT, this isn’t right. UDT 1.1 chooses an input-output mapping that maximizes expected utility. Even assuming that all people who read LW run UDT 1.1, this choice still only determines the input-output behavior of a couple of programs (humans). The outputs of programs that don’t depend on our outputs because those programs aren’t running UDT are held constant. Therefore, if you formalized this problem, UDT’s output could be “stockpile food” even if [every human doing that] would lead to a disaster.
I think “pretend as if everyone runs UDT” was neither intentioned by Wei Dei nor is it a good idea.
Differently put, UDT agents don’t cooperate in a one-shot prisoner’s dilemma if they play vs. CDT agents.
Also: if a couple of people stockpile food, but most people don’t, that seems like a preferable outcome to everyone doing nothing (provided stockpiling food is worth doing). It means some get to prepare, and the food market isn’t significantly affected. So this particular situation actually doesn’t seem to be isomorphic to the prisoner’s dilemma (if modeled via game theory).
I agree with avturchin, it’s an appropriate thought to be having. UDT-like reasoning is actually fairly common in populations that have not been tainted with CDT rationality (IE, normal people) (usually it is written off by cdt rationalists as moralising or collectivism). This line of thinking doesn’t require exact equivalence, the fact that there are many other people telling many other communities to prep is enough that all of those communities should consider the aggregate effects of that reasoning process. They are all capable of saying “what if everyone else did this as well? Wouldn’t it be bad? Should we really do it?”
This doesn’t seem very similar to actual UDT reasoning though. It seems like a perfectly consistent outcome if “normal people” reason like this and conclude that they should refrain from hoarding food, and UDT agents do hoard food because they calculate a low logical correlation between themselves and “normal people”.
How do you calculate logical correlation? Do we know anything about how this would work under UDT? Does UDT not really discuss it, or is it bad at it?
I think that cooperating only with those who are provably UDT-agents would make the whole UDT-idea weaker. However, in our case people don’t need to know the word “UDT” to understand that by buying food they are limiting other’s chances to buy it.
I don’t think there is a UDT-idea that prescribes cooperating with non-UDT agents. UDT is sufficiently formalized that we know what happens if a UDT agent plays a prisoner’s dilemma with a CDT agent and both parties know each other’s algorithm/code: they both defect.
If you want to cooperate out of altruism, I think the solution is to model the game differently. The outputs that go into the game theory model should be whatever your utility function says, not your well-being. So if you value the other person’s well-being as much as yours, then you don’t have a prisoner’s dilemma because cooperate/defect is a better outcome for you than defect/defect.
But they’re only doing that if there will, in fact, be a supply shortage. That was my initial point – it depends on how many other people will stockpile food.
What worries me here is that while playing, say, prisoner dilemma, an agent needs to perform an act of communication with another prisoner to learn her decision theory, which kills all the problem: if we can communicate, we can have some coordination strategy. In one shot prisoner’s dilemma we don’t know if the other side UDT or CDT agent, and other side also don’t know this about us. So the both are using similar lines of reasoning trying to guess if other agent is CDT or UDT. This similar reasoning itself could be a subject of UDT on meta-level, as we both would win more, if we assume that the other agent is UDT-agent.