I wasn’t expressing skepticism that unlosing agents exist, only that they would be VNM-rational. Aside from the example I described in the linked comment about how such an agent could violate the independence axiom, it sounds like the agent could also violate transitivity. For example, suppose there are 3 outcomes A, B, and C, and that P says A>B, B>C, and C>A. If given a choice between A and B, the agent chooses A. If it is given an opportunity to switch to C after that, and then an opportunity to switch to B again after that, it will avoid getting stuck in a loop. But that doesn’t remove the problem that, before any of that, it would pick A if offered a choice between A and B, B if offered a choice between B and C, and C if offered a choice between A and C. This still seems pretty bad, even though it doesn’t get caught in dutch-book loops.
But if the agent can’t be subject to Dutch books, what’s the point of being VNM-rational? (in fact, in my construction, the agent need not be initially complete).
But the main point is that VNM-rational isn’t clearly defined. Is it over all possible decisions, or just over decisions the agent actually faces? Given that rationality is often defined on Less Wrong in a very practical way (generalised “winning”) I see no reason to need to assume the first. It weakens the arguments for VNM-rationality, makes it into a philosophical ideal rather than a practical tool.
And so while it’s clear that an AI would want to make itself into an unlosing agent, it’s less clear that it would want to make itself into an expected utility maximiser. In fact, it’s very clear that in some cases it wouldn’t: if it knew that outcomes A and B were impossible, and it currently didn’t have preferences between them, then there is no reason it would ever bother to develop preferences there (baring social signalling and similar).
Suppose you have A>B>C>A, with at least a $1 gap at each step of the preference ordering. Consider these 3 options:
Option 1: I randomly assign you to get A, B, or C Option 2: I randomly assign you to get A, B, or C, then I give you the option of paying $1 to switch from A to C (or C to B, or B to A), and then I give you the option of paying $1 to switch again Option 3: I take $2 from you and randomly assign you to get A, B, or C
Under standard utility theory Option 2 dominates Option 1, which in turn strictly dominates Option 3. But for an unlosing agent which initially has cyclic preferences, Option 2 winds up being equivalent to Option 3.
Incidentally, if given the choice, the agent would choose option 1 over option 3. When making choices, unlosing agents are indistinguishable from vNM expected utility maximisers.
Or another way of seeing it, the unlosing agent could have three utility functions remaining: A>B>C, B>C>A, and C>A>B, and all of these would prefer option 1 to option 3.
What’s more interesting about your example is that it shows that certain ways of breaking transitivities are better than others.
There’s actually no need to settle for finite truncations of a decision agent. The unlosing decision function (on lotteries) can be defined in first-order logic, and your proof that there are finite approximations of a decision function is sufficient to use the compactness theorem to produce a full model.
You avoid falling into Dutch book loops where you iterately pay to go around in a circle at each step, but you still fall into single-step Dutch books. Unnamed gave a good example.
Those aren’t technically Dutch Books. And there’s no reason a forward-looking unlosing agent couldn’t break circles at the beginning rather than at the end.
Ok, but it is still an example of the agent choosing a lottery over a strictly better one.
And there’s no reason a forward-looking unlosing agent couldn’t break circles at the beginning rather than at the end.
Then it would be VNM-rational. Completeness is necessary to make sense as an agent, transitivity and independence are necessary to avoid making choices strictly dominated by other options, and the Archimedian axiom really isn’t all that important.
Unnamed’s example is interesting. But if given the choice, unlosing agents would chose option 1 over option 3 (when choosing, unlosing agents act as vNM maximisers).
Unnamed example points at something different, namely that certain ways of resolving intransitives are strictly better than others.
I wasn’t expressing skepticism that unlosing agents exist, only that they would be VNM-rational. Aside from the example I described in the linked comment about how such an agent could violate the independence axiom, it sounds like the agent could also violate transitivity. For example, suppose there are 3 outcomes A, B, and C, and that P says A>B, B>C, and C>A. If given a choice between A and B, the agent chooses A. If it is given an opportunity to switch to C after that, and then an opportunity to switch to B again after that, it will avoid getting stuck in a loop. But that doesn’t remove the problem that, before any of that, it would pick A if offered a choice between A and B, B if offered a choice between B and C, and C if offered a choice between A and C. This still seems pretty bad, even though it doesn’t get caught in dutch-book loops.
But if the agent can’t be subject to Dutch books, what’s the point of being VNM-rational? (in fact, in my construction, the agent need not be initially complete).
But the main point is that VNM-rational isn’t clearly defined. Is it over all possible decisions, or just over decisions the agent actually faces? Given that rationality is often defined on Less Wrong in a very practical way (generalised “winning”) I see no reason to need to assume the first. It weakens the arguments for VNM-rationality, makes it into a philosophical ideal rather than a practical tool.
And so while it’s clear that an AI would want to make itself into an unlosing agent, it’s less clear that it would want to make itself into an expected utility maximiser. In fact, it’s very clear that in some cases it wouldn’t: if it knew that outcomes A and B were impossible, and it currently didn’t have preferences between them, then there is no reason it would ever bother to develop preferences there (baring social signalling and similar).
Suppose you have A>B>C>A, with at least a $1 gap at each step of the preference ordering. Consider these 3 options:
Option 1: I randomly assign you to get A, B, or C
Option 2: I randomly assign you to get A, B, or C, then I give you the option of paying $1 to switch from A to C (or C to B, or B to A), and then I give you the option of paying $1 to switch again
Option 3: I take $2 from you and randomly assign you to get A, B, or C
Under standard utility theory Option 2 dominates Option 1, which in turn strictly dominates Option 3. But for an unlosing agent which initially has cyclic preferences, Option 2 winds up being equivalent to Option 3.
Incidentally, if given the choice, the agent would choose option 1 over option 3. When making choices, unlosing agents are indistinguishable from vNM expected utility maximisers.
Or another way of seeing it, the unlosing agent could have three utility functions remaining: A>B>C, B>C>A, and C>A>B, and all of these would prefer option 1 to option 3.
What’s more interesting about your example is that it shows that certain ways of breaking transitivities are better than others.
Which is a good argument to break circles early, rather than late.
There’s actually no need to settle for finite truncations of a decision agent. The unlosing decision function (on lotteries) can be defined in first-order logic, and your proof that there are finite approximations of a decision function is sufficient to use the compactness theorem to produce a full model.
You avoid falling into Dutch book loops where you iterately pay to go around in a circle at each step, but you still fall into single-step Dutch books. Unnamed gave a good example.
Those aren’t technically Dutch Books. And there’s no reason a forward-looking unlosing agent couldn’t break circles at the beginning rather than at the end.
Ok, but it is still an example of the agent choosing a lottery over a strictly better one.
Then it would be VNM-rational. Completeness is necessary to make sense as an agent, transitivity and independence are necessary to avoid making choices strictly dominated by other options, and the Archimedian axiom really isn’t all that important.
Unnamed’s example is interesting. But if given the choice, unlosing agents would chose option 1 over option 3 (when choosing, unlosing agents act as vNM maximisers).
Unnamed example points at something different, namely that certain ways of resolving intransitives are strictly better than others.