If you’re not vNM-coherent you will get Dutch-booked if there are Dutch-bookers around.
This especially applies to multipolar scenarios with AI systems in competition.
I have an intuition that this also applies in degrees: if you are more vNM-coherent than I am (which I think I can define), then I’d guess that you can Dutch-book me pretty easily.
My contention is that I don’t think the preconditions hold.
Agents don’t fail to be VNM coherent by having incoherent preferences given the axioms of VNM. They fail to be VNM coherent by violating the axioms themselves.
Completeness is wrong for humans, and with incomplete preferences you can be non exploitable even without admitting a single fixed utility function over world states.
I notice I am confused. How do you violate an axiom (completeness) without behaving in a way that violates completeness? I don’t think you need an internal representation.
Elaborating more, I am not sure how you even display a behavior that violates completeness. If you’re given a choice between only universe-histories a and b, and your preferences are imcomplete over them, what do you do? As soon as you reliably act to choose one over the other, for any such pair, you have algorithmically-revealed complete preferences.
If you don’t reliably choose one over the other, what do you do then?
Choose randomly? But then I’d guess you are again Dutch-bookable. And according to which distribution?
Your choice is undefined? That seems both kinda bad and also Dutch-bookable to me tbh. Alwo don’t see the difference between this and random choice (shodt of going up in flames, which would constigute a third, hitherto unassumed option).
Go away/refuse the trade &c? But this is denying the premise! You only have universe-histories a and b tp choose between! I think what happens with humans is that they are often incomplete over very low-ranking worlds and are instead searching for policies to find high-ranking worlds while not choosing. I think incomplwteness might be fine if there are two options you can guarantee to avoid, but with adversarial dynamics that becomes more and more difficult.
The interesting part is how systems/pre-agents/egregores/whatever become complete.
If it already satisfies the other VNM axioms we can analyse the situation as follows:
Recall that ain inexploitable but incomplete VNM agents acts like a Vetocracy of VNM agents. The exact decomposition is underspecified by just the preference order and is another piece of data (hidden state).
However, given sure-gain offers from the environment there is selection pressure for the internal complete VNM Subagents to make trade agreements to obtain a pareto improvement.
If you analyze this it looks like a simple prisoner dilemma type case which can be analyzed the usual way in game theory. For instance, in repeated offers with uncertain horizon the Subagents may be able to cooperate.
Once they are (approximately) complete they will be under selection pressure to satisfy the other axioms. You could say this the beginning of ‘emergence of expected utility maximizers’
As you can see the key here is that we really should be talking about Selection Theorems not the highly simplified Coherence Theorems. Coherence theorems are about ideal agents.
Selection theorems are about how more and more coherent and goal-directed agents may emerge.
If you’re not vNM-coherent you will get Dutch-booked if there are Dutch-bookers around.
This especially applies to multipolar scenarios with AI systems in competition.
I have an intuition that this also applies in degrees: if you are more vNM-coherent than I am (which I think I can define), then I’d guess that you can Dutch-book me pretty easily.
My contention is that I don’t think the preconditions hold.
Agents don’t fail to be VNM coherent by having incoherent preferences given the axioms of VNM. They fail to be VNM coherent by violating the axioms themselves.
Completeness is wrong for humans, and with incomplete preferences you can be non exploitable even without admitting a single fixed utility function over world states.
I notice I am confused. How do you violate an axiom (completeness) without behaving in a way that violates completeness? I don’t think you need an internal representation.
Elaborating more, I am not sure how you even display a behavior that violates completeness. If you’re given a choice between only universe-histories a and b, and your preferences are imcomplete over them, what do you do? As soon as you reliably act to choose one over the other, for any such pair, you have algorithmically-revealed complete preferences.
If you don’t reliably choose one over the other, what do you do then?
Choose randomly? But then I’d guess you are again Dutch-bookable. And according to which distribution?
Your choice is undefined? That seems both kinda bad and also Dutch-bookable to me tbh. Alwo don’t see the difference between this and random choice (shodt of going up in flames, which would constigute a third, hitherto unassumed option).
Go away/refuse the trade &c? But this is denying the premise! You only have universe-histories a and b tp choose between! I think what happens with humans is that they are often incomplete over very low-ranking worlds and are instead searching for policies to find high-ranking worlds while not choosing. I think incomplwteness might be fine if there are two options you can guarantee to avoid, but with adversarial dynamics that becomes more and more difficult.
If you define your utility function over histories, then every behaviour is maximising an expected utility function no?
Even behaviour that is money pumped?
I mean you can’t money pump any preference over histories anyway without time travel.
The Dutchbook arguments apply when your utility function is defined over your current state with respect to some resource?
I feel like once you define utility function over histories, you lose the force of the coherence arguments?
What would it look like to not behave as if maximising an expected utility function for a utility function defined over histories.
Agree.
There are three stages:
Selection for inexploitability
The interesting part is how systems/pre-agents/egregores/whatever become complete.
If it already satisfies the other VNM axioms we can analyse the situation as follows: Recall that ain inexploitable but incomplete VNM agents acts like a Vetocracy of VNM agents. The exact decomposition is underspecified by just the preference order and is another piece of data (hidden state). However, given sure-gain offers from the environment there is selection pressure for the internal complete VNM Subagents to make trade agreements to obtain a pareto improvement. If you analyze this it looks like a simple prisoner dilemma type case which can be analyzed the usual way in game theory. For instance, in repeated offers with uncertain horizon the Subagents may be able to cooperate.
Once they are (approximately) complete they will be under selection pressure to satisfy the other axioms. You could say this the beginning of ‘emergence of expected utility maximizers’
As you can see the key here is that we really should be talking about Selection Theorems not the highly simplified Coherence Theorems. Coherence theorems are about ideal agents. Selection theorems are about how more and more coherent and goal-directed agents may emerge.