But how does an agent introduce its values in the mixture?
I don’t accept Will’s overall position or reasoning but this particular part is relatively straightforward. It’s just the same as how anyone negotiates. In this case the negotiation is just a little… indirect. (Expanded below.)
The agent is the way it decides, so at least in one interpretation its values must be reflected in its decisions (reasons for its decisions), seen in them, even if in a different interpretation its decisions reflect the mixed values of all things (for that is one thing the agent might want to take into account, as it becomes more capable of doing so).
An agent’s decisions are determined by it’s values but this relationship is many to one. For any given circumstances that an agent could be in all sorts of preferences will end up resolving to the same decision. If you decided to throw away all that information by only considering what can be inferred from the resultant decision then you will end up wrong. More importantly this isn’t what the other agents will be doing so you will be wrong about them too.
Consider the coordination game as described by paulfchristiano, adopted as a metaphysics of morality by Will and that I’ll consider as a counterfactual:
There are a bunch of agents located beyond the range at which they can physically or causally interact. (This premise is sometimes includes altogether esoteric degrees of non-interaction.)
The values of each agent includes things that can be influenced by the other agents.
The agents have full awareness of both the values and decision procedures of all the other agents.
It is trivial* to see that this game reduces to equivalent to a simple two party prisoners dilemma with full mutual information. Each agent calculates the most efficient self interested bargains that could be made between them all and chooses to either act as if those bargains have been made or doesn’t depending on whether it (reliably) predicts the other agents do likewise.
For all the agents when we look at their behavior we see them all acting equivalently to whatever the negotiated outcome comes out to. That tells us little about their individual values—we’ve thrown that information away and just elected to keep “Cooperate with negotiated preferences”. But the individual preferences have been ‘thrown into the mix’ already back at the point where each of the agents considers the expected behavior of the others. (And there is no way that one of the agents will cooperate without it’s values in the mix and all the agents like to win, etc, etc, and a lot more ‘trivial’.)
I don’t accept the premises here and so definitely don’t accept any ‘universal morality’ but the “But how does an agent introduce its values in the mixture?” just isn’t the weakpoint of the reasoning. It’s tangent and to the extent that it is presented as objection it is a red herring.
It is trivial* to see that this game reduces to equivalent to a simple two party prisoners dilemma with full mutual information.
It only reduces to/is equivalent to a prisoner’s dilemma for certain utility functions (what you’re calling “values”). The prisoners’ dilemma is characterized by the fact that there is a dominant strategy equilibrium which is not Pareto optimal. But if the utility functions of the agents are such that the game is zero-sum, then this can’t be the case, as every outcome is Pareto optimal in a zero-sum game.
Furthermore, in a zero-sum game, no cooperation between all of the agents is possible. So it’s crazy to believe that an arbitrary set of sufficiently intelligent agents will cooperate to achieve a single “overgoal”. Collaboration is only possible if the agents’ preferences are such that collaboration can be mutually beneficial.
Yes, this entire scenario is based around scenarios where there is benefit to cooperation. In the edge case where such benefit is ‘0 expected utilons’ the behavior of the agents will, unsurprisingly, not be changed at all by the considerations we are talking about.
So I should interpret Will’s “Omega = objective morality” comment as meaning “sufficiently wise agents sometimes cooperate, when cooperation is the best way to achieve their ends”? I don’t think so.
So I should interpret Will’s “Omega = objective morality” comment as meaning “sufficiently wise agents sometimes cooperate, when cooperation is the best way to achieve their ends”?
No. Will thinks thought along these lines then goes ahead and bites imaginary bullets.
I didn’t intend to suggest throwing out information: a “public” decision, the action, is not the only decision that happens, and there is also “a priori” of agent’s whole initial construction. Rather, my point was that there is more to agent’s values than just the agent as it’s initially presented, with its future decisions marking the points where additional information (for the purposes of other decisions that coordinate) is being revealed, even if those decisions follow deterministically from agent’s initial construction.
I don’t accept Will’s overall position or reasoning but this particular part is relatively straightforward. It’s just the same as how anyone negotiates. In this case the negotiation is just a little… indirect. (Expanded below.)
An agent’s decisions are determined by it’s values but this relationship is many to one. For any given circumstances that an agent could be in all sorts of preferences will end up resolving to the same decision. If you decided to throw away all that information by only considering what can be inferred from the resultant decision then you will end up wrong. More importantly this isn’t what the other agents will be doing so you will be wrong about them too.
Consider the coordination game as described by paulfchristiano, adopted as a metaphysics of morality by Will and that I’ll consider as a counterfactual:
There are a bunch of agents located beyond the range at which they can physically or causally interact. (This premise is sometimes includes altogether esoteric degrees of non-interaction.)
The values of each agent includes things that can be influenced by the other agents.
The agents have full awareness of both the values and decision procedures of all the other agents.
It is trivial* to see that this game reduces to equivalent to a simple two party prisoners dilemma with full mutual information. Each agent calculates the most efficient self interested bargains that could be made between them all and chooses to either act as if those bargains have been made or doesn’t depending on whether it (reliably) predicts the other agents do likewise.
For all the agents when we look at their behavior we see them all acting equivalently to whatever the negotiated outcome comes out to. That tells us little about their individual values—we’ve thrown that information away and just elected to keep “Cooperate with negotiated preferences”. But the individual preferences have been ‘thrown into the mix’ already back at the point where each of the agents considers the expected behavior of the others. (And there is no way that one of the agents will cooperate without it’s values in the mix and all the agents like to win, etc, etc, and a lot more ‘trivial’.)
I don’t accept the premises here and so definitely don’t accept any ‘universal morality’ but the “But how does an agent introduce its values in the mixture?” just isn’t the weakpoint of the reasoning. It’s tangent and to the extent that it is presented as objection it is a red herring.
* In the come back 20 minutes later and say “Oh, it’s trivial” sense.
It only reduces to/is equivalent to a prisoner’s dilemma for certain utility functions (what you’re calling “values”). The prisoners’ dilemma is characterized by the fact that there is a dominant strategy equilibrium which is not Pareto optimal. But if the utility functions of the agents are such that the game is zero-sum, then this can’t be the case, as every outcome is Pareto optimal in a zero-sum game.
Furthermore, in a zero-sum game, no cooperation between all of the agents is possible. So it’s crazy to believe that an arbitrary set of sufficiently intelligent agents will cooperate to achieve a single “overgoal”. Collaboration is only possible if the agents’ preferences are such that collaboration can be mutually beneficial.
Yes, this entire scenario is based around scenarios where there is benefit to cooperation. In the edge case where such benefit is ‘0 expected utilons’ the behavior of the agents will, unsurprisingly, not be changed at all by the considerations we are talking about.
So I should interpret Will’s “Omega = objective morality” comment as meaning “sufficiently wise agents sometimes cooperate, when cooperation is the best way to achieve their ends”? I don’t think so.
No. Will thinks thought along these lines then goes ahead and bites imaginary bullets.
I don’t think that’s a very good model. Also, I’m curious: what’s your impression of this quote?
Worse than useless.
I didn’t intend to suggest throwing out information: a “public” decision, the action, is not the only decision that happens, and there is also “a priori” of agent’s whole initial construction. Rather, my point was that there is more to agent’s values than just the agent as it’s initially presented, with its future decisions marking the points where additional information (for the purposes of other decisions that coordinate) is being revealed, even if those decisions follow deterministically from agent’s initial construction.