DavidHolmes comments on Toy model piece #1: Partial preferences revisited

[DavidHolmes]

This seems really neat, but it seems quite sensitive to how one defines the worlds under consideration, and whether one counts slightly different worlds as actually distinct. Let me try to illustrate this with an example.

Suppose we have a $W$ consisting of 7 worlds, $W=\{A,B,C,D,X,Y,W\}$, with preferences
$$A<B<C<D,X<Y<Z$$ and no other non-trivial preferences. Then (from the `sensible case’), I think we get the following utilities:
$A\mapsto -3$
$X\mapsto -2$
$B\mapsto -1$
$Y\mapsto 0$
$C\mapsto 1$
$Z\mapsto 2$
$D\mapsto 3$.

Suppose now that I create two new copies $X^{\prime }$, $X^{\prime \prime }$ of the world $X$ which each differ by the position of a single atom, so as to give me (extremely weak!) preferences $X^{\prime \prime }<X^{\prime }<X$, so all the non-trivial preferences in the new $W$ are now summarised as $$A<B<C<D,X^{\prime \prime }<X^{\prime }<X<Y<Z.$$

Then the resulting utilities are (I think):
$X^{\prime \prime }\mapsto -4$
$A\mapsto -3$
$X^{\prime }\mapsto -2$
$B\mapsto -1$
$X\mapsto 0$
$C\mapsto 1$
$Y\mapsto 2$
$D\mapsto 3$
$Z\mapsto 4$.

In particular, before adding in these ‘trivial copies’ we had $U\left(Z\right)<U\left(D\right)$, and now we get $U\left(D\right)<U\left(Z\right)$. Is this a problem? It depends on the situation, but to me it suggests that, if using this approach, one needs to be careful in how the worlds are specified, and the ‘fine-grainedness’ needs to be roughly the same everywhere.

[Stuart_Armstrong]

Each partial preference is meant to represent a single mental model inside the human, with all preferences weighted the same (so there can’t be “extremely weak” preferences, compared with other preference in the same partial preference). Things like “increased income is better”, “more people smiling is better”, “being embarrassed on stage is the worse”.

We can imagine a partial preference with more internal structure, maybe internal weights, but I’d simply see that as two separate partial preferences. So we’d have the utilities you gave to $A$ through to $Z$ for one partial preference (actually, my formula doubles the numbers you gave), and $X^{\prime \prime }\to -2$, $X^{\prime }\to 0$, $X\to 2$ for the other partial preference—which has a very low weight by assumption. So the order of $Z$ and $D$ is not affected.

EDIT: I’m pretty sure we can generalise my method for different weights of preferences, by changing the formula that sums the squares of utility difference.

[DavidHolmes]

(actually, my formula doubles the numbers you gave)

Are you sure? Suppose we take $W=W_1\bigsqcup W_2$ with $W_1=\{A,B,C,D\}$, $W_2=\{X,Y,Z\}$, then $n_1=3$, so the values for $W_1$ should be $-3,-1,1,3$ as I gave them. And similarly for $W_2$, giving values $-2,0,2$. Or else I have mis-understood your definition?

I’d simply see that as two separate partial preferences

Just to be clear, by “separate partial preference” you mean a separate preorder, on a set of objects which may or may not have some overlap with the objects we considered so far? Then somehow the work is just postponed to the point where we try to combine partial preferences?

EDIT (in reply to your edit): I guess e.g. keeping conditions 1,2,3 the same and instead minimising
$g\left(G\right)={\sum }_{w\leftarrow w^{\prime }}{\lambda }_{w\leftarrow w^{\prime }}\left(U\right(w^{\prime })-U(w){)}^2,$
where ${\lambda }_{w\leftarrow w^{\prime }}\in R_{>0}$ is proportion to the reciprocal of the strength of the preference? Of course there are lots of variants on this!

[Stuart_Armstrong]

Yep, sorry, I saw −3, −2, −1, etc… and concluded you weren’t doing the 2 jumps; my bad!

Then somehow the work is just postponed to the point where we try to combine partial preferences?

Yes. But unless we have other partial preferences or meta-preferences, then the only resonable way of combining them is just to add them, after weighting.

I like your reciprocal weighting formula. It seems to have good properties.