Rohin Shah comments on [AN #81]: Universality as a potential solution to conceptual difficulties in intent alignment

Rohin Shah 13 Jan 2020 1:56 UTC
LW: 4 AF: 3
AF
In this, belief set A and belief set B are analogous to A[C] and C (or some c in C), right?
Yes.
If we replace our beliefs with A[C]’s, then how is that us trusting it “over” c or C? It seems like it’s us trusting it, full stop
So I only showed the case where $i n f o$ contains information about $A [C]$ ’s predictions, but $i n f o$ is allowed to contain information from $A [C]$ and $C$ (but not other agents). Even if it contains lots of information from C, we still need to trust $A [C]$ .
In contrast, if $i n f o$ contained information about $A [A [C]]$ ’s beliefs, then we would not trust $A [C]$ over that.