This paper by Critch is relevant: it argues that agents with different beliefs will bet their future share of a merged utility function, such that it skews towards whoever’s predictions are more correct.
I had read that paper recently, but I think we can abstract away from the issue by saying that (if merging is a thing for them) AIs will use decision procedures that are “closed under” merging, just like we currently focus attention on decision procedures that are “closed under” self-modification. (I suspect that modulo logical uncertainty, which Critch’s paper also ignores, UDT might already be such a decision procedure, in other words Critch’s argument doesn’t apply to UDT, but I haven’t spent much time thinking about it.)
This paper by Critch is relevant: it argues that agents with different beliefs will bet their future share of a merged utility function, such that it skews towards whoever’s predictions are more correct.
I had read that paper recently, but I think we can abstract away from the issue by saying that (if merging is a thing for them) AIs will use decision procedures that are “closed under” merging, just like we currently focus attention on decision procedures that are “closed under” self-modification. (I suspect that modulo logical uncertainty, which Critch’s paper also ignores, UDT might already be such a decision procedure, in other words Critch’s argument doesn’t apply to UDT, but I haven’t spent much time thinking about it.)