Ah, I was just being an idiot on the bargaining system w.r.t. small numbers of people being able to hold it to ransom. Oops. Agreed that more majority power isn’t desirable. [re iteration, I only meant that the bargaining could become iterated if the initial bargaining result were to decide upon iteration (to include more future users). I now don’t think this is particularly significant.]
I think my remaining uncertainty (/confusion) is all related to the issue I first mentioned (embedded copy experiences). It strikes me that something like this can also happen where minds grow/merge/overlap.
This operator will declare both the manifesting and evaluation of the source codes of other users to be “out of scope” for a given user. Hence, a preference of i to observe the suffering of j would be “satisfied” by observing nearly anything, since the maximization can interpret anything as a simulation of j.
Does this avoid the problem if i’s preferences use indirection? It seems to me that a robust pointer to j may be enough: that with a robust pointer it may be possible to implicitly require something like source-code-access without explicitly referencing it. E.g. where i has a preference to “experience j suffering in circumstances where there’s strong evidence it’s actually j suffering, given that these circumstances were the outcome of this bargaining process”.
If i can’t robustly specify things like this, then I’d guess there’d be significant trouble in specifying quite a few (mutually) desirable situations involving other users too. IIUC, this would only be any problem for the denosed bargaining to find a good d1: for the second bargaining on the true utility functions there’s no need to put anything “out of scope” (right?), so win-wins are easily achieved.
Ah, I was just being an idiot on the bargaining system w.r.t. small numbers of people being able to hold it to ransom. Oops. Agreed that more majority power isn’t desirable.
[re iteration, I only meant that the bargaining could become iterated if the initial bargaining result were to decide upon iteration (to include more future users). I now don’t think this is particularly significant.]
I think my remaining uncertainty (/confusion) is all related to the issue I first mentioned (embedded copy experiences). It strikes me that something like this can also happen where minds grow/merge/overlap.
Does this avoid the problem if i’s preferences use indirection? It seems to me that a robust pointer to j may be enough: that with a robust pointer it may be possible to implicitly require something like source-code-access without explicitly referencing it. E.g. where i has a preference to “experience j suffering in circumstances where there’s strong evidence it’s actually j suffering, given that these circumstances were the outcome of this bargaining process”.
If i can’t robustly specify things like this, then I’d guess there’d be significant trouble in specifying quite a few (mutually) desirable situations involving other users too. IIUC, this would only be any problem for the denosed bargaining to find a good d1: for the second bargaining on the true utility functions there’s no need to put anything “out of scope” (right?), so win-wins are easily achieved.