A note—it looks like what Eliezer is suggesting here is not the same as UDASSA. See my analysis here—and endoself’s reply—and here.
The big difference is that UDASSA won’t impose the same locational penalty on nodes in extreme situations, since the measure is shared unequally between nodes. There are programs q with relatively short length that can select out such extreme nodes (parties getting genuine offers from Matrix Lords with the power of 3^^^3) and so give them much higher relative weight than 1/3^^^3. Combine this with an unbounded utility, and the mugger problem is still there (as is the divergence in expected utility).
I agree that what Eliezer described is not exactly UDASSA. At first I thought it was just like UDASSA but with a speed prior, but now I see that that’s wrong. I suspect it ends up being within a constant factor of UDASSA, just by considering universes with tiny little demons that go around duplicating all of the observers a bunch of times.
If you are using UDT, the role of UDASSA (or any anthropic theory) is in the definition of the utility function. We define a measure over observers, so that we can say how good a state of affairs is (by looking at the total goodness under that measure). In the case of UDASSA the utility is guaranteed to be bounded, because our measure is a probability measure. Similarly, there doesn’t seem to be a mugging issue.
As lukeprog says here, this really needs to be written up. It’s not clear to me that just because the measure over observers (or observer moments) sums to one then the expected utility is bounded.
Here’s a stab. Let’s use s to denote a sub-program of a universe program p, following the notation of my other comment. Each s gets a weight w(s) under UDASSA, and we normalize to ensure Sum{s} w(s) = 1.
Then, presumably, an expected utility looks like E(U) = Sum{s} U(s) w(s), and this is clearly bounded provided the utility U(s) for each observer moment s is bounded (and U(s) = 0 for any sub-program which isn’t an “observer moment”).
But why is U(s) bounded? It doesn’t seem obvious to me (perhaps observer moments can be arbitrarily blissful, rather than saturating at some state of pure bliss). Also, what happens if U bears no relationship to experiences/observer moments, but just counts the number of paperclips in the universe p? That’s not going to be bounded, is it?
A note—it looks like what Eliezer is suggesting here is not the same as UDASSA. See my analysis here—and endoself’s reply—and here.
The big difference is that UDASSA won’t impose the same locational penalty on nodes in extreme situations, since the measure is shared unequally between nodes. There are programs q with relatively short length that can select out such extreme nodes (parties getting genuine offers from Matrix Lords with the power of 3^^^3) and so give them much higher relative weight than 1/3^^^3. Combine this with an unbounded utility, and the mugger problem is still there (as is the divergence in expected utility).
I agree that what Eliezer described is not exactly UDASSA. At first I thought it was just like UDASSA but with a speed prior, but now I see that that’s wrong. I suspect it ends up being within a constant factor of UDASSA, just by considering universes with tiny little demons that go around duplicating all of the observers a bunch of times.
If you are using UDT, the role of UDASSA (or any anthropic theory) is in the definition of the utility function. We define a measure over observers, so that we can say how good a state of affairs is (by looking at the total goodness under that measure). In the case of UDASSA the utility is guaranteed to be bounded, because our measure is a probability measure. Similarly, there doesn’t seem to be a mugging issue.
As lukeprog says here, this really needs to be written up. It’s not clear to me that just because the measure over observers (or observer moments) sums to one then the expected utility is bounded.
Here’s a stab. Let’s use s to denote a sub-program of a universe program p, following the notation of my other comment. Each s gets a weight w(s) under UDASSA, and we normalize to ensure Sum{s} w(s) = 1.
Then, presumably, an expected utility looks like E(U) = Sum{s} U(s) w(s), and this is clearly bounded provided the utility U(s) for each observer moment s is bounded (and U(s) = 0 for any sub-program which isn’t an “observer moment”).
But why is U(s) bounded? It doesn’t seem obvious to me (perhaps observer moments can be arbitrarily blissful, rather than saturating at some state of pure bliss). Also, what happens if U bears no relationship to experiences/observer moments, but just counts the number of paperclips in the universe p? That’s not going to be bounded, is it?
I agree it would be nice if things were better written up; right now there is the description I linked and Hal Finney’s.
If individual moments can be arbitrarily good, then I agree you have unbounded utilities again.
If you count the number of paperclips you would again get into trouble; the analogous thing to do would be to count the mesure of paperclips.