The consequentialists can do better by restricting to “interesting” worlds+locations (i.e. those where someone is applying SI in a way that determines what happens with astronomical resources)
Ah, okay, now I see how the reasoning helps them, but it seems like there’s a strong form of this and a weak form of this.
The weak form is just that the consequentialists, rather than considering all coherent world models, can do some anthropic-style arguments about worldmodels in order to restrict their attention to world models that could in theory sustain intelligent life. This plausibly gives them an edge over ‘agent-free’ SI, which is naively spending measure on all possible world models, that covers the cost of having the universe that contains consequentialists. [It seems unlikely that it covers the cost of the consequentialists having to guess where their output channel is, unless this is also a cost paid by the ‘agent-free’ hypothesis?]
The strong form relates to the computational limitations of the consequentialists that you bring up—it seems like they have to solve something halting-problem like in order to determine that (given you correctly guessed the outer universe dynamics) a particular random seed leads to agents running SI and giving it large amounts of control, and so you probably can’t do search on random seeds (especially not if you want to seriously threaten completeness). This seems like a potentially more important source of reducing wasted measure, and so if the consequentialists didn’t have computational limitations then this would seem more important. [But it seems like most ways of giving them more computational resources also leads to an increase in the difficulty of them finding their output channel; perhaps there’s a theorem here? Not obvious this is more fruitful than just using about a speed prior instead.]
Ah, okay, now I see how the reasoning helps them, but it seems like there’s a strong form of this and a weak form of this.
The weak form is just that the consequentialists, rather than considering all coherent world models, can do some anthropic-style arguments about worldmodels in order to restrict their attention to world models that could in theory sustain intelligent life. This plausibly gives them an edge over ‘agent-free’ SI, which is naively spending measure on all possible world models, that covers the cost of having the universe that contains consequentialists. [It seems unlikely that it covers the cost of the consequentialists having to guess where their output channel is, unless this is also a cost paid by the ‘agent-free’ hypothesis?]
The strong form relates to the computational limitations of the consequentialists that you bring up—it seems like they have to solve something halting-problem like in order to determine that (given you correctly guessed the outer universe dynamics) a particular random seed leads to agents running SI and giving it large amounts of control, and so you probably can’t do search on random seeds (especially not if you want to seriously threaten completeness). This seems like a potentially more important source of reducing wasted measure, and so if the consequentialists didn’t have computational limitations then this would seem more important. [But it seems like most ways of giving them more computational resources also leads to an increase in the difficulty of them finding their output channel; perhaps there’s a theorem here? Not obvious this is more fruitful than just using about a speed prior instead.]