But the SI consequentialists feel instead like a world model plus a random seed that identifies a location and a world model plus a random seed that happens to match the external world’s random seed, and this feels much smaller than a world model plus a random seed that happens to match the external world’s random seed.
Solomonoff induction has to “guess” the world model + random seed + location.
The consequentialist also have to guess the world model + random seed + location. The basic situation seems to be the same. If the consequentialists just picked a random program and ran it, then they’d do as well as Solomonoff induction (ignoring computational limitations). Of course, they would then have only a small amount of control over the predictions, since they only control a small fraction of the measure under Solomonoff induction.
The consequentialists can do better by restricting to “interesting” worlds+locations (i.e. those where someone is applying SI in a way that determines what happens with astronomical resources), e.g. by guessing again and again until they get an interesting one. I argue that the extra advantage they get, from restricting to interesting worlds+locations, is probably significantly larger than the 1/(fraction of measure they control). This is because the fraction of measure controlled by consequentialists is probably much larger than the fraction of measure corresponding to “interesting” invocations of SI.
(This argument ignores computational limitations of the consequentialists.)
The consequentialists can do better by restricting to “interesting” worlds+locations (i.e. those where someone is applying SI in a way that determines what happens with astronomical resources)
Ah, okay, now I see how the reasoning helps them, but it seems like there’s a strong form of this and a weak form of this.
The weak form is just that the consequentialists, rather than considering all coherent world models, can do some anthropic-style arguments about worldmodels in order to restrict their attention to world models that could in theory sustain intelligent life. This plausibly gives them an edge over ‘agent-free’ SI, which is naively spending measure on all possible world models, that covers the cost of having the universe that contains consequentialists. [It seems unlikely that it covers the cost of the consequentialists having to guess where their output channel is, unless this is also a cost paid by the ‘agent-free’ hypothesis?]
The strong form relates to the computational limitations of the consequentialists that you bring up—it seems like they have to solve something halting-problem like in order to determine that (given you correctly guessed the outer universe dynamics) a particular random seed leads to agents running SI and giving it large amounts of control, and so you probably can’t do search on random seeds (especially not if you want to seriously threaten completeness). This seems like a potentially more important source of reducing wasted measure, and so if the consequentialists didn’t have computational limitations then this would seem more important. [But it seems like most ways of giving them more computational resources also leads to an increase in the difficulty of them finding their output channel; perhaps there’s a theorem here? Not obvious this is more fruitful than just using about a speed prior instead.]
Solomonoff induction has to “guess” the world model + random seed + location.
The consequentialist also have to guess the world model + random seed + location. The basic situation seems to be the same. If the consequentialists just picked a random program and ran it, then they’d do as well as Solomonoff induction (ignoring computational limitations). Of course, they would then have only a small amount of control over the predictions, since they only control a small fraction of the measure under Solomonoff induction.
The consequentialists can do better by restricting to “interesting” worlds+locations (i.e. those where someone is applying SI in a way that determines what happens with astronomical resources), e.g. by guessing again and again until they get an interesting one. I argue that the extra advantage they get, from restricting to interesting worlds+locations, is probably significantly larger than the 1/(fraction of measure they control). This is because the fraction of measure controlled by consequentialists is probably much larger than the fraction of measure corresponding to “interesting” invocations of SI.
(This argument ignores computational limitations of the consequentialists.)
Ah, okay, now I see how the reasoning helps them, but it seems like there’s a strong form of this and a weak form of this.
The weak form is just that the consequentialists, rather than considering all coherent world models, can do some anthropic-style arguments about worldmodels in order to restrict their attention to world models that could in theory sustain intelligent life. This plausibly gives them an edge over ‘agent-free’ SI, which is naively spending measure on all possible world models, that covers the cost of having the universe that contains consequentialists. [It seems unlikely that it covers the cost of the consequentialists having to guess where their output channel is, unless this is also a cost paid by the ‘agent-free’ hypothesis?]
The strong form relates to the computational limitations of the consequentialists that you bring up—it seems like they have to solve something halting-problem like in order to determine that (given you correctly guessed the outer universe dynamics) a particular random seed leads to agents running SI and giving it large amounts of control, and so you probably can’t do search on random seeds (especially not if you want to seriously threaten completeness). This seems like a potentially more important source of reducing wasted measure, and so if the consequentialists didn’t have computational limitations then this would seem more important. [But it seems like most ways of giving them more computational resources also leads to an increase in the difficulty of them finding their output channel; perhaps there’s a theorem here? Not obvious this is more fruitful than just using about a speed prior instead.]