The easiest way to specify an important prediction problem (in the sense of a prediction that would be valuable for someone to influence) is likely to be by saying “Run the following Turing machine, then pick an important decision from within it.” Let’s say the complexity of that specification is N bits.
You think that if consequentialists dedicate some fraction of their resources to doing something that’s easy for the universal prior to output, it will still likely take more than N bits or not much less.
[Probably] You think the differences may be small enough that they can be influenced by factors of 1/1000 or 1/billion (i.e. 10-30 bits) of improbability of consequentialists spending significant resources in this task.
[Probably] You think the TM-definition update (where the manipulators get to focus on inductors who put high probability on their own universe) or the philosophical sophistication update (where manipulators use the “right” prior over possible worlds rather than choosing some programming language) are small relative to these other considerations.
I think the biggest disagreement is about 1+2. It feels implausible to me that “sample a data stream that is being used by someone to make predictions that would be valuable to manipulate” is simpler than any of the other extraction procedures that consequentialists could manipulate (like sample the sequence that appears the most times, sample the highest energy experiments, sample the weirdest thing on some other axis...)
But suppose they picked only one string to try to manipulate. The cost would go way down, but then it probably wouldn’t be us that they hit.
I think we’re probably on the same page now, but I’d say: the consequentialists can also sample from the “important predictions” prior (i.e. the same thing as that fragment of the universal prior). If “sample output channel controlled by consequentialists” has higher probability than “Sample an important prediction,” then the consequentialists control every important prediction. If on the other hand “Sample an important prediction” has higher probability than the consequentialists, I guess maybe they could take over a few predictions, but unless they were super close it would be a tiny fraction and I agree we wouldn’t care.
I think with 4, I’ve been assuming for the sake of argument that manipulators get free access to the right prior, and I don’t have a strong stance on the question, but it’s not complicated for a directly programmed anthropic update to be built on that right prior too.
I guess I can give some estimates for how many bits I think are required for each of the rows in the table. I’ll give a point estimate, and a range for a 50% confidence interval for what my point estimate would be if I thought about it for an hour by myself and had to write up my thinking along the way.
I don’t have a good sense for how many bits it takes to get past things that are just extremely basic, like an empty string, or an infinite string of 0s. But whatever that number is, add it to both 1 and 6.
1) Consequentialists emerge, 10 − 50 bits; point estimate 18
2) TM output has not yet begun, 10 − 30 bits; point estimate 18
3) make good guesses about controllable output, 18 − 150 bits; point estimate 40
4) decide to output anthropically updated prior, 8 − 35 bits; point estimate 15
5) decide to do a treacherous turn. 1 − 12 bits; point estimate 5
vs. 6) direct program for anthropic update. 18-100 bits; point estimate 30
By (3) do you mean the same thing as “Simplest output channel that is controllable by advanced civilization with modest resources”?
I assume (6) means that your “anthropic update” scans across possible universes to find those that contain important decisions you might want to influence?
If you want to compare most easily to models like that, then instead of using (1)+(2)+(3) you should compare to (6′) = “Simplest program that scans across many possible worlds to find those that contain some pattern that can be engineered by consequentialists trying to influence prior.”
Then the comparison is between specifying “important predictor to influence” and whatever the easiest-to-specify pattern that can be engineered by a consequentialist. It feels extremely likely to me that the second category is easier, indeed it’s kind of hard for me to see any version of (6) that doesn’t have an obviously simpler analog that could be engineered by a sophisticated civilization.
With respect to (4)+(5), I guess you are saying that your point estimate is that only 1/million of consequentialists decide to try to influence the universal prior. I find that surprisingly low but not totally indefensible, and it depends on exactly how expensive this kind of influence is. I also don’t really see why you are splitting them apart, shouldn’t we just combine them into “wants to influence predictors”? If you’re doing that presumably you’d both use the anthropic prior and then the treacherous turn.
But it’s also worth noting that (6′) gets to largely skip (4′) if it can search for some feature that is mostly brought about deliberately by consequentialists (who are trying to create a beacon recognizable by some program that scans across possible worlds looking for it, doing the same thing that “predictor that influences the future” is doing in (6)).
I assume (6) means that your “anthropic update” scans across possible universes to find those that contain important decisions you might want to influence?
Yes, and then outputs strings from that set with probability proportional to their weight in the universal prior.
By (3) do you mean the same thing as “Simplest output channel that is controllable by advanced civilization with modest resources”?
I would say “successfully controlled” instead of controllable, although that may be what you meant by the term. (I decomposed this as controllable + making good guesses.) For some definitions of controllable, I might have given a point estimate of maybe 1 or 5 bits. But there has to be an output channel for which the way you transmit a bitstring out is the way the evolved consequentialists expect. But recasting it in these terms, implicitly makes the suggestion that the specification of the output channel can take on some of the character of (6′), makes me want to put my range down to 15-60; point estimate 25.
instead of using (1)+(2)+(3) you should compare to (6′) = “Simplest program that scans across many possible worlds to find those that contain some pattern that can be engineered by consequentialists trying to influence prior.”
Similarly, I would replace “can be” with “seems to have been”. And just to make sure we’re talking about the same thing, it takes this list of patterns, and outputs them with probability proportional to their weight in the universal prior.
Yeah, this seems like it would make some significant savings compared to (1)+(2)+(3). I think replacing parts of the story from being specified as [arising from natural world dynamics] to being specified as [picked out “deliberately” by a program] generally leads to savings.
Then the comparison is between specifying “important predictor to influence” and whatever the easiest-to-specify pattern that can be engineered by a consequentialist. It feels extremely likely to me that the second category is easier, indeed it’s kind of hard for me to see any version of (6) that doesn’t have an obviously simpler analog that could be engineered by a sophisticated civilization.
I don’t quite understand the sense in which [worlds with consequentialist beacons/geoglyphs] can be described as [easiest-to-specify controllable pattern]. (And if you accept the change of “can be” to “seems to have been”, it propagates here). Scanning for important predictors to influence does feel very similar to me to scanning for consequentialist beacons, especially since the important worlds are plausibly the ones with consequentialists.
There’s a bit more work to be done in (6′) besides just scanning for consequentialist beacons. If the output channel is selected “conveniently” for the consequentialists, since the program is looking for the beacons, instead of the consequentialists giving it their best guess(es) and putting up a bunch of beacons, there has to be some part of the program which aggregates the information of multiple beacons (by searching for coherence, e.g.), or else determines which beacon takes precedence, and then also determines how to interpret their physical signature as a bitstring.
Tangent: in heading down a path trying to compare [scan for “important to influence”] vs. [scan for “consequentialist attempted output messages”] just now, my first attempt had an error, so I’ll point it out. It’s not necessarily harder to specify “scan for X” than “scan for Y” when X is a subset of Y. For instance “scan for primes” is probably simpler than “scan for numbers with less than 6 factors”.
Maybe clarifying or recasting the language around “easiest-to-specify controllable pattern” will clear this up, but can you explain more why it feels to you that [scan for “consequentialists’ attempted output messages”] is so much simpler than [scan for “important-to-influence data streams”]? My very preliminary first take is that they are within 8-15 bits.
I also don’t really see why you are splitting them [(4) + (5)] apart, shouldn’t we just combine them into “wants to influence predictors”? If you’re doing that presumably you’d both use the anthropic prior and then the treacherous turn.
I split them in part in case there is there is a contingent of consequentialists who believes that outputting the right bitstring is key to their continued existence, believing that they stop being simulated if they output the wrong bit. I haven’t responded to your claim that this would be faulty metapyhsics on their part; it still seems fairly tangential to our main discussion. But you can interpret my 5 bit point estimate for (5) as claiming that 31 times out of 32 that a civilization of consequentialists tries to influence their world’s output, it is in an attempt to survive. Tell me if you’re interested in a longer justification that responds to your original “line by line comments” comment.
Here’s my current understanding of your position:
The easiest way to specify an important prediction problem (in the sense of a prediction that would be valuable for someone to influence) is likely to be by saying “Run the following Turing machine, then pick an important decision from within it.” Let’s say the complexity of that specification is N bits.
You think that if consequentialists dedicate some fraction of their resources to doing something that’s easy for the universal prior to output, it will still likely take more than N bits or not much less.
[Probably] You think the differences may be small enough that they can be influenced by factors of 1/1000 or 1/billion (i.e. 10-30 bits) of improbability of consequentialists spending significant resources in this task.
[Probably] You think the TM-definition update (where the manipulators get to focus on inductors who put high probability on their own universe) or the philosophical sophistication update (where manipulators use the “right” prior over possible worlds rather than choosing some programming language) are small relative to these other considerations.
I think the biggest disagreement is about 1+2. It feels implausible to me that “sample a data stream that is being used by someone to make predictions that would be valuable to manipulate” is simpler than any of the other extraction procedures that consequentialists could manipulate (like sample the sequence that appears the most times, sample the highest energy experiments, sample the weirdest thing on some other axis...)
I think we’re probably on the same page now, but I’d say: the consequentialists can also sample from the “important predictions” prior (i.e. the same thing as that fragment of the universal prior). If “sample output channel controlled by consequentialists” has higher probability than “Sample an important prediction,” then the consequentialists control every important prediction. If on the other hand “Sample an important prediction” has higher probability than the consequentialists, I guess maybe they could take over a few predictions, but unless they were super close it would be a tiny fraction and I agree we wouldn’t care.
Yeah, seems about right.
I think with 4, I’ve been assuming for the sake of argument that manipulators get free access to the right prior, and I don’t have a strong stance on the question, but it’s not complicated for a directly programmed anthropic update to be built on that right prior too.
I guess I can give some estimates for how many bits I think are required for each of the rows in the table. I’ll give a point estimate, and a range for a 50% confidence interval for what my point estimate would be if I thought about it for an hour by myself and had to write up my thinking along the way.
I don’t have a good sense for how many bits it takes to get past things that are just extremely basic, like an empty string, or an infinite string of 0s. But whatever that number is, add it to both 1 and 6.
1) Consequentialists emerge, 10 − 50 bits; point estimate 18
2) TM output has not yet begun, 10 − 30 bits; point estimate 18
3) make good guesses about controllable output, 18 − 150 bits; point estimate 40
4) decide to output anthropically updated prior, 8 − 35 bits; point estimate 15
5) decide to do a treacherous turn. 1 − 12 bits; point estimate 5
vs. 6) direct program for anthropic update. 18-100 bits; point estimate 30
The ranges are fairly correlated.
By (3) do you mean the same thing as “Simplest output channel that is controllable by advanced civilization with modest resources”?
I assume (6) means that your “anthropic update” scans across possible universes to find those that contain important decisions you might want to influence?
If you want to compare most easily to models like that, then instead of using (1)+(2)+(3) you should compare to (6′) = “Simplest program that scans across many possible worlds to find those that contain some pattern that can be engineered by consequentialists trying to influence prior.”
Then the comparison is between specifying “important predictor to influence” and whatever the easiest-to-specify pattern that can be engineered by a consequentialist. It feels extremely likely to me that the second category is easier, indeed it’s kind of hard for me to see any version of (6) that doesn’t have an obviously simpler analog that could be engineered by a sophisticated civilization.
With respect to (4)+(5), I guess you are saying that your point estimate is that only 1/million of consequentialists decide to try to influence the universal prior. I find that surprisingly low but not totally indefensible, and it depends on exactly how expensive this kind of influence is. I also don’t really see why you are splitting them apart, shouldn’t we just combine them into “wants to influence predictors”? If you’re doing that presumably you’d both use the anthropic prior and then the treacherous turn.
But it’s also worth noting that (6′) gets to largely skip (4′) if it can search for some feature that is mostly brought about deliberately by consequentialists (who are trying to create a beacon recognizable by some program that scans across possible worlds looking for it, doing the same thing that “predictor that influences the future” is doing in (6)).
Yes, and then outputs strings from that set with probability proportional to their weight in the universal prior.
I would say “successfully controlled” instead of controllable, although that may be what you meant by the term. (I decomposed this as controllable + making good guesses.) For some definitions of controllable, I might have given a point estimate of maybe 1 or 5 bits. But there has to be an output channel for which the way you transmit a bitstring out is the way the evolved consequentialists expect. But recasting it in these terms, implicitly makes the suggestion that the specification of the output channel can take on some of the character of (6′), makes me want to put my range down to 15-60; point estimate 25.
Similarly, I would replace “can be” with “seems to have been”. And just to make sure we’re talking about the same thing, it takes this list of patterns, and outputs them with probability proportional to their weight in the universal prior.
Yeah, this seems like it would make some significant savings compared to (1)+(2)+(3). I think replacing parts of the story from being specified as [arising from natural world dynamics] to being specified as [picked out “deliberately” by a program] generally leads to savings.
I don’t quite understand the sense in which [worlds with consequentialist beacons/geoglyphs] can be described as [easiest-to-specify controllable pattern]. (And if you accept the change of “can be” to “seems to have been”, it propagates here). Scanning for important predictors to influence does feel very similar to me to scanning for consequentialist beacons, especially since the important worlds are plausibly the ones with consequentialists.
There’s a bit more work to be done in (6′) besides just scanning for consequentialist beacons. If the output channel is selected “conveniently” for the consequentialists, since the program is looking for the beacons, instead of the consequentialists giving it their best guess(es) and putting up a bunch of beacons, there has to be some part of the program which aggregates the information of multiple beacons (by searching for coherence, e.g.), or else determines which beacon takes precedence, and then also determines how to interpret their physical signature as a bitstring.
Tangent: in heading down a path trying to compare [scan for “important to influence”] vs. [scan for “consequentialist attempted output messages”] just now, my first attempt had an error, so I’ll point it out. It’s not necessarily harder to specify “scan for X” than “scan for Y” when X is a subset of Y. For instance “scan for primes” is probably simpler than “scan for numbers with less than 6 factors”.
Maybe clarifying or recasting the language around “easiest-to-specify controllable pattern” will clear this up, but can you explain more why it feels to you that [scan for “consequentialists’ attempted output messages”] is so much simpler than [scan for “important-to-influence data streams”]? My very preliminary first take is that they are within 8-15 bits.
I split them in part in case there is there is a contingent of consequentialists who believes that outputting the right bitstring is key to their continued existence, believing that they stop being simulated if they output the wrong bit. I haven’t responded to your claim that this would be faulty metapyhsics on their part; it still seems fairly tangential to our main discussion. But you can interpret my 5 bit point estimate for (5) as claiming that 31 times out of 32 that a civilization of consequentialists tries to influence their world’s output, it is in an attempt to survive. Tell me if you’re interested in a longer justification that responds to your original “line by line comments” comment.