It seems totally different from what we’re doing, I may be misunderstanding the analogy.
Suppose I look out at the world and do some science, e.g. discovering the standard model. Then I use my understanding of science to design great prediction algorithms that run fast, but are quite complicated owing to all of the approximations and heuristics baked into them.
The speed prior gives this model a very low probability because it’s a complicated model. But “do science” gives this model a high probability, because it’s a simple model of physics, and then the approximations follow from a bunch of reasoning on top of that model of physics. We aren’t trading off “shortness” for speed—we are trading off “looks good according to reasoning” for speed. Yes they are both arbitrary orders, but one of them systematically contains better models earlier in the order, since the output of reasoning is better than a blind prioritization of shorter models.
Of course the speed prior also includes a hypothesis that does “science with the goal of making good predictions,” and indeed Wei Dai and I are saying that this is the part of the speed prior that will dominate the posterior. But now we are back to potentially-malign consequentialistism. The cognitive work being done internally to that hypothesis is totally different from the work being done by updating on the speed prior (except insofar as the speed prior literally contains a hypothesis that does that work).
In other words:
Suppose physics takes n bits to specify, and a reasonable approximation takes N >> n bits to specify. Then the speed prior, working in the intended way, takes N bits to arrive at the reasonable approximation. But the aliens take n bits to arrive at the standard model, and then once they’ve done that can immediately deduce the N bit approximation. So it sure seems like they’ll beat the speed prior. Are you objecting to this argument?
(In fact the speed prior only actually takes n + O(1) bits, because it can specify the “do science” strategy, but that doesn’t help here since we are just trying to say that the “do science” strategy dominates the speed prior.)
I’m not sure which of these arguments will be more convincing to you.
Yes they are both arbitrary orders, but one of them systematically contains better models earlier in the order, since the output of reasoning is better than a blind prioritization of shorter models.
This is what is what I was trying to contextualize above. This is an unfair comparison. You’re imagining that the “reasoning”-based order gets to see past observations, and the “shortness”-based order does not. A reasoning-based order is just a shortness-based order that has been updated into a posterior after seeing observations (under the view that good reasoning is Bayesian reasoning). Maybe the term “order” is confusing us, because we both know it’s a distribution, not an order, and we were just simplifying to a ranking. A shortness-based order should really just be called a prior, and a reasoning-based order (at least a Bayesian-reasoning-based order) should really just be called a posterior (once it has done some reasoning; before it has done the reasoning, it is just a prior too). So yes, the whole premise of Bayesian reasoning is that updating based on reasoning is a good thing to do.
Here’s another way to look at it.
The speed prior is doing the brute force search that scientists try to approximate efficiently. The search is for a fast approximation of the environment. The speed prior considers them all. The scientists use heuristics to find one.
In fact the speed prior only actually takes n + O(1) bits, because it can specify the “do science” strategy
Exactly. But this does help for reasons I describe here. The description length of the “do science” strategy (I contend) is less than the description length of the “do science” + “treacherous turn” strategy. (I initially typed that as “tern”, which will now be the image I have of a treacherous turn.)
a reasoning-based order (at least a Bayesian-reasoning-based order) should really just be called a posterior
Reasoning gives you a prior that is better than the speed prior, before you see any data. (*Much* better, limited only by the fact that the speed prior contains strategies which use reasoning.)
The reasoning in this case is not a Bayesian update. It’s evaluating possible approximations *by reasoning about how well they approximate the underlying physics, itself inferred by a Bayesian update*, not by directly seeing how well they predict on the data so far.
The description length of the “do science” strategy (I contend) is less than the description length of the “do science” + “treacherous turn” strategy.
I can reply in that thread.
I think the only good arguments for this are in the limit where you don’t care about simplicity at all and only care about running time, since then you can rule out all reasoning. The threshold where things start working depends on the underlying physics, for more computationally complex physics you need to pick larger and larger computation penalties to get the desired result.
Given a world model ν, which takes k computation steps per episode, let νlog be the best world-model that best approximates ν (in the sense of KL divergence) using only logk computation steps.νlog is at least as good as the “reasoning-based replacement” of ν.
The description length of νlog is within a (small) constant of the description length of ν. That way of describing it is not optimized for speed, but it presents a one-time cost, and anyone arriving at that world-model in this way is paying that cost.
One could consider instead νlogε, which is, among the world-models that ε-approximate ν in less than logk computation steps (if the set is non-empty), the first such world-model found by a searching procedure ψ. The description length of νlogε is within a (slightly larger) constant of the description length of ν, but the one-time computational cost is less than that of νlog.
νlog, νlogε, and a host of other approaches are prominently represented in the speed prior.
If this is what you call “the speed prior doing reasoning,” so be it, but the relevance for that terminology only comes in when you claim that “once you’ve encoded ‘doing reasoning’, you’ve basically already written the code for it to do the treachery that naturally comes along with that.” That sense of “reasoning” really only applies, I think, to the case where our code is simulating aliens or an AGI.
(ETA: I think this discussion depended on a detail of your version of the speed prior that I misunderstood.)
Given a world model ν, which takes k computation steps per episode, let νlog be the best world-model that best approximates ν (in the sense of KL divergence) using only logk computation steps. νlog is at least as good as the “reasoning-based replacement” of ν.
The description length of νlog is within a (small) constant of the description length of ν. That way of describing it is not optimized for speed, but it presents a one-time cost, and anyone arriving at that world-model in this way is paying that cost.
To be clear, that description gets ~0 mass under the speed prior, right? A direct specification of the fast model is going to have a much higher prior than a brute force search, at least for values of β large enough (or small enough, however you set it up) to rule out the alien civilization that is (probably) the shortest description without regard for computational limits.
One could consider instead νlogε, which is, among the world-models that ε-approximate ν in less than logk computation steps (if the set is non-empty), the first such world-model found by a searching procedure ψ. The description length of νlogε is within a (slightly larger) constant of the description length of ν, but the one-time computational cost is less than that of νlog.
Within this chunk of the speed prior, the question is: what are good ψ? Any reasonable specification of a consequentialist would work (plus a few more bits for it to understand its situation, though most of the work is done by handing it ν), or of a petri dish in which a consequentialist would eventually end up with influence. Do you have a concrete alternative in mind, which you think is not dominated by some consequentialist (i.e. a ψ for which every consequentialist is either slower or more complex)?
Do you have a concrete alternative in mind, which you think is not dominated by some consequentialist (i.e. a ψ for which every consequentialist is either slower or more complex)?
Well one approach is in the flavor of the induction algorithm I messaged you privately about (I know I didn’t give you a completely specified algorithm). But when I wrote that, I didn’t have a concrete algorithm in mind. Mostly, it just seems to me that the powerful algorithms which have been useful to humanity have short descriptions in themselves. It seems like there are many cases where there is a simple “ideal” approach which consequentialists “discover” or approximately discover. A powerful heuristic search would be one such algorithm, I think.
(ETA: I think this discussion depended on a detail of your version of the speed prior that I misunderstood.)
I don’t think anything here changes if K(x) were replaced with S(x) (if that was what you misunderstood).
It seems totally different from what we’re doing, I may be misunderstanding the analogy.
Suppose I look out at the world and do some science, e.g. discovering the standard model. Then I use my understanding of science to design great prediction algorithms that run fast, but are quite complicated owing to all of the approximations and heuristics baked into them.
The speed prior gives this model a very low probability because it’s a complicated model. But “do science” gives this model a high probability, because it’s a simple model of physics, and then the approximations follow from a bunch of reasoning on top of that model of physics. We aren’t trading off “shortness” for speed—we are trading off “looks good according to reasoning” for speed. Yes they are both arbitrary orders, but one of them systematically contains better models earlier in the order, since the output of reasoning is better than a blind prioritization of shorter models.
Of course the speed prior also includes a hypothesis that does “science with the goal of making good predictions,” and indeed Wei Dai and I are saying that this is the part of the speed prior that will dominate the posterior. But now we are back to potentially-malign consequentialistism. The cognitive work being done internally to that hypothesis is totally different from the work being done by updating on the speed prior (except insofar as the speed prior literally contains a hypothesis that does that work).
In other words:
Suppose physics takes n bits to specify, and a reasonable approximation takes N >> n bits to specify. Then the speed prior, working in the intended way, takes N bits to arrive at the reasonable approximation. But the aliens take n bits to arrive at the standard model, and then once they’ve done that can immediately deduce the N bit approximation. So it sure seems like they’ll beat the speed prior. Are you objecting to this argument?
(In fact the speed prior only actually takes n + O(1) bits, because it can specify the “do science” strategy, but that doesn’t help here since we are just trying to say that the “do science” strategy dominates the speed prior.)
I’m not sure which of these arguments will be more convincing to you.
This is what is what I was trying to contextualize above. This is an unfair comparison. You’re imagining that the “reasoning”-based order gets to see past observations, and the “shortness”-based order does not. A reasoning-based order is just a shortness-based order that has been updated into a posterior after seeing observations (under the view that good reasoning is Bayesian reasoning). Maybe the term “order” is confusing us, because we both know it’s a distribution, not an order, and we were just simplifying to a ranking. A shortness-based order should really just be called a prior, and a reasoning-based order (at least a Bayesian-reasoning-based order) should really just be called a posterior (once it has done some reasoning; before it has done the reasoning, it is just a prior too). So yes, the whole premise of Bayesian reasoning is that updating based on reasoning is a good thing to do.
Here’s another way to look at it.
The speed prior is doing the brute force search that scientists try to approximate efficiently. The search is for a fast approximation of the environment. The speed prior considers them all. The scientists use heuristics to find one.
Exactly. But this does help for reasons I describe here. The description length of the “do science” strategy (I contend) is less than the description length of the “do science” + “treacherous turn” strategy. (I initially typed that as “tern”, which will now be the image I have of a treacherous turn.)
Reasoning gives you a prior that is better than the speed prior, before you see any data. (*Much* better, limited only by the fact that the speed prior contains strategies which use reasoning.)
The reasoning in this case is not a Bayesian update. It’s evaluating possible approximations *by reasoning about how well they approximate the underlying physics, itself inferred by a Bayesian update*, not by directly seeing how well they predict on the data so far.
I can reply in that thread.
I think the only good arguments for this are in the limit where you don’t care about simplicity at all and only care about running time, since then you can rule out all reasoning. The threshold where things start working depends on the underlying physics, for more computationally complex physics you need to pick larger and larger computation penalties to get the desired result.
Given a world model ν, which takes k computation steps per episode, let νlog be the best world-model that best approximates ν (in the sense of KL divergence) using only logk computation steps.νlog is at least as good as the “reasoning-based replacement” of ν.
The description length of νlog is within a (small) constant of the description length of ν. That way of describing it is not optimized for speed, but it presents a one-time cost, and anyone arriving at that world-model in this way is paying that cost.
One could consider instead νlogε, which is, among the world-models that ε-approximate ν in less than logk computation steps (if the set is non-empty), the first such world-model found by a searching procedure ψ. The description length of νlogε is within a (slightly larger) constant of the description length of ν, but the one-time computational cost is less than that of νlog.
νlog, νlogε, and a host of other approaches are prominently represented in the speed prior.
If this is what you call “the speed prior doing reasoning,” so be it, but the relevance for that terminology only comes in when you claim that “once you’ve encoded ‘doing reasoning’, you’ve basically already written the code for it to do the treachery that naturally comes along with that.” That sense of “reasoning” really only applies, I think, to the case where our code is simulating aliens or an AGI.
(ETA: I think this discussion depended on a detail of your version of the speed prior that I misunderstood.)
To be clear, that description gets ~0 mass under the speed prior, right? A direct specification of the fast model is going to have a much higher prior than a brute force search, at least for values of β large enough (or small enough, however you set it up) to rule out the alien civilization that is (probably) the shortest description without regard for computational limits.
Within this chunk of the speed prior, the question is: what are good ψ? Any reasonable specification of a consequentialist would work (plus a few more bits for it to understand its situation, though most of the work is done by handing it ν), or of a petri dish in which a consequentialist would eventually end up with influence. Do you have a concrete alternative in mind, which you think is not dominated by some consequentialist (i.e. a ψ for which every consequentialist is either slower or more complex)?
Well one approach is in the flavor of the induction algorithm I messaged you privately about (I know I didn’t give you a completely specified algorithm). But when I wrote that, I didn’t have a concrete algorithm in mind. Mostly, it just seems to me that the powerful algorithms which have been useful to humanity have short descriptions in themselves. It seems like there are many cases where there is a simple “ideal” approach which consequentialists “discover” or approximately discover. A powerful heuristic search would be one such algorithm, I think.
I don’t think anything here changes if K(x) were replaced with S(x) (if that was what you misunderstood).