This will most probably not answer your question but I hope you will find this interesting.
The Kolmogorov complexity prior is, as you have stated, merely a formalization of Occam’s razor. Another possibility to formalize Occam’s razor is Schmidhuber’s prior where the probability of an algorithm is its speed, roughly speaking. It has the advantage of being computable in the limit, as opposed to the other option.
The interesting thing is now that we can formalize various inductive hypotheses as priors such as “Everything goes” as a uniform distribution. There was a discussion on this a few weeks before. The point is that, to my knowledge, beyond practicability there is no theoretical justification of Occam’s razor. For an atheist though, Occam’s razor has the nice property of cutting off the probability of any algorithm that incorporates the concept of a god.
The interesting thing is now that we can formalize various inductive hypotheses as priors such as “Everything goes” as a uniform distribution.
A uniform distribution on what? If you start with a uniform distribution on binary sequences, you don’t get to perform inductive reasoning at all, as the observables X(1), X(2), etc. are all independent in that distribution. If you wanted to start with a uniform distribution on computable universes, you can’t, because there is no uniform distribution with countable support.
A uniform distribution on all algorithms, that is a uniform distribution on all binary strings. Intuitively we can compute probability ratios for any two algorithms given evidence since the identical prior probability cancels in that ratio.
But, as you say, the problem is that there is actually no uniform distribution with countable support. At best, we can circumvent the problem by computing the probability ratios which is almost as good.
God is the least of all problems with a uniform (improper) prior. The winning hypothesis will always, provably, be a “just so” hypothesis, where everything that happened was absolutely necessary with probability 1. However, if the truth is simpler, then these sorts of hypotheses “overfit the curve” and give predictions different from what humans make. Every time a scientist makes a correct prediction, the “just so” hypothesis would imply that they were just lucky.
The point is that, to my knowledge, beyond practicability there is no theoretical justification of Occam’s razor. For an atheist though, Occam’s razor has the nice property of cutting off the probability of any algorithm that incorporates the concept of a god.
Er… isn’t this like saying “To my knowledge, there is no theoretical justification of biblical inerrancy. For a Christian though, biblical inerrancy has the nice property of cutting off the probability of any algorithm that incorporates atheism”?
Occam’s razor better have an independent reason to be sensible, other than reaffirming a previous (held on which basis?) belief in atheism.
Not exactly. There is no way to discount biblical evidence a priori. We can, however massively discount biblical evidence by deriving empirical predicitons and find that they do not match reality. For example, earth is massively older than the computed 6000 years.
The point of a prior distribution is to incorporate any principle or knowledge we have at hand. If we have no prior knowledge, that is, no evidence, no observation, a literal tabula rasa, the only guide we have in designing a prior are principles. That is the point which I tried to make: Before we look at the evidence we can think about which hypotheses we would prefer. Problem is, that we can now make another prior distribution for the principles in designing a prior. And another one ad infinitum. That is the fundamental problem of Bayesian reasoning that there is no canonical way to choose a prior, which is not as bad as it sounds since for infinitely much evidence the posterior will approach the true distribution.
Occam’s razor is an axiom. We can justify it by practicability considerations such as “simpler hypotheses are easier to compute” but ultimately there is no “fundamental” reason for it. My last remark about atheism was just a sidenote to illustrate that one motivation for Occam’s razor could be belief in atheism, but as you have already noted, atheism itself has to be founded on some basis: That is the position of agnosticism, that wich has no evidence speaking for or against it we can not decide. Occam’s razor merely disregards hypotheses that assert the existence of entities with no empirical effects.
This will most probably not answer your question but I hope you will find this interesting.
The Kolmogorov complexity prior is, as you have stated, merely a formalization of Occam’s razor. Another possibility to formalize Occam’s razor is Schmidhuber’s prior where the probability of an algorithm is its speed, roughly speaking. It has the advantage of being computable in the limit, as opposed to the other option.
The interesting thing is now that we can formalize various inductive hypotheses as priors such as “Everything goes” as a uniform distribution. There was a discussion on this a few weeks before. The point is that, to my knowledge, beyond practicability there is no theoretical justification of Occam’s razor. For an atheist though, Occam’s razor has the nice property of cutting off the probability of any algorithm that incorporates the concept of a god.
A uniform distribution on what? If you start with a uniform distribution on binary sequences, you don’t get to perform inductive reasoning at all, as the observables X(1), X(2), etc. are all independent in that distribution. If you wanted to start with a uniform distribution on computable universes, you can’t, because there is no uniform distribution with countable support.
A uniform distribution on all algorithms, that is a uniform distribution on all binary strings. Intuitively we can compute probability ratios for any two algorithms given evidence since the identical prior probability cancels in that ratio.
But, as you say, the problem is that there is actually no uniform distribution with countable support. At best, we can circumvent the problem by computing the probability ratios which is almost as good.
Did you find the rest of my post useful?
God is the least of all problems with a uniform (improper) prior. The winning hypothesis will always, provably, be a “just so” hypothesis, where everything that happened was absolutely necessary with probability 1. However, if the truth is simpler, then these sorts of hypotheses “overfit the curve” and give predictions different from what humans make. Every time a scientist makes a correct prediction, the “just so” hypothesis would imply that they were just lucky.
That depends on the language you use for your Kolmogorov prior.
Er… isn’t this like saying “To my knowledge, there is no theoretical justification of biblical inerrancy. For a Christian though, biblical inerrancy has the nice property of cutting off the probability of any algorithm that incorporates atheism”?
Occam’s razor better have an independent reason to be sensible, other than reaffirming a previous (held on which basis?) belief in atheism.
Not exactly. There is no way to discount biblical evidence a priori. We can, however massively discount biblical evidence by deriving empirical predicitons and find that they do not match reality. For example, earth is massively older than the computed 6000 years.
The point of a prior distribution is to incorporate any principle or knowledge we have at hand. If we have no prior knowledge, that is, no evidence, no observation, a literal tabula rasa, the only guide we have in designing a prior are principles. That is the point which I tried to make: Before we look at the evidence we can think about which hypotheses we would prefer. Problem is, that we can now make another prior distribution for the principles in designing a prior. And another one ad infinitum. That is the fundamental problem of Bayesian reasoning that there is no canonical way to choose a prior, which is not as bad as it sounds since for infinitely much evidence the posterior will approach the true distribution.
Occam’s razor is an axiom. We can justify it by practicability considerations such as “simpler hypotheses are easier to compute” but ultimately there is no “fundamental” reason for it. My last remark about atheism was just a sidenote to illustrate that one motivation for Occam’s razor could be belief in atheism, but as you have already noted, atheism itself has to be founded on some basis: That is the position of agnosticism, that wich has no evidence speaking for or against it we can not decide. Occam’s razor merely disregards hypotheses that assert the existence of entities with no empirical effects.