Any non-uniform prior inherently encodes a bias toward simplicity. This isn’t an additional assumption we need to make—it falls directly out of the mathematics.
For any hypothesis h, the information content is I(h)=−log(P(h)), which means probability and complexity have an exponential relationship: P(h)=e−I(h)
This demonstrates that simpler hypotheses (those with lower information content) are automatically assigned higher probabilities. The exponential relationship creates a strong bias toward simplicity without requiring any special mechanisms.
The “simplicity prior” is essentially tautological—more probable things are simple by definition.
Simplicity Priors are Tautological
Any non-uniform prior inherently encodes a bias toward simplicity. This isn’t an additional assumption we need to make—it falls directly out of the mathematics.
For any hypothesis h, the information content is I(h)=−log(P(h)), which means probability and complexity have an exponential relationship: P(h)=e−I(h)
This demonstrates that simpler hypotheses (those with lower information content) are automatically assigned higher probabilities. The exponential relationship creates a strong bias toward simplicity without requiring any special mechanisms.
The “simplicity prior” is essentially tautological—more probable things are simple by definition.