In general, rules of thumb have two dimensions—applicability (that is the size of the domain where it applies) and efficacy (the amount or degree of guidance that the rule provides).
Simplicity, a.k.a Occam’s Razor, is mentioned frequently as a guide in these (philosophy of science/atheist/AI aficionado) circles. However, it is notable more for its broad applicability than for its efficacy compared to other, less-broadly-applicable guidelines.
Try formulating a rule for listing natural numbers (positive integers) without repeats that does not generally trend upwards. For example, you could alternate between powers of ten and powers of two: 1, 10, 2, 100, 4, 1000, … Regardless of the rule, you cannot list the natural numbers from largest to smallest; there is no largest. Whichever you pick as your first, you will eventually be forced past by the “no repeats” clause.
A general learner can be viewed as outputting a list of numbers (coding for hypotheses). Occam’s razor is roughly the observation “the numbers will generally trend upwards”. There’s still a lot up in the air after that observation.
In general, rules of thumb have two dimensions—applicability (that is the size of the domain where it applies) and efficacy (the amount or degree of guidance that the rule provides).
Simplicity, a.k.a Occam’s Razor, is mentioned frequently as a guide in these (philosophy of science/atheist/AI aficionado) circles. However, it is notable more for its broad applicability than for its efficacy compared to other, less-broadly-applicable guidelines.
Try formulating a rule for listing natural numbers (positive integers) without repeats that does not generally trend upwards. For example, you could alternate between powers of ten and powers of two: 1, 10, 2, 100, 4, 1000, … Regardless of the rule, you cannot list the natural numbers from largest to smallest; there is no largest. Whichever you pick as your first, you will eventually be forced past by the “no repeats” clause.
A general learner can be viewed as outputting a list of numbers (coding for hypotheses). Occam’s razor is roughly the observation “the numbers will generally trend upwards”. There’s still a lot up in the air after that observation.
What’s with the Occam bashing? Yes, the OP wrote:
“Nature doesn’t care all that much for mathematical simplicity.”
...but that doesn’t make it true: Occam’s Razor is great!