Imagine the ten experiments produced the following numbers as results: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
The first scientists hypothesis is this function: if n < 20 then n else 5, (where n is the number of the variable being tested in the experiment)
10 more experiments are done and of course it predicts the answers perfectly. Scientist two comes up with his hypothesis: n. That’s it, just the value of n is the value that will be measured by the experiment.
Now, would you really trust the first hypothesis because it happened to have been made before the next ten experimental results were known?
In practice it’s usually better to choose the hypothesis that has made more successful predictions in the past, because that is evidence that it isn’t overfit. But complexity is also a more general way to keep overfitting in check. Useful when you can’t just perform more experiments to test your hypothesises, or where it would be expensive to do so.
Imagine the ten experiments produced the following numbers as results: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
The first scientists hypothesis is this function: if n < 20 then n else 5, (where n is the number of the variable being tested in the experiment)
10 more experiments are done and of course it predicts the answers perfectly. Scientist two comes up with his hypothesis: n. That’s it, just the value of n is the value that will be measured by the experiment.
Now, would you really trust the first hypothesis because it happened to have been made before the next ten experimental results were known?
In practice it’s usually better to choose the hypothesis that has made more successful predictions in the past, because that is evidence that it isn’t overfit. But complexity is also a more general way to keep overfitting in check. Useful when you can’t just perform more experiments to test your hypothesises, or where it would be expensive to do so.