No, what Thomas is saying is that we should compare the model’s predictions with the actual results and use that to calibrate how much we should trust the model.
I expressed myself poorly, “should we trust our model or the actual results?” was a restating of “Should we trust models or observations?” to make it more clear what the original quote actually meant (did it?); that you will never have future observations only past observation, so when dealing with future events one can only depend on models. Of course when the future unfolds we will be able to do the observations, but then future observations has become past observation. One can only stear the course of the future, never the past. Thus trust in predictions.
I think people are somewhat talking past each other, and the following basically summarizes everyone’s position:
1) When dealing with the future, we have to make use of the best models available—we can’t base decisions now on data we don’t have yet.
2) New data should be used both to evaluate and improve models.
2a) It is important to test models against data that were not used in formulating the model, to avoid over-fitting. This can be new data as it becomes available, but should also be existing data reserved for the purpose.
No, what Thomas is saying is that we should compare the model’s predictions with the actual results and use that to calibrate how much we should trust the model.
I expressed myself poorly, “should we trust our model or the actual results?” was a restating of “Should we trust models or observations?” to make it more clear what the original quote actually meant (did it?); that you will never have future observations only past observation, so when dealing with future events one can only depend on models. Of course when the future unfolds we will be able to do the observations, but then future observations has become past observation. One can only stear the course of the future, never the past. Thus trust in predictions.
I think people are somewhat talking past each other, and the following basically summarizes everyone’s position:
1) When dealing with the future, we have to make use of the best models available—we can’t base decisions now on data we don’t have yet.
2) New data should be used both to evaluate and improve models.
2a) It is important to test models against data that were not used in formulating the model, to avoid over-fitting. This can be new data as it becomes available, but should also be existing data reserved for the purpose.