In the Bayesian setting where probabilities are subjective beliefs there shouldn’t be too many problems with the “probability of a model” expression.
In Jaynes’ bayesian setting, a probability is a number you assign to a proposition. Models as generally used are not propositions.
that the current model is the appropriate one for the task at hand.
Don’t like that one. For any model, you can generally conceive of an infinite number of slightly tweaked, slightly better versions, so that for any particular model P(model is the appropriate one) is 0.
What if you define “probability of a model” as 1 - (probability that replacing it with a different model will improve things)?
The probability that some “random sample” from some set of models will have improved performance? What aggregated error function to quantify “better”? How was the domain of the model sampled for the error function?
I see an ocean of structural commitments being imposed on the problem, commitments about how you choose to think about the problem, to define a “probability of a model”.
And after all that, I still don’t see a proposition that you’re assigning a probability to, I see a model. I could just as well define the probability of my shoe. I could have all sorts of structural commitments about the meaning of “the probability of my shoe”. But in the end, that doesn’t make my shoe a proposition, nor the probability of a shoe that I’ve just defined the same category of thing as the probability of a proposition.
The Map is not the Territory. There is no “true” map. There is no “true” model. The relevant thing for a model is how well it gets you to where you want to go.
It’s true that models are maps. It’s also true, to recall a George Box quote, that “all models are false but some are useful”.
I agree that
The relevant thing for a model is how well it gets you to where you want to go
...and that, to my mind, supports the notion of the “probability of a model”, or, rather, the “probability of this particular model being sufficiently good to get you to where you want to go”.
I think it’s a fairly practical concept—if I’m modeling something and I am fitting several models which give me various trade-offs, it’s useful for me to think in terms of, say, the probability that a linear model will be sufficient for my purposes. If I define my purposes rigorously enough, the “model is sufficient” becomes a proposition.
But in a more general and more handwavy sense, I think it’s fine to assign to whole maps the probability of being correct. Take a literal example, say a nautical chart. Let’s say I have a chart of a coast unknown to me and as I explore it, I find that the chart is partially correct, but partially off. It depicts this peninsula, but fails to show that rock and the sandbar on the chart doesn’t exist in reality. After a while my belief in the accuracy of chart becomes partial so when I go around a point and the chart says there will be shoals, I expect to actually find these shoals with the probability of X%.
In Jaynes’ bayesian setting, a probability is a number you assign to a proposition. Models as generally used are not propositions.
Don’t like that one. For any model, you can generally conceive of an infinite number of slightly tweaked, slightly better versions, so that for any particular model P(model is the appropriate one) is 0.
The probability that some “random sample” from some set of models will have improved performance?
What aggregated error function to quantify “better”? How was the domain of the model sampled for the error function?
I see an ocean of structural commitments being imposed on the problem, commitments about how you choose to think about the problem, to define a “probability of a model”.
And after all that, I still don’t see a proposition that you’re assigning a probability to, I see a model. I could just as well define the probability of my shoe. I could have all sorts of structural commitments about the meaning of “the probability of my shoe”. But in the end, that doesn’t make my shoe a proposition, nor the probability of a shoe that I’ve just defined the same category of thing as the probability of a proposition.
The Map is not the Territory. There is no “true” map. There is no “true” model. The relevant thing for a model is how well it gets you to where you want to go.
It’s true that models are maps. It’s also true, to recall a George Box quote, that “all models are false but some are useful”.
I agree that
...and that, to my mind, supports the notion of the “probability of a model”, or, rather, the “probability of this particular model being sufficiently good to get you to where you want to go”.
I think it’s a fairly practical concept—if I’m modeling something and I am fitting several models which give me various trade-offs, it’s useful for me to think in terms of, say, the probability that a linear model will be sufficient for my purposes. If I define my purposes rigorously enough, the “model is sufficient” becomes a proposition.
But in a more general and more handwavy sense, I think it’s fine to assign to whole maps the probability of being correct. Take a literal example, say a nautical chart. Let’s say I have a chart of a coast unknown to me and as I explore it, I find that the chart is partially correct, but partially off. It depicts this peninsula, but fails to show that rock and the sandbar on the chart doesn’t exist in reality. After a while my belief in the accuracy of chart becomes partial so when I go around a point and the chart says there will be shoals, I expect to actually find these shoals with the probability of X%.