Yes, this is usually the right approach—use a single, more complex, model that has the various models you were considering as special cases. It’s likely that the best parameters of this extended model won’t actually turn out to be one of the special cases. (But note that this approach doesn’t necessarily eliminate the need for careful consideration of the prior, since unwise priors for a single complex model can also cause problems.)
However, there are some situations where discrete models make sense. For instance, you might be analysing old Roman coins, and be unsure whether they were all minted in one mint, or in two (or three, …) different mints. There aren’t really any intermediate possibilities between one mint or two. Or you might be studying inheritance of two genes, and be considering two models in which they are either on the same chromosome or on different chromosones.
If you’re thinking of a stick-breaking prior such as a Dirichlet process mixture model, they typically produce an infinite number of components (which would be mints, in this case), though of course only a finite number will be represented in your finite data set. But we know that the number of mints producing coins in the Roman Empire was finite. So that’s not a reasonable prior (though of course you might sometimes be able to get away with using it anyway).
Yes, this is usually the right approach—use a single, more complex, model that has the various models you were considering as special cases. It’s likely that the best parameters of this extended model won’t actually turn out to be one of the special cases. (But note that this approach doesn’t necessarily eliminate the need for careful consideration of the prior, since unwise priors for a single complex model can also cause problems.)
However, there are some situations where discrete models make sense. For instance, you might be analysing old Roman coins, and be unsure whether they were all minted in one mint, or in two (or three, …) different mints. There aren’t really any intermediate possibilities between one mint or two. Or you might be studying inheritance of two genes, and be considering two models in which they are either on the same chromosome or on different chromosones.
Good points, but can’t you still solve the discrete problem with a single model and a stick breaking prior on the number of mints, right?
If you’re thinking of a stick-breaking prior such as a Dirichlet process mixture model, they typically produce an infinite number of components (which would be mints, in this case), though of course only a finite number will be represented in your finite data set. But we know that the number of mints producing coins in the Roman Empire was finite. So that’s not a reasonable prior (though of course you might sometimes be able to get away with using it anyway).