Thanks! I get your arguments about “knowledge” being restricted to predictive domains, but I think it’s (mostly) just a semantic issue. I also don’t think the specifics of the word “knowledge” are particularly important to my points which is what I attempted to clarify at the start, but I’ve clearly typical-minded and assumed that of course everyone would agree with me about a dog/fish classifier having “knowledge”, when it’s more of an edge-case than I thought! Perhaps a better version of this post would have either tabooed “knowledge” altogether or picked a more obviously-knowledge-having model.
Well, it certainly has mutual information with the training data, even if it only acts as a classifier (actually, classifiers can be seen as inverse generative models, so there is some generative-ish information there, as well). From that perspective, your arguments certainly hold. Although, I’m not sure if “mutual information” is precisely what you’re going for, either. Yes, I agree, I should have tabooed “knowledge” in how I read it.
Thanks! I get your arguments about “knowledge” being restricted to predictive domains, but I think it’s (mostly) just a semantic issue. I also don’t think the specifics of the word “knowledge” are particularly important to my points which is what I attempted to clarify at the start, but I’ve clearly typical-minded and assumed that of course everyone would agree with me about a dog/fish classifier having “knowledge”, when it’s more of an edge-case than I thought! Perhaps a better version of this post would have either tabooed “knowledge” altogether or picked a more obviously-knowledge-having model.
Well, it certainly has mutual information with the training data, even if it only acts as a classifier (actually, classifiers can be seen as inverse generative models, so there is some generative-ish information there, as well). From that perspective, your arguments certainly hold. Although, I’m not sure if “mutual information” is precisely what you’re going for, either. Yes, I agree, I should have tabooed “knowledge” in how I read it.