I guess in that case I’d worry that you go and look at the features and come away with some impression of what those features represent and it turns out you’re totally wrong? I keep coming back to the example of a text-classifier where you find “”“the French activation directions””” except it turns out that only one of them is for French (if any at all) and the others are things like “words ending in x and z” or “words spoken by fancy people in these novels and quotes pages”.
I guess in that case I’d worry that you go and look at the features and come away with some impression of what those features represent and it turns out you’re totally wrong? I keep coming back to the example of a text-classifier where you find “”“the French activation directions””” except it turns out that only one of them is for French (if any at all) and the others are things like “words ending in x and z” or “words spoken by fancy people in these novels and quotes pages”.