Yeah, makes me think about trying to have ‘rotation and translation invariant’ representations of objects in ML vision research.
Seems like if you can subtract out general, longer span terms (but note their presence for the decoder to add them back in), that would be much more intuitive. Language, as you mentioned, is an obvious one. Some others which occur to me are: Whether the text is being spoken by a character in a dialogue (vs the ‘AI assistant’ character).
Whether the text is near the beginning/middle/end of a passage.
Patterns of speech being used in this particular passage of text (e.g. weird punctuation / capitalization patterns).
Yeah, makes me think about trying to have ‘rotation and translation invariant’ representations of objects in ML vision research.
Seems like if you can subtract out general, longer span terms (but note their presence for the decoder to add them back in), that would be much more intuitive. Language, as you mentioned, is an obvious one. Some others which occur to me are:
Whether the text is being spoken by a character in a dialogue (vs the ‘AI assistant’ character).
Whether the text is near the beginning/middle/end of a passage.
Patterns of speech being used in this particular passage of text (e.g. weird punctuation / capitalization patterns).