What encourages the helper model to generate correct explanations as opposed to false/spurious ones?
I.e., let’s say the text is a list of fruit, and the correct next word is Klingon for “pineapple”. I’m imagining that the helper model could just say “The next word is [Klingon for pineapple]” or give an alternate/spurious explanation of the Klingon text (“The text is discussing a spiky fruit that goes on pizza”). Both of the above, unhelpful/spurious explanations would make me predict the next Klingon word correctly.
What encourages the helper model to generate correct explanations as opposed to false/spurious ones?
I.e., let’s say the text is a list of fruit, and the correct next word is Klingon for “pineapple”. I’m imagining that the helper model could just say “The next word is [Klingon for pineapple]” or give an alternate/spurious explanation of the Klingon text (“The text is discussing a spiky fruit that goes on pizza”). Both of the above, unhelpful/spurious explanations would make me predict the next Klingon word correctly.