Have you considered other response modalities? Would it worthwhile having numeric representations of trait strength for example over plain text classification?
I have an observation that LLMs are highly effective at approximations but not effective at precision, do you think the variety in responses has an effect on accuracy? Eg sexuality, was represented commonly as a boolean value, forces a level of precision that the model isn’t efficient with?
Meaning eg seeing what it can infer if it’s responding by voice? Or what do you mean by response modalities here?
Would it worthwhile having numeric representations of trait strength for example over plain text classification?
Absolutely! I’m looking at the token probabilities of the top five most-likely tokens, and treating that as a probability distribution over possible answers; that definitely provides usefully greater info than just looking at the top token.
I have an observation that LLMs are highly effective at approximations but not effective at precision, do you think the variety in responses has an effect on accuracy? Eg sexuality, was represented commonly as a boolean value, forces a level of precision that the model isn’t efficient with?
Can you talk a bit more about what you’re imagining as an alternative approach here? For sexuality I’m offering it ‘straight’, ‘gay’, and ‘bi’ as valid answers, and those are ~always the top three most-likely tokens (in different orders for different profiles, of course); the other tokens that show up most often in the top five are the same text but capitalized or with/without a leading space.
Have you considered other response modalities? Would it worthwhile having numeric representations of trait strength for example over plain text classification?
I have an observation that LLMs are highly effective at approximations but not effective at precision, do you think the variety in responses has an effect on accuracy? Eg sexuality, was represented commonly as a boolean value, forces a level of precision that the model isn’t efficient with?
Meaning eg seeing what it can infer if it’s responding by voice? Or what do you mean by response modalities here?
Absolutely! I’m looking at the token probabilities of the top five most-likely tokens, and treating that as a probability distribution over possible answers; that definitely provides usefully greater info than just looking at the top token.
Can you talk a bit more about what you’re imagining as an alternative approach here? For sexuality I’m offering it ‘straight’, ‘gay’, and ‘bi’ as valid answers, and those are ~always the top three most-likely tokens (in different orders for different profiles, of course); the other tokens that show up most often in the top five are the same text but capitalized or with/without a leading space.