“But this is really not stable and the ordering of the explanation didn’t seem to help the score that much. I will work on a post investigating this in more detail.”
This was making the metric very unstable and this doesn’t seems to help the score. I am working on a metric to evaluate that but i think this doesn’t help. I have the intuition it might be some kind of search or complex heuristic inside LLMs (kind of LLM computing the explanation in advance and weighting the score with this precomputed explanation). I might be totally wrong but i am trying to think of a way to test that.
kind of LLM computing the explanation in advance and weighting the score with this precomputed score
I think this would work fine for things that don’t essentially require step-by-step reasoning (“detailed reasoning” might still be accessing this capability), which is why the order from the post is not as bad as writing down the bottom line before the argument. When you ask for the argument, it’s being anticipated, as much as that’s possible to do without seeing its prefix explicitly. But this should be a problem when the outcome of the argument can’t be anticipated.
“But this is really not stable and the ordering of the explanation didn’t seem to help the score that much. I will work on a post investigating this in more detail.”
This was making the metric very unstable and this doesn’t seems to help the score. I am working on a metric to evaluate that but i think this doesn’t help. I have the intuition it might be some kind of search or complex heuristic inside LLMs (kind of LLM computing the explanation in advance and weighting the score with this precomputed explanation). I might be totally wrong but i am trying to think of a way to test that.
I think this would work fine for things that don’t essentially require step-by-step reasoning (“detailed reasoning” might still be accessing this capability), which is why the order from the post is not as bad as writing down the bottom line before the argument. When you ask for the argument, it’s being anticipated, as much as that’s possible to do without seeing its prefix explicitly. But this should be a problem when the outcome of the argument can’t be anticipated.
Good point, i should add this in limitation and futur directions. Do you have an example in mind?