Why wouldn’t simply ‘reversing the text during pretraining’ fix this for a causal decoder LLM?
I had basically the same idea here! I also expect that would work.
More generally, I think this kind of research (and also a lot of interpretability work) is interesting as a characterization and categorization of the workings and deficiencies of current systems and training processes, but not likely to be particularly useful for predicting trends or modelling systems in even the very near future (or the present, arguably… if you want an LLM to tell you about Mary Lee Pfieffer or Ed Witten, just use Bing).
I had basically the same idea here! I also expect that would work.
More generally, I think this kind of research (and also a lot of interpretability work) is interesting as a characterization and categorization of the workings and deficiencies of current systems and training processes, but not likely to be particularly useful for predicting trends or modelling systems in even the very near future (or the present, arguably… if you want an LLM to tell you about Mary Lee Pfieffer or Ed Witten, just use Bing).