Really cool. I read some of these kinds of papers last week, but this is better context on the topic. Redundancy seems like evidence in favor of a narrow loss basin, but e.g. the fact that fine-tuned BERT models generalize very differently is evidence of multiple local minima. Your guess that linear mode connectivity works in simple image classification domains but not in language models seems like the most likely answer to me, but I would be interested to see it tested.
Really cool. I read some of these kinds of papers last week, but this is better context on the topic. Redundancy seems like evidence in favor of a narrow loss basin, but e.g. the fact that fine-tuned BERT models generalize very differently is evidence of multiple local minima. Your guess that linear mode connectivity works in simple image classification domains but not in language models seems like the most likely answer to me, but I would be interested to see it tested.