Perhaps people have the intuition that it is impossible to understand the world through language alone because people themselves find it impossible to understand the world through language alone. For example, assembly instructions for even a simple object could be quite incomprehensible if written as a text, but easily understandable from a simple diagram. In your parlance, language does not model the world well, so a good model of a language does not translate to a good model of the world. This is why we have technical drawings and not technical descriptions.
Perhaps people have the intuition that it is impossible to understand the world through language alone because people themselves find it impossible to understand the world through language alone. For example, assembly instructions for even a simple object could be quite incomprehensible if written as a text, but easily understandable from a simple diagram. In your parlance, language does not model the world well, so a good model of a language does not translate to a good model of the world. This is why we have technical drawings and not technical descriptions.