If your map significantly doesn’t match the territory, natural selection is likely to be brutal to you.
Another way to think about his idea:
Natural selection is equally brutal to all life. Moss and ants have horrible maps, but they are still successful in terms of natural selection.
All natural selection “cares about” is genes copied. Claws, peacock tail feathers, and “maps” can all “make a difference,” but natural selection only acts on the results (genes copied); natural selection itself doesn’t favor any particular kind of adaptation, that’s why I think the original quote is not an overstatement.