By “annoyance” I assume you mean you still have the feeling but work around it?
In that case, it may be a problem in ways you’re not aware of. Other people, prospective mates especially, can pick up on that feeling in tricky subtle ways and react to it.
I only mentioned that to explain the origin of a false belief. It is not currently a problem for me, just an annoyance.
By “annoyance” I assume you mean you still have the feeling but work around it?
In that case, it may be a problem in ways you’re not aware of. Other people, prospective mates especially, can pick up on that feeling in tricky subtle ways and react to it.
Ok—sorry for unneccesary advice.