You are not over reacting. They are too close and he should not be this close to ANY other woman. I agree with valixy-stop inviting her to things, keep your distance as much as possible and make excuses if she asks to come round..
Do they text or email? Does he spend time alone with her now? Do they confide in each other about personal matters?
The fact this started right after you had a baby is a big red flag. Is there any issues in your marriage? Are you paying him enough attention? Fitting in intimate time? Make a big effort to make him feel special over the next few weeks and see if that helps. Plan a romantic night and you could buy a game (such as the board game "monogamy" for couples. I have seen v good reviews and heard it really helps bring you closer. I tried to otder it myself but they were out of stock).
This is in no way your fault but your husband IS getting to close to another woman and this IS how affairs start so try to push her out and pull him closer. Keep us updated on your progress and if there is any red flags-such as him texting her, emailing her, wanting to meet her alone or talking about her a lot then tell him you want marriage counselling now.
You have told him how you feel about this and he is YOUR husband and if he doesnt respect your feelings on this and create some distance soon-i would be worried this has already gone too far.
"Don't ask a question if you can't handle the answer".