Find other sources of self-esteem than your husband. That's like the blind leading the blind. Do you have a job you enjoy?
Really, this kind of conversation is beyond useless in a marriage. Why are you even allowing it to happen? Why would he be thinking about other women when he has you? Remind him of this. Maybe you need to laugh at him and tell him the reason they don't hit on him is b/c they see he's married to an awesome woman and they don't stand a chance. If you don't believe this, then you need to spend some time working on your self-esteem. Yours, not his.
Second thoughts can generally be amended with judicious action; injudicious actions can seldom be recovered with second thoughts.
--Cyteen by C.J.Cherryh