You shouldn't let it bother you. Those are the trust issues rearing it's ugly head, not much more. If you don't deal with these trust issues, it will likely break up any good relationship you find. So, it's better to deal with these things early in your life. I recommend a counselor. Find one that will not only listen to you, but will make a plan for you to illustrate that, when you trust someone, it most often ends up fine. Your brain might be locked into only remembering when bad things happened when you trust someone. You need help seeing the good and the bad things that happened whe you trust someone, and realizing that more good things happened than bad things.
I should know, I was there. I had trouble trusting someone so I could feel close to them. Some counseling and several patient girlfriends helped me get through this.
I have a long time interest in psychology, specializing in relationship dynamics for 30 years.
(Please note, we give the best advice we can based on the information given in a post. For better advice, please include the age of all romantic partners.)