I am the same! except my boyfriend my first reationship so I was hoping that was the reason but I can understand how you are feeling. I used to drive myself crazy thinking of his past and now I am better with this (took year and a half) but I admit now I am worried that he may meet someone else, cheat etc but guess what so could I and that makes me feel better.
You shouldn't judge a man past, you have a past and at your age not many girls have a child etc so how can you judge? you just can't.
You have your child, you should be happy you have her. Men can come and go but he is not your be all and end all of life. We can all if we wanted too leave our partners and find a new love, its just the way life is. You need to stop being so scared, if he cheats its his lost and your move on. You should start thinking like this.
You are placing everything on him and thats why you are so insecure.