Then if you love him, you're building a life together and he's done NOTHING to indicate to you that he's cheated or ever going to cheat and he's showing you in words and actions that match that he loves you and no one else but you, then what is your reasoning for being fixated on his past? There is NO good reason. PERIOD.
You are going to eventually whittle away at his respect for you if you don't cut out this silliness. If you can't stop obsessing then I suggest you go to your family doctor and tell him/her that you want a psych referral so that you can talk to someone about your obesession that is ruining a perfectly good relationship because of your OCD thinking on this.
“The willingness to accept responsibility for one’s own life is the source from which self-respect springs.” ~Joan Didion