Are you the thought police? Do you have to know EVERYTHING? (If yes, YOU have issues not him). He's allowed to have a private life is that private life remains faithful in all aspects at that point it's none of your business (if in fact it's really harmless).
I talk to my boss more than 1 hour per week and he ain't my friend.
I'm going to say exactly what you need to hear but don't want to:
He's done nothing at all wrong and the entire problem, issue and all that surrounds it is all YOUR issues. You are insecure, you are the thought police and you do need to know everything about all things. You need to work on believing what he says, and becoming a secure person in a secure relationship.