I have had issues with this and you have to think of it in a different light. He is choosing to be with you right? If he loves you, then he won't jeopardize that. He is with you for a reason. If he wanted his old f*ck buddy back, then you would be out the door.
If anything ever DID happen, you;ll probably find out. But who is to say you will? Sometimes you just never know. If he spends his time with you and chooses to keep you in his life as a priority then try to feel confident about that. Focus on that and the other "friends" are less threatening.
You can let him know it makes you feel uncomfortable, but you don't want to drive people away with mis trust. Maybe let him know if he is ever to hang out with this person, that you would feel more comfortable being present. Good luck and try to focus on the fact that he has chosen you.