Your boyfriend already told you that nothing happened, but you still are not sure you believe him. That signals major trust issues between the two of you and THAT is what you should be trying to figure out together. Do you have a reason to doubt him? Even if something happened long ago, why does it matter to you at present? These are the questions you should be asking yourself before you confront him about something that he probably doesn't even consider an issue.
He might have said, "What does it matter?" because he is only concerned with the present and future with you.
Good luck.
Brought to you by Dating With Devon!