Personally, I would tell him I wasn't the kiss-and-tell type, and tell him to quit asking questions he didn't REALLY want to know the answer to. If he persisted on pressuring me to answer, I would bluntly tell him I considered his badgering a violation of my privacy. If he still pushed it, I would consider it a problem. Who wants to be with someone who can't respect your boundaries?
Of course, another approach would be to simply lie to him, and I think guys who push too much in this area of questioning are asking to be lied to. If a girl has only had one or two lovers, she will generally make this known.