Yeah, in future you will need to be more sensitive and reassure him that you are not interested in getting off with other men, that other men don't interest you'. That is what I do anyway and it usually works.
Or I guess you could just stop telling him where you are going, make out like you are sitting at home every night, lol