I think you should end it. You had your fun when he was in town, but it seems like that's going to be the extent of it. If I met a woman while I was traveling for work, and I knew I wanted to be with her; 5 states wouldn't hold me back from expressing my feelings and letting her know how much she meant to me. If I don't contact a female on a regular basis, it means I'm not really into her. It sounds like he likes the attention from you. Also, by him giving you the silent treatment and blaming it on him being a man, that sounds like BS. It seems like he's not very mature. In order for relationships to work, communication is key. Do you really want to be with a man that won't open up to you when you've angered him a bit? I think you should just move on, and be happy with whatever time you spent together and be done with it.
Always tell people exactly how you feel, at that exact moment. If the moment has passed, let us help you:
www.HopeYouGetThis.Com
L’esprit de l’escalier