Dating but I have a crush on someone else

I have been dating a wonderful guy for 2 years. What we have is very special to me and I would never do anything to get in the way of that. There is one other guy that I only see a couple weeks a year that I have always had a crush on. I know I would never act on my feelings because I love my boyfriend dearly and would never ruin that but even having a crush on someone else makes me feel so guilty. I wouldn't want to distance myself from him (the guy I'm visiting) because we are still great friends who barely see each other... should I feel guilty?