The thing that is different with you and him and why it's crossing primary relationship boundaries is becuse you're taking your conversations personal, you're going beyond friendly chatter and you've done this enough that you're at the point where you're anticipating seeing him. You're allowing yourself to become vulnerable to him and when you do that, you have just started to hammer the first wedge between the emotional connection you have with your S.O. This is how emotional affairs begin and when those things begin it changes the dynamic that you currently enjoy at home.
It doesn't matter how old a person is, we still experience the same reactions and emotions that attraction to a member of the opposit sex evokes. Do youself and your husband/bf (whatever he is) a favour and quit allowing this man to penetrate your emotional vulnerabilty. A friendly smile hello and lovely weather we're having is all either of you should be indulging in. If your conversations go beyond that you need to put up relationship boundary blocks by including references to your husband e.g. "Oh my husband does that" or Gee that's what my husband would do kinda thing. Keep it forefront in both of your minds that you're taken.
Even while in a relationship we will all, always find others attractive. It's up to us to keep it at the visual.
JMO.
“The willingness to accept responsibility for one’s own life is the source from which self-respect springs.” ~Joan Didion