Hmm.. what do you talk about when you're golfing? Is there any flirting?
If I were you, I wouldn't ask him why he hasn't invited you-- it's a weird thing to ask and puts him in an awkward spot.
As it stands now.. you can either back off and see if he takes an initiative, or you can try asking him to something other than golf. I got the vibe that he may see you as a golf buddy or just not want to turn down the opportunity to play.