I don't know why he has to choose either spending time with you or playing sports. It seems healthy that he has something he enjoys doing. It's not like you are not spending time at all. His enjoying playing sports doesn't mean cos he doesn't want to spend more time with you. He simply enjoys sports as much as he spends time with you.
That being said, if you don't feel fulfilling in a relationship in any way, you should definitely talk to him.
I personally would be happy with what you have. I mean, we are all busy, aren't we? But each person's expectations are different.
“Really, sex and laughter do go very well together, and I wondered - and I still do - which is more important.” - Hermione Gingold-