Flirting is the game?

You are at work, but only have a temporary position and will leave the company in half a year. You felt an immediate click with one of your colleagues. You know however that there are certain differences between you, that you won't overcome. You know there is only physical attraction from your part. Now the question is what he thinks.

He once said "with some people, like you, you immediately feel a connection". Once, at a party, being drunk, you danced way too close, even in the vicinity of your colleagues. He wispers in your ear how much he'd like to feel you etc. You both confess you feel this physical attraction, but are smart enough not to pursue it.

At work again you treat each other like any colleague. Except for that tiny laugh every time your eyes meet? Or the fact that the two of you manage to get at the same place together alone more often than with your other colleagues. And then he tells you he is going for a swim, since it's so hot, whether you want to join. You can't because of work in progress and refuse.
But you do send him a text message in the weekend whether he wants to travel together (you both have to travel for two hours). And then he doesn't respond. Half a week later he apologizes for that.

Consider this all a big game, no chances on a relationship. What would now the next step be? Keep pretending nothing is happening, like before the drunk talk? Try to invite him to something? Wait till he acts again?

And to add: you both have a relationship, but only see your partners in the weekends. So more physical contact than a drunk dance is to be avoided. But still the game is too much fun to stop playing.

Two questions:
1. What on earth could he be playing at? Playing the same game as I am playing? Or is he too shy to really react? Or does he want to protect his relationship (but then why asking for a swim, or talking bout attraction?)?
2. What should I do? Please don't say quit with the nonsense. I know that. Just wanna have some fun. Would be awesome to see his reaction if I had the guts to ask him like that if he wanted to go out drinking tonight or something.

Please don't ask me what I want to achieve with this, or tell me to stop. I know it's useless, but it's so enjoyable... Those hourly adrenalin kicks at work...

I think that you need to be sure your bf is okay with you doing that. Otherwise, it's extremely disrespectful to your relationship if this is something that would bother him.

By the way, this is a very risky situation. If there's a strong physical attraction and you put yourselves in too many bad situations, there's a very good chance something else might happen.