I really think counting the actual days since he's exchanged info with you should be abolished. Deep down, he's keeping your number IN CASE...IN CASE he feels like cheating on his wife and IN CASE you break up with your boyfriend. Telling you about meeting his wife etc, is just a safety net for that situation so you don't feel guilty - and neither does he. This also gives him a chance to say to his wife "why would I be cheating on you with that girl - you met her!"
It's easier for a man when he believes his wife knows of the girl - it looks less suspicious.
If this man is WAY older than you, like in his 60s or 70s and he's inviting you and your boyfriend to go out with him and his wife, that's a nice guesture. But if he's complimenting you and saying he would date you if y'all were single...That's immoral and who cares who you're cutting off or not being sociable with. You need friends who aren't making you question their intensions.
Good luck and please make the right decision.