He also said: "if we go out for a drink, because you are an attractive girl, I will get "bad thoughts" and I don't want that. I said I don't mind his "bad thoughts" He said still he doesn't want that to happen.
Like I said, at first he agreed to "make an exception" and go with me, but then cancelled.
Sounds like he feels guilty about having bad thoughts about you. If you want to keep trying to tell him you understand and are ok with his "bad thoughts", but I also think he's not over his ex. Taking 2-3 years for him to get over her is just ridiculous and unrealistic. It shows he does not have the experience to manage his emotions.
I have a long time interest in psychology, specializing in relationship dynamics for 30 years.
(Please note, we give the best advice we can based on the information given in a post. For better advice, please include the age of all romantic partners.)