Bad first impression

I went out for a friend's birthday which I met a guy through me friend. We were talking and got on well. I made a joke about one of the other guys looking like a celebrity. Suddenly this guy stopped talking to me and I realised the joke had not gone down well. A few weeks later my friend tried to set me up with him which this guy said he did not like me making the comment about his friend.

I feel really bad as I am not a horrible person and now he has this bad first impression of me. My birthday friend did not see what the problem was as he thinks the person does look like a celebrity. I still feel bad about the situation and wonder how I can rectify it. Is it best not to joke with people when you first meet?