Boyfriends family...

Hi everyone! I'm new here, I've lurked for a while but have never posted before. :)

I'm almost 21 years old and have been dating my almost 23 year old boyfriend for 14 months.. he's my first real boyfriend and I love him lots! I'm white and he's Asian, which shouldn't really matter but I think it is the root of the problem for his family unfortunately. I don't talk to his parents, I see them probably once a month when I go buy something from their store (where they're always all smiles!) other than that there really is no contact (his younger siblings love me though! haha). He recently moved out (I wasn't welcome at their home anyways) and it's been really good for our relationship. :D I thought his mom had gotten over her hatred towards me and even though I knew she didn't like me, I thought she was just dealing with it but then a family friend of theirs was talking to my boyfriend and saying how I was such a horrible, inappropriate person and his mom was so upset that he was still with me! I'm pretty upset over this, number one I've never met this lady and she has no right saying that to him and number two I'm mad that his mom is willing to spread horrible rumours about which clearly aren't true since she's never had a conversation longer than 30 seconds with me! While I've accepted the fact that she doesn't like me and I don't care anymore and I can usually just brush it off (I'm used to it by now) I'm really bothered that she just goes around talking negatively behind my back, it's one thing for her to hate my but to share it with the world is another. I'm someone who hates conflict, I'm a quiet, shy person who will go out of my way to do nice things (I even used to try with his family but gave up months ago! haha) so I really don't want to make a big deal out of this but it's really, really bothering me. I spoke to my boyfriend and he's really upset at the lady who said that about me but I don't think he'll really do anything in fear of angering the family (which I understand since they're crazy). I told him pretty soon I want to say something to his mom if he doesn't, but he freaks out when I say that because we both know nothing good could ever come from that, it would just make me feel better! lol.

So basically my question after all my rambling is, what do I do? :upset: I love my boyfriend but whenever his mom says stuff like this I start to wonder if I'm like ruining his life.. would everything have just been easier for him if I wasn't with him? Is he going to end up hating me for making his family hate him? I'm so lost right now and feel like I need an outside opinion...