Emereldess, just skip over the negative comments and let them waste their time typing.
I agree with you that you should care, and not because of what anyone thinks, but as you said of how negatively it can impact your relationship. So you are smart to think about the consequences and not just let them talk. Words can put worlds in ruins. So you do need to stand your ground - if you let them walk all over you, they will do it and it could have a big negative impact on your relationship if you let them do this. Your boyfriend should be always on your side, and by your side with this. Your relationship is only between the two of you, if other people get involved it'd be like two people trying to steer a car. You two should be as one, and if that's the case steering your relationship would not be a problem.