If think the only issue (if there even is an issue) is the wording: "size matter". "Matters" would imply that size is a specific concern. I think the reality of it is that certain sizes (whether it means length or girth) are just more preferable to others. So sure a particular woman may prefer a giant vagina-splitting penis which tickles the bottom of her lungs, but if she doesn't get that from the man she loves, does that really matter? I would say probably not, it would have just the cherry on the cake.
I think it's curious that the flip-side doesn't really come up. Vagina's come in a wide range of shapes and sizes, just like penis do, but you don't really hear "tightness matters".