I don't buy into the notion that women are more emotionally vulnerable than men. Many men just believe they have to hide their feelings. In some ways, that makes getting hurt emotionally worse for a guy. So seeing that somebody has to say, "I love you" first, I don't see why it can't be the woman.
Being afraid that a guy will hold it against you isn't a reason not to say it either. If anybody is going to hold your feelings against you, then you shouldn't be with that person anyway. Period. So the possibility that the person would continue to use it against you shouldn't exist. Besides, women can be just as verbally abusive as men. In my house, it was always my mother who was verbally abusive and my father was the one who just stayed and took it.