No it WAS sexist what you said, Michelle. It's not a big deal to just let it go and learn from it so that you don't keep saying things like "most men" but rather use other terms like "The men I know" or "many men." It's symantics but is riles people up when you make blanket statements about one sex or the other.
You can't base what a whole gender does the MOST of on the few men in your inner circle.