I've had similar feelings...I love my boyfriend and I do feel he's the one for me, but I occasionally wonder what'd it be like to have sex with another man, especially since my BF is the only guy I have ever had sex with.
My advice? There's nothing wrong with fantasizing about the women you want to have sex with. You don't need to break up, unless the urge to have sex with other women is just too overpowering and you just don't want to be with your girlfriend any longer because she doesn't fulfill your needs. And that doesn't sound like the case...yet. In my experience, the feeling comes and goes, but never fully disappears.