I don't think it's selfish to not want to date people with children. I think it's hard enough to get a relationship going when it's just one-on-one, nevermind throwing a child in the mix. It's pretty much getting into a relationship with two or three or more people. I think single people are better suited to other single people. Not to say there isn't exceptions.
For me, it's just a case of not wanting to play mommy to offspring that isn't mine. I don't want to deal with ex-drama, which is always worse when they had a kid together because now YOU'RE part of the kids life and so the ex feels they have the right to scrutinize you.
I worked with a woman who married a guy who was divorced with five kids. After they got engaged, all I could hear her talking about was how the ex was a 'bitch' and that she wasn't going to get one penny more of child support after they were married (and they'd never even met). I mean, come on. What a poisonous environment for the kids.