He's not thinking anything about what you said. He's just not interested enough to consider what you meant. If he was interested enough, he would have replied even if his response totally ignored your comment.
You say you didn't have sex. Did you have sexual interaction? (oral or anything in the like?) Sorry, have to ask because too many people these days think that just because there was no intercourse, they think there was no sex.
Anyway: I'd forget him and stop worrying that you said the wrong thing. If something as banal as "I wish you lived closer" scared him off, then he certainly wouldn't be a good prospect for anything serious.
“The willingness to accept responsibility for one’s own life is the source from which self-respect springs.” ~Joan Didion