Wow, so many replies!
First to all the people saying I'm fat, should lose weight, etc - I understand these things, I live in a society where everyone has to compete to be skinny and beautiful. Trust me, I've tried to lose weight, and I'd like to think my lifestyle choices are healthy. I eat a Japanese diet of rice, vegetables, and fish (and Japanese portion sizes), I work out about an hour a day, and I can run a good three miles without pausing. So anything that'd make me lose weight would require a painful change of life (starvation or hours at the gym a day) and I have other things in life I'd rather focus on. Also, if losing weight is really the answer to my question about my relationship with this guy, are you guys saying I'm supposed change my life that much for one person? Seems a little drastic... Thanks for being concerned about my health though.
To those who asked about specifics of our conversation...
So he didn't specifically call me "fat" or "ugly", but it could be implied. He basically agreed with my conjecture that his getting colder was related to my body. He said I'm totally within his passable zone and that my face is cute, but still... :-/
His last female encounter was with a friend with benefits he had for a couple of years, who he didn't elevate to "girlfriend status" because she was ugly and fat (according to him). She eventually got a boyfriend, stopped having sex with him, and then he realized that he actually really loved her but could now never be with her. So he has some strange complicated interaction with his feelings for people and their appearance. He's had many friends with benefit in the past, so him having sex with me doesn't necessarily mean anything, and I knew this going into it. He has no choice but to meet me again because our whole trip is planned out and uncancellable. He's also someone who cares very much about his own appearance, would be considered "metro" by American standards.
I did tell him it'll be hard for me to have sex with someone who finds me unattractive, and that I've had many boyfriends who find me attractive so there's no reason to be with someone who doesn't. He said not to worry about it, that he didn't care that much. Though at the same time, his attitude definitely changed - he stopped calling me pet names, didn't seem as interested in what was going on in my life, etc. I can understand his situation - if you really like someone's personality, but aren't attracted to them, how are you supposed to solve it. It's complicated because at the same time, other things were going on with him that made him more down than I've ever seen, so it's hard to tease apart what's related to us.
One of my problems is that I'm all-or-nothing, I can choose to really get attached to someone, or to just not care and use them for the sex. I guess I'm wondering if I should close my heart off to someone like this, or give it a try with the possibility of my self-confidence getting really damaged.