man never wants to have sex

ok so my bf and i have been together for just over two years. we started off as friends, and i had let him know i was into him early on. he wasn't ready for a relationship, but over time he says he fell in love with me. right before we made if official i asked him if my weight was a problem. (i had seen his ex gf, an she was super skinny, almost anorexic skinny, but she wasn't pretty at all) i asked him to be honest with me because i could handle the truth especially in the beginning, before we had got to serious. he told me very addimently that it wasn't a problem for him. a few months later we moved in together. one thing always lacked for me in the relationship, and that was our sex life. we hardly ever had sex, and it always seemed like he didn't' want it. which is strange to me, as i was a very sexual person in my past and i use to have to 'beat guys away'. i asked him about it and he said he was just not a very sexual person. that bothered me a lot because he was only 25 at the time, and i had dated someone significantly older then me who wanted to have sex all the time!
it got to the point where i told him i didn't want to cheat and i didn't want to be with anyone else, but i couldn't go on not being satisfied in our relationship. i asked him if it was to much to have sex 1x a week. he said it wasn't a problem. but yet, it never happened. keep in mind that when we did have sex, it was mostly just fooling around, i would give him head, and he would use his fingers. but when we had sex, i ALWAYS have to be on top, and he never lasts more then two minutes. (i'm not exaggerating here)
last night i told him i felt like he didn't want me, and i could see he was trying to make me feel better. we were watching a movie on the couch and he kept asking me to cuddle with him. i could tell he was horny and when i went to touch his dick it was rock hard. i thought i would get us started by giving him some head, but when i stopped so he wouldn't cum, and we could have sex, he just wanted me to keep doing it. so i did, and he came. after i brushed my teeth he asked me if i was ok, and i said no, because i was left horny, AGAIN. then he sheepishly said "i can still have sex with you later tonight"...
i was upset, but still cooked dinner for him and then he went to bed. later i joined him and started touching him when i climbed into bed. he woke up and got angry with me (even though his dick was hard) and went into the living room for a while (probably until he thought i was sleeping) and then came back to bed and turned on his side. then he got up for work without saying a word to me.
once, a long time ago, when we were drinking i asked him if our sex life was affected by my weight. he admitted to me that it was, and after that i was heartbroken. why wouldn't he be straight up with me BEFORE we moved in together? a lot of time has went by since he admitted that to me, and to this day he won't bring it up or talk about it, and if i bring it up he tells me he loves me and that he was drunk when he said that and it has no meaning.
how am i suppose to believe that?! especially when he doesn't' care if i get off or not. i've had opportunities to cheat, and there are a lot of guys that are interested in me. i just can't bring myself to cheat (when i know he's never cheated on me)
also he has told me his fantasy is 'anime porn'... how the hell am i suppose to compete with that??
i can't move out because i'm not working at the moment but im getting really angry/sad so i don't know what i should do
any advice would be greatly appreciated.