You want an emotional relationship with him, he just wants the physical side. He cannot give you the emotional closeness you want. So he wants you to move on. And he's been honest about that this whole time. You have no right to be angry with him.
We fooled around because he told me he just wasn't ready for a relationship. Then later he told me he just never had any romantic interest in me. It started a fight between the two of us because I felt like he lied to me.
...that we couldn't start a relationship just because we are [only physically] attracted to each other.
I have a long time interest in psychology, specializing in relationship dynamics for 30 years.
(Please note, we give the best advice we can based on the information given in a post. For better advice, please include the age of all romantic partners.)