Long term relationship should end?

Hello lovely users!
Im a first time poster here and need some advice...obviously. I'll try to give You as much info as i can about my current relationship...
My partner and I have been together for roughly 4 years. Throughout this time we have had a lot of ups and downs and now I'm once again, having doubts.
To start at the beginning...
I had recently broken up with my long term boyfriend (2 1/2 years) when I met my current boyfriend. He had a girlfriend who was at the time overseas for 3 months. They had been together for about 6 years (since they were around 15/16) and even though he had previously had other girlfriends she was the first person he had slept with etc etc
Basically, he cheated on her with me. We fell in love very quickly and I had never felt this way about anyone before. The guilt of cheating ended up getting to both of us and he called her while she was overseas and confessed. We took about a month of for everything to cool down. There were a lot of tears and talks from both of us but then we ended up officially being together. The first few months of our official relationship were very intense as we were both very in love but also still trying to deal with the cheating. I soon found out that he was writing emails to his ex that were very intimate and nostalgic. He would also lie to me saying he was with friends but actually meet up with her. To this day I am unsure if anything happened between them. Things settled down after a while but the 'sneaking around' continued. When I asked him about it he told me they were just working things out.
Anyway...
About a year later I found out he has been flirting and texting a girl I knew. It wasn't particularly harmless and was actually a continuation that was also happening in his last relationship. He told me it was harmless - an ego boost for him but that he would end it and would never contact her again.
He decided that he wanted to move to another state and I went with him. This was a huge strain on our relationship as we were living together in a city where I knew 0 people and he knew heaps. I became like a puppy - very dependent and we almost broke up because I was so miserable. I ended up moving out with some girls which 'saved ' our relationship. I became very independant and made my own group of friends. He has never been one to hang out with my friends so our social lives hardly ever mix.
Now, his ex moved to the same state as us and basically history repeated itself and they would hang out (which I didn't mind) but he would lie about it (which I did mind!). More dramas.
The girl who he had been texting/chatting with in the past popped back up and I was ready to end it. Stupidly I didn't and we stayed together with promises (of course)
Another 6nonths or so he decided that we should go on a break. He said it was about him figurig out suff. We were hardly spending time together - about once a week, so I was sure it had other reasons. He basically strung me aling for about 4 months. During this break I met a new guy who was absolutely amazing but for some reason I kept getting that 'but it's not him' feeling so I ended it when my boyfriend wanted to end our break and get together again. Things were great for about 6 months and then we both became distant again. Like I mentioned we had separate social groups and didn't spend much time together. I got a new friend who introduced me to a lot of new people and so many were surprised to find out I had a bf as I never spoke about him or mentioned him. Our sex life soon became lacking and I had no desire to be intimate with him. On my birthday we broke up. Basically his thought were 'if she doesn't have sex with me tonight I'm breaking up with her' so we did. Vey soon after he wante to get back together but I said no. I started having very bad anxiety attacks and became very detached from a lot of things and people. He became very obsessive and would call me crying, show up at my house out of the blue etc etc. it got so bad that I moved back to my hometown with my parents. While there I was vey depressed and was still in contact with my ex. I met a new guy but once again got that 'it's not him' feeling and was reluctant to go much further than hanging out. One night we hooked up while we were out drinking. As I live in a small town word got back to my ex and we had a huge fight. When he visited our hometown he flirted with a girl that I dislike to get back at me and I soon found out she had been the one spying in me and reporting my actions back to my ex. He also started seeing someone else back in the other state. He wa still constantly messing and calling me asking to be together. One day (I guess I had a bad day!) I said yes and I decided that I only wanted to be with him blah blah blah. I move back to the other state to be with him and since then we have moved in together as are traveling overseas. A few months back the girl he flirted with while we were broken up has caused a lot of trouble. Texting him in an over friendly manner, when I visited my parents made up lies about me in txt messages to him (which he believed) and then made up rumors about me going home with another guy. We recently had a fight over it as he said its not an issue, he
Did nothing wrong, etc etc
now what I guess I'm asking for advice on is...
It's happening again. I don't want to be intimate with him at all and am rejecting him alot. He tells me how horrible this makes him feel. We have been fighting quite a lot and sometimes I've been wishing I was in Europe wih my friends cause id be having more fun. Some days are good and we get along great though. I'm worried because this same issue happened with my last ex- not wanting to be intimate. My boyfriend tells me that it's me that has the issues and that I need to sort them out. I'm wondering if I just can't get over everything that's happened or what. He is my best friend and I can't imagine him not being in my life but maybe we just aren't meant to be together as a couple? The idea of him with someone else is devastating but sometimes I think maybe he needs to be alone (as he was with his ex for 6 years and now we have been together for 4) Breaking up also seems to emotionally crushing and it was so hard last time. I know this isn't a reason to stay together but it's still in my mind. We hardly have any common interests anymore and lately we seem to want different things Eg he wants 1 child, I want a few, he wants to buy and live in the state we are in now forever, when I have a family I want to move back to my home town to be closer to my family, he doesn't think family is very important, I do! Etc etc
I know this is long and probably boring for you all to read but my head is spinning....
What do you all make of this?