PLEASE HELP: How do you know when to fight for a relationship and when to let it go?

My ex-boyfriend and I have had a history of four years (both off and on, but emotionally connected nearly the whole time). It started out as a very special friendship where we spent five or six hours in rapt conversations. He was older and he made me think, challenged me, and I also challenged and enchanted him. We brought each other to life. I had not been in a relationship before, due to a strict family upbringing, and he was divorced, but we both said we had never encountered another human being that really rocked our world like we each had.
Unfortunately, I was young and didn't know how much I really cared for him, so when he asked me out, I turned him down. Later, he went on to immediately date someone else, which broke my heart for a while, and made me despise his neediness. Later, however, they broke up and we had a chance to get reacquainted, and I really started falling in love, as did he.
We dated, and I had to endure a lot of disapproval from my family for dating a divorcee, but we kept the relationship strong -- over many obstacles, long distances, moving back and forth from state to state, and a lot of personality differences.

The good part about the relationship with him is that he has a side that is extremely romantic, sweet and caring, and very sensitive. He was the epitome of romance. A couple books would not contain all the memories we have together, what he taught me about life, and how I was swept off my feet and still have a drawer full of love letters from him now.

The bad part about the relationship is that I was so overwhelmed a few years ago with all the obstacles that I broke up, and we started a bad off-and-on pattern. I also began to see quite a bit of instability in him, which he is now taking medication for (after four years). The down-side of being with him is that he can turn on me, say verbally demeaning or insensitive things, and the conflicts can escalate very badly. The fights are very draining. A lot of what we fight (or fought) about was my lack of commitment, and his instability or anger. It has gotten completely exhausting at times. But there were so many good times together too, and we held it together (mostly) with love and loyalty.

After spending a lot of time together in the spring/ summer, he had to go back to his home state. We haven't seen each other for four months or so, and couldn't afford plane tickets. He was still upset that I was ambivalent about getting married. I have been pretty ambivalent all along, and trying to figure out why. Fear? His mood problems? The fact that I had never been in a relationship before and sometimes wanted to see if I could date other people? I have been torn for so long, but I felt like for the most part I still loved him.

I haven't dated anyone this summer, although I am in my thirties and asked out frequently. He didn't either. But that brings us to now. I felt so exhausted and numb with the relationship over the summer. He kept wanting me to fight for him, do something, and although I kept in touch and did what I could, I couldn't see the forest for the trees... I couldn't decide if with all the instability and conflict, that our best-friend relationship was worth saving or what.

So now, he has suddenly gone out with someone else that he said he may get interested in very soon. At the same time, he's been asking me to come out there for a trip. I had a meltdown about him going out with this other girl. (he's quite the flirt). I didn't think that I cared about him going out with someone, but I hit the roof. I said I wasn't comfortable coming out for a trip if he was going to insist on seeing the other girl (on outings, preliminary to dating I suspect), at the same time. I told him that is equivalent to me checking out dating websites and going out with a few men before I take a vacation with him. I told him it sounds like he wants to have his cake and eat it too, but he says that I made it clear that I wasn't in love with him anymore and wanted to go our separate ways.

I am so confused. I have a hard time letting go of first love. There was a time when it was hard for him to imagine not having me in his life, but he's adjusting, and now I am wondering if I need to fight for this relationship or just let it go? ? ? He has so many qualities that I loved, but also some that drove me mad and caused me a lot of pain. I have also been guilty of some flaws too (being too critical, being ambivalent). I think he wants to make me jealous with the other girl, and I understand that we haven't been together for months and he needs companionship. But I haven't been able to afford a ticket until now. I don't know how to sort through this mess and do what is best for both of us ... leaving for good or fighting for it.

I once complained to my dad about the fact that I was considering getting back with my ex for the 3rd time and I was getting frustrated because part of me wanted to and part of me was so blown away over all the disagreements we've had.

He said "(my name here), envision what your life is going to be like if you go back with her. Envision every detail of moving in together and putting all of your clothes in dressers together and sharing closet space, and envision what job you will have and what your week is going to look like and what you are going to do on weekends. Envision how you are going to feel when you get back with her."

Then he said "Is this what you want? If so then go back with her. Eventually you will either work it out between eachother or after perhaps your 7th, 8th, 9th, 10th breakup you will finally get tired of trying and move on to someone else."


EDIT: What he's saying is in order to understand your limits, you need to test them by pushing them as far as you can. Constantly ask yourself if you've gone over your limits (reevaluate/reflect). Then you will know when enough is enough and it will become easier and easier.

I never got back with my ex and am now don't even consider the option. I know it wouldn't be our last breakup if I got back with her, and the emotional stress is too draining to withstand another breakup from her.