Bad dating experience has affected my self-esteem

So basically, I'm pretty confident when it comes to dating. But a couple of months ago I went through a horrendous experience. I went on holiday with my brother, who himself is going through a hard time, and he lashed out at me. He spent six days relentlessly destroying my character. I tried to stick up for myself, I tried to walk away, but I couldn't get away from him. Me and my brother are extremely close and I've looked up to him throughout my life so this was kind of like my idol telling me I was a waste of space.

When we got home, I all but cut him out. I wasn't having any of that in my life. We still speak on occasion but I don't consider us friends and I don't think he does either. Having done that, I thought I was fine.

But six weeks later I started dating someone. He was the hottest man I've met in about eight years, we had loads in common, had a really amazing first two dates and we really liked each other. I was totally confident after date 2, but I sent him a text between date 2 and date 3 that he didn't respond to and I started worrying. I knew my worries were crazy - I discussed them with a friend and we laughed together about it - but when it got to the day before date 3 and I texted - then six hours later called - to organise what we were doing, to no response, I totally freaked out. I thought the date wasn't on, got upset about it because I was really looking forward to it, but accepted it. Then late at night I got a text saying sorry he hadn't answered my call but he was very hungover from a stag do and recovering. I told myself I need to chill out and stop being silly, but after my assumption I wasn't hearing from him again, I felt quite anxious.

The next day, he came over to mine for the date and I couldn't chill out. We'd had loads of fun and I'd felt really safe on the first two dates but this one was more uncomfortable. I really couldn't act like myself at all and I could tell I was making him uncomfortable.

Then started the really bad anxiety. I went through the next five days - in which he didn't ask me out again - feeling sure I'd screwed up. I texted him and he texted back, but he didn't ask me out. I tried to feel reassured but couldn't. (Normally in this situation, I would just ask him out...so this is unusual for me already). By the fifth day I was paralysed with anxiety and I realised this wasn't fun for me anymore. There was no point in dating someone if I'm going to feel anxious about it, I thought, I've never felt anxious when dating before. So about an hour after sending him a text to no response (although I didn't see that as a sign of anything) I decided I'd just ring him and find out if we were still dating. If he said no, at least I'd know, I thought. So I rang him, and the moment he picked up all the pieces came together in my mind and I thought "Oh God, I'm acting insecure because of what happened on holiday...". I was suddenly really embarrassed and acted that way on the phone. It was awkward, we laughed it off. My confidence was totally restored.

The next morning he rang and sounded terse with me. He wanted to know what the call was about. I didn't think it was a big deal, so I just said 'oh I'm still embarrassed about that, can we talk about that later?' and he said 'see ya' and hung up. Confidence restored, as I said, I texted him to ask him out. He got back and said it all depended on what the call was about because he got the idea I was checking up on him! I was so shocked, I didn't even have time to process the fact that's really quite an offensive thing to say to someone you barely know and phoned immediately to put him straight. He didn't pick up and I got more anxious. Thinking the real reason was, although embarrassing, not as bad as his impression I texted to tell him I'd phoned to find out where I stood and if he wanted to see me again and I knew it was a bit weird, which was why I was embarrassed.

Later, he called and was very terse over the phone saying I shouldn't expect streams of texts every night (I wasn't, frankly), he doesn't always get back to texts when he's out with people - or even on his own (I now realise he thought I'd phoned because he hadn't texted back within the hour, but this didn't occur to me at the time), that he was very busy and couldn't say when he could see me again, but would text me in a couple of days. When I brought up the word dating he said 'Well if you have to put a label on it, I suppose you could call it that'.

I was so anxious on the phone - I felt like I was being told off, basically - I wasn't at all assertive. Instead I just got progressively angry and the next day texted to tell him that basically everything he'd said the day before sounded like bullshit (he'd texted me every day before the first date - I actually wasn't keen on that but I didn't tell him this - he'd asked me out for the second date three times in one day for three separate and coinciding days etc.) and that I needed to know in advance when I was seeing him (that that's just courtesy) and it wasn't normal for me to not be sure whether I was seeing someone again and it wasn't fun for me.

He phoned to call it off immediately and I shouted at him. He hung up and texted to tell me to phone back once I was calm. I did and I totally crumbled. He reminded me this had only been three dates and I was getting very "deep" and I felt so ashamed of the way I'd behaved that I didn't call him out at all on his frankly rude and controlling behaviour the day before.

The anxiety after that day went mental and I actually suffered a total emotional collapse. I couldn't stop thinking about what had happened - I was obsessive about it - and kept wanting to fix it so I could feel better about myself. I wrote about it on here - I'd gone completely insane! I even ended up sending a message to him trying to explain what had happened, then a week later feeling very embarrassed I'd done that so sending another to say that it had all turned out positively because it had prompted me to make changes in my life and I was feeling better than I had in months (this was true, but the fact I felt I had to tell him that after only three dates makes me look weird...)

So I've come away feeling like a complete weirdo. I keep telling myself that it's natural to lose confidence after something like what happened to me on holiday and when under extreme stress people do act a bit crazy, but I still can't stand the fact there's someone walking around out there who thinks I'm a crazy woman. I know someone with higher self esteem wouldn't care, as well, but I can't seem to stop caring!

Anyone know what I can do to build myself back up?