Sweetie, take a deep breath and look at this objectively. Yes, it takes a lot of courage to ask someone out - but much of the time it doesn't work out and we need to be prepared for this. This is why finally finding The One is so special.
You were with him for only a date or two. It's so important to NOT get emotionally invested this early on because this is just the 'getting to know you' stage. To be honest, he did nothing which should cause you hurt. Disappointment, certainly but he wasn't hurtful. He simply saw you a time or two and then was honest with you about why he felt that seeing you further wouldn't work out.
This post, combined with him mentioning you were rushing him makes me wonder if you were too intense for him. If perhaps you were expecting a lot to come of this while barely knowing the man. While PC thinks it was sweet of you to hold his shaking hands, I'd have to say that if someone I had only dated once or twice held my shaking hands and spoke quietly and gently to me about my drinking, I'd run a mile! It's WAAAAY to intense.
I don't want to make you feel worse than you do, but please try to look at this with some clarity. All he did was accept an invitation out with you and then shortly after said it wasn't working for him. He didn't make false promises and he didn't string you along. He didn't take advantage of you.
Never regret anything that has happened in your life. It cannot be changed, forgotten or undone. So, take it as a lesson learned and move on.