Hes an asshole. You need to cut ALL contact with him and move on. He has made it almost impossible for you to move on so far by giving you false hope and mixed signals. You need to tell him to f**k off and also throw in their for good measure that its his loss and he is a fool to let you go coz your the best thing he could ever have and if he doubts that-its his problem-the next guy wont let you go.
Seriously move on-he is an asshole.
In future stop being so cautious. Why should he be allowed to "set the pace". It should be equal. If your two passive and have no fire in your belly-its unattractive to a lot of guys. Be more fierce and passionate and go after what you want and in future if a guy isnt meeting YOUR standards or expectations-you get rid of him
"Don't ask a question if you can't handle the answer".