Based on this, it sounds exactly like that, let the man plan everything, let him so all the work, and never initiate because your fear seeming needy and desperate.
The balance is doing what you feel comfortable with, calling and texting when you feel like it, sometimes planning things, and sometimes just doing your own thing. You're here pretty much telling us that you expect the man to come to you, and as soon as he pulls back, he is guilty of disappearing. What have you done to retain these men? When the guy pulls back, to you contact him or just let him go?
I'm starting to understand this a bit better now. I think you're just not putting in the effort when you find someone who does like you.