I could NOT be with a man who hated my child. That's straight up dsturbing.....and callous.
I was married to a man like yours. I took him back again and again while he waffled back and forth. It took me a long time to realize how selfish he was and how much emotional damage he was doing to me. I divorced him, and in doing so learned an important lesson about what behaviour is unacceptable in a relationship. I'm now engaged to someone who loves me absolutely and treats me with the respect I deserve.
You can have this too. If he can't be a respectful partner, he needs to move out and you need to stop letting him come back. Tell him until he makes up his mind completely, he needs to stay out of your life in a relationship sense. Maybe while you're on your own it will give you time to reflect on your own needs and happiness. Don't stay with him simply because you're afraid of being alone. You can't be in a healthy relationship until you are self-confident, happy, and in control of your own life anyways.
Live as if you were to die tomorrow. Learn as if you were to live forever. - Mohandas Gandhi