Was the break up from his ex fairly recent?
How long were they together?
I've been in this situation recently. I'm a guy - so I guess that's the point of view you're looking for. Hopefully some of this can help..
When I was in this situation, little things that she did reminded me of my ex and it upset me and made me uncomfortable. It wasn't that I didn't like the girl that I was with, because I really did.
I'd say I still have feelings for my ex. Everyone does to some extent, whether it still be love, hate, or anything in between. There will always be history and feelings don't just dissapear - they change. So, it wasn't that I wanted to be back with my ex, I just felt uncomfortable being reminded of her because it hurt.
It may be different in your/his situation, I don't know. I'm just letting you know how I felt.
My advice would be to take things slowly. If you want to be with him, and he is showing signs of liking you (from what you've posted - he is) then you just have to stick with it and don't push for anything. Right now, intamacy in HIS mind is linked to his ex. It just needs some time to be transferred over to you.
You COULD try asking him about it if he pulls away again - but if you do you have to remember to be very understanding and supportive of what he says. Don't force him to respond, and don't interrogate him.
Put it in a light, friendly way and he might tell you what he's thinking.