Women guard the gateway to sex. Men guard the gateway to commitment. He is not gonna give it up too easily. Do you know why he doesn't want a relationship? Is he at least willing to be exclusive? Does his history show a lack of intimate relationships? I think it is important to determine if he just isn't ready yet, or if he has a lifelong pattern of relationship avoidance, or a long string of short-term flings. By the late-thirties, most men have been burned enough to learn to be careful. That is not to say they don't want a relationship, but they have become jaded or skeptical.
You didn't say if he is making any moves on you. By 4 dates there should be some sexual interest at least. If not, you may not be up to his standards.