Don't ever go back. If your curiosity is too much then just call him and ask him why he wants to meet you? If this is a married man def don't ever speak to him again.
I've gone backwards before and it was dumb. It was my mistake and my doing. I thought waiting 3 years would change things, it didn't. Luckily I didn't have sex with the ex again, but that's where it was headed. I think he just wanted sex and when he realized it wasn't going to happen and things with me were too complicated for him...he bailed (again). Douche.