Look, he has kids to care about and wants to be with them. He also wants you to be a part of his life, and as such has invited you along.
So, either you can back out and look like a chicken to the ex wife, or you can suck it up, sneak a few shots of rum in medicinally, then take him home as dinner and violate him 3 ways from Sunday. Or, go buy a remote controlled vibrating egg, and hand him the remote in the car. He'll smile, you'll smile, and the whole time you're violating the ex's house.
Win/win.
"Well, then," the Cat went on, "you see a dog growls when it's angry, and wags its tail when it's pleased. Now I growl when I'm pleased, and wag my tail when I'm angry. Therefore I'm mad."