I don't think this is something you should ignore. Think about it... He knew you were gone and attempted to deliberately disrespect you and your marriage by touching your wife during that time. Nothing has happened since that time, but something tells me he's rather cunning in this regard. I'm sure he knows better than to attempt something again while you're around. Does he know of your anger management history?
I guarantee you that not addressing this in a calm fashion early on could result in you collecting deep-seated resentment toward this guy. Repress it and it'll only become a bigger monster. I think you should very calmly write out a detailed way to handle this, all the way down to your lines of conversation. And rehearse it. Stick to what you've written. Talk to a counselor before you execute this plan if you want. Get some feedback.
Unlike your wife I would have told that MF off myself in that moment. But that's because I don't shy away from confrontation. If this guy is as much of a creep as I think, he'll view her silence as permission to continue to pursue when the timing is right.
