I feel for you Tessa. I can't give advice on what to do, but I can give some ideas to think through which would give more clarity.
About your husband wanting to be with you still. Some people are understanding and forgiving of an affair. While I wouldn't condone an affair, I would forgive if it was a one off. So, perhaps your guy is one of these types. Alternately, perhaps he's the kind who can't function without you - in a sad and desperate kind of way. This can happen to people who are lacking in self esteem and self worth. Do you think he's forgiving or sad and desperate?
As for the affair, I tend to believe that we become much more at risk of having an affair if our needs aren't being met in the relationship. Now, you say you want to leave; is this because you have been unhappy for a long time or is it because you don't like the current situation?
I'll come back after you've had a chance to read this.
Never regret anything that has happened in your life. It cannot be changed, forgotten or undone. So, take it as a lesson learned and move on.