Wow...why are you still with him? How about this... Take all his belongings (whatever you can't sell) and throw it in the front yard. Then, have he locks changed. Get a lawyer, Get his name taken of all your accounts/house/bills and file for a divorce. Do it. Who cares where he was?
He isn't even treating you like a friend let alone a wife and mother of his child.