A lot of people say things before they think of the outcome. I think this is one of these examples. he had said something, not realizing it would hit you the way it did and then when you got so upset he wanted to drop it instead of talking about it. He may not be trying to blame you for holding him back but just thinking of how differnet his life could be. I think it's good to go to the therapist, and try to figure out what his intentions were on that remark and go from there.
If you can't handle the thorns, don't crave the rose!!