I have a feeling your relationship has lasted as long as it has simply because you only have to be around each other on weekends. If you were in a position to see each other regularily, both of you would soon get sick of one another and each other's quirks and ideas and inability to be flexible.
You are having trouble already and if you do the math and add up the times you've actually been in one anothers lives it's only 1/4 of the actual time you consider yourself to be in a relationship with this man. You are seriously still in the honeymoon period of your relationship where you should both be wanting to please one another and strive to compromise when things can't be 100% mutually agreed upon. I think his behaviour is a red flag that you shouldn't ignore. You need to make a rule that one weekend you go to his and the next he goes to yours and if he can't agree to the compromise and stick to the agreement you should heed the red flag and call your relationship as having run it's course.
He's got a psychological disorder where (IMO) he needs to have to undo extra stress or anxiety and set rules and relationship boundaries should be in place in order to avoid confusion and power struggles. If he can't adhere to the rules then you know you need to distance yourself from a relationship with him.
I do want to talk to him, but I dont want him flipping out on me.
If he would do that over you trying to make things less complicated and happy then there's another good reason why this relationship will never last the test of time.
“The willingness to accept responsibility for one’s own life is the source from which self-respect springs.” ~Joan Didion