So whats your problem? Hes still your boyfriend doing the right thing, wanting to be with you, quit even his job what was unecessary I think.
Last year I had a GF of 8 months and relationship wasnt fresh anymore - I was not interested in sexual things anymore and When girl asked to I really love her I hesitated with an answer. Anyway I was going tru some major health problems and my GF supported me emotionaly and was sticking by my side. But I fell in love with nurse who cared for me. Sure few weeks cant be enough to really fall in mature love but it could be called infatuation. Anyway despite that I was still a good boyfriend cause I paid attention to my GF on dates and still wanted to kiss her and want to be with her. I was doing the right thing. Never made a move on nurse or let her now otherwise but feelings was there for few weeks.
See it as a test to your relationship. Your BF cleary have passed the test and gotten over it but you still have to pass this little test in relationship and got over your trust issues.
So it seems like your BF have the same problem only mistake he did was telling you about it cause you act too sensitive what is not recipe to happiness. When you are confident you know your BF will choose you over other girls and your long relationship are safe even with some other girl temporary on a horizon. Feeling of jealousy and insecurity dont exist when you know what you are worth. Looks like you dont know it but your BF did show you it by doing all the things to be with you.
I say take him back but if after few years trust is not back then it will never will.
Also I think you could do with couples counselor or therapist and perhaps see sex therapist too if you come back together and want to restore and refresh sexual life.
Doubt kills more dreams than failure ever will