Confused on what I should do? Break up?

Ok so clearly my bf is not over his ex. She has a bf at her college which is an hour and a half away from our school. They live in the same town at home. I live an hour and half away from his town. We go to the same college. He said they dated for 3 months after high school before he went of to college a year ago. He said she ended it with him because she found someone else at her college which basically means she cheated. He was really upset and depressed for months. She was his first girlfriend. We've been together for 4 months. He told me 2 weeks ago when he went home, he went to her house to visit her and check up on her. He said he was really close with her family and that they really liked him. I didn't like this at all because of his ex really hurt him, why would he go visit her? I got upset. He said they talked and agreed to be friends. They text each other now. He saw that I was so upset about this and that he didn't understand what was wrong about seeing her and I said, exes are in the past. We fought for a couple days about this and he said he's sorry for treating me that way and that he wants to make this work and that he swears on his mom, there are no other girls. When we were together, last week, she texted him asking what he's doing tonight. His tone was almost like 'wtf' he ignored the notification. She then texted him '(his full name) helloooooo??' He ignored it again. I realize I shouldn't of done what I'm going to say next but when he was sleeping, I looked at his phone messages and it was like after the 'hellooooo?' He said, 'getting drunk and having sex' that's what we did. She then texted him again but I saw he ignored her message again saying 'helloooo?'. Before all this happened, once when we were in the shower and we were getting out, he said, "it's a dangerous world out there Nat" his exes name is Natalie. I ignored it because his younger sister's name is also Natalie. Today, he was driving my car. He had to go to the library to tutor someone so I was going to drop him off. He was backing out of the parking spot and he said, "you shouldn't use your wipers when it's freezing like that because they can get ruined Nat." He was backing out and he's like, "****" like I think he realized what he did wrong. It was a quiet car ride. He said sorry for being edgy that he was anxious because he had to be somewhere. He then turned my car around so it would be easier for me to drive out. We kissed goodbye and he asked if I'd see him tomorrow and I said yep. He then kisses me goodbye tells me he loves me, and closes the door. We mentioned taking a class together next semester. He also invited me to his fire department banquet back where he lives in 2 weeks. I'm not sure if we'll be together over the summer but I'm scared he could cheat on me with his ex. I love spending time with him and he's the first guy I've continually have sex with. I've met his family too. Sometimes I'll mention a guy or say "someone else" and he gets concerned like, "who's this other person?" So it seems like he could be jealous. What are your thoughts on this? Should I do anything? All I know is I don't want to break up with him but there is definitely a lack of trust now. I don't know what to do. 😞