If you don't initiate conversation, it looks like you're not interested, and he will find someone else. It's that simple. Show you're interested, or he goes away.
Starting a conversation is not "being a pest". Whatever gave you a crazy idea like that?
I have a long time interest in psychology, specializing in relationship dynamics for 30 years.
(Please note, we give the best advice we can based on the information given in a post. For better advice, please include the age of all romantic partners.)