I'm starting to be able to see myself passing time with him just as a friend, without trying to flirt with him or get physically close to him
If you think you can handle being a friend at this point and are certain there are no feelings for wanting anything else then sure it is ok to stay friends. You just need to be ok with stuff like if he is flirting with others or meets someone else (not for a relationship just casually). Don't be his friend for the sake of being able to be in front of him and in his life so you can keep tabs on him. Maybe ask him if he is feeling comfortable with the whole friends thing at this point or if he needs more time? It sounds at this point the feelings are still there for you both so if it is only friendship you are both after more time may be beneficial. Then you can start again as friends down the track and see if it develops into something more. Be careful not to put your life on hold waiting for him though, you could miss out on a whole heap of wonderful opportunities otherwise.
Some people are drains and some are radiators... Keep clear of the drains and hug the radiators!