This doesn't seem like a very sticky situation. It sounds like you are single and are able to date whomever you choose. And why would you be worried about being hurt by this guy? You said you were casually dating him when you get serious with someone else. So unless I am missing something, you dated this guy casually a few times and then ended things because something you thought was more promising came along.
Personally, I think that Bob is the one that should be a little cautious here. But the fact that he gave you his number if you want to talk is a step that he made anyway.
At this point, it is probably just friendship, but you never know what it could turn out to be. Don't close yourself off from possibility.
Good luck.
Brought to you by Dating With Devon!