I'm the same way and I use self-depreciating humor to balance that. I also tend to wait for a while to whip that out, as it can put women off (see example: you. )
I'm only being harsh to try and show you how silly you're being. If the above is true, then stop worrying about it. He's NOT, only you are.