Wait a minute. There is nothing wrong with that. I fondle my woman often when she's asleep, and she simply smiles and continues to sleep. Sometimes, I pull down her top to suck on her breasts. Sometimes, I pull down her underwear to rub her bottom/front.
One time I tried to stick my penis inside, but the position wasn't right, so I was foiled. She woke up and asked me why I didn't proceed. So the next time I just got on top of her and humped away. She was very pleased. So I am wondering what the problem is for the OP. It makes no sense!!!