The act itself wouldn't be immoral in retrospect. However, the decision (before the fact) to do it would be immoral because you are taking a risk that is likely to cause pain. You cannot have "good" casual sex without making an immoral decision to do so first. Therefore, casual sex is necessarily immoral.