You have no choice but to follow the court order or you will be in contempt. And one of the things that may happen if you are in contempt enough is that you lose custody.
Have you tried talking to your ex about your son's feelings? Encouraged your son to talk to his dad about them?