The only person we can advise is you. You need to stop allowing your son to control what you do. Go on your vacation. Leave the children up to him. He is their parent. It doesn't matter how much you have them - the court does not care.
Your son needs to start taking care of these things. Three years is a long time for your grandchildren to be held hostage to these two people.