Document everything that he does. How much notice he gave you. His reaction when you wouldn't change what he had requested (and document why you woulnd't). I would also document the extra time that you give him.

He is going to come across for what he is, someone who isn't reasonable and is trying to use the courts to control you.

The "situation" is of his own doing. He gave you dates for the summer, you made plans around those dates. You have him extra time, and he still isn't happy.

BTW, the no contact rule works well after divorce, too. Obviously, you have a child so there is going to be some contact. But once it starts straying away from the child, end it. And if the conversation becomes an attack on your parenting or telling you how to parent on your time, end the conversation.