Although he is Mr. A$$wipe he is still Dad to both kids and didn't put offensive woring in his request. What is your problem letting the kids spend some summer vacation with their father? At best they will be happy, at worst they will see Mr. A$$wipe for themselves.

You're not obligated to anything other than the business agreement you signed. You can spend money and litigous time and go through court if that's what you think is best for the kids. Or you can be reasonable, just keep it business, and agree. The court will not refuse him to just spend time with his children.