Do you keep your phone records? If so, go through them and write down how many times he has called or you have called him so that the kids could talk to him.
And if you do get that letter, respond to the dates: Those won't work could he send another list of dates for you to consider? I would also tell him that the telephone harassment needs to stop, everything can be handled via snail mail.
Be sure to send it certified, return receipt. You're not going to be able to put him off much longer. And I wouldn't send him anything that states that you won't consider any of his dates. Just let him figure it out on his own.