I'm sorry he is acting that way, there is no reason for a Father to take his frustrations out on his children.

Have you set up a visitation schedule yet? That may be one way he will step up and do the right thing. If he knows when and how long etc etc then he can plan accordingly. Being able to say you can see the kids whenever you want to is not the same as saying I get to see the kids on XYZ. It will also take the burden off of you as the bad guy. Unfortunately it sounds like you will still be answering the questions of "why doesn't dad see me?" and my heart breaks for you and your children because of it.