I just realized some of your argument of him not taking the kids during certain times was due to school...but then you went on to say they were 2 & 3 years old? The kids aren't in school yet and there should be no reason why he shouldn't have extended time with him until they reach school age which includes summer breaks vs time during the winter.
Why are you being so difficult? Do you not understand that a father has just as much right to parent the child as you do? He probably did the simplest thing he could when you divorced and due to the age of the children at the time, it was practical...but the kids aren't "babies" anymore, their toddlers and almost preschoolers. As their ages increase so should the time they spend with their father.
If you were critical of him and his right to parent right when the kids were born, it would explain why he so quickly agreed to such little.
I can only imagine how you come off to him in your communication...