He may have forced you into an agreement, but the state will not uphold it when he is violating it and not taking his time with his daughter. You need to find out the minimum number the state will give you and then understand that you don't have "normal" and the state will give you more to help offset care issues.