Whether he's punching you in face or not, you said he never really "beat" you, he is in fact abusing you. That abuse will eventually turn towards your children as well. You stated that he refuses to go to counseling. There is your answer...if he won't seek help then he will not change. He's probably giving the same excuse my ex did which is that "it's not all that bad, he never actually hit you, he's not abusive" next it will be him stating that the abuse inflicted on the kids is "discipline".
Leaving is never an easy road - but neither is staying. Get support from everyone you can and at very least go to counseling for yourself.