Divorce Support Forums: Husband has custody Ex wife needs visitation

OK long story-- I have 2 children and me and my bf had them for the weekend and my youngest got a diaper rash from peeing in his diaper. I immedietly started treating it and it looked like it got better during the day. At the end of the day we returned him to my ex husbands mothers house whom took it upon herself to take him to the doctor. The Hospital decided that they thought it was a burn and not a diaper rash which was total bs. So my ex husband said that if we did not meet with him at 6 the next night he was going to file a report against me for child abuse. I considered this as a threat so me and my boy friend called the police and met us at his parents house which we thought would be the right thing instead of making a big uproar. In the midst of all the conversation his girlfriend tried to take my smallest child away from me and physically tugged on him. The ooficer stoped us and told her to go inside. Can that be labeled as child abuse? He then said that we would be going back to court. He also said that he was not going to let me see the boys every weekend like we had originally agreed 1 year ago. He has joint physical custody. Can he do that or is that against the law for him to take my kids away from me? I miss them dearly and I would do anything to be able to see them. Any input would be helpful. Thanks