Custody battles have certainly been initiated over less serious, less substantiated concerns than yours. Your chances of prevailing are going to depend on the history of your case, the disposition of the judge and how well each or you is represented.