Worst case scenerio you will have to stay where you are....if he says he doesn't mind you going back close to your family then maybe he doesn't...I don't understand why he would want you to move away and it really isn't fair to the kids that he doesn't mind but I would do what works for you and the kids and make the best of it...personally I would try to stay close and let the kids have a relationship with him if he can be a positive, loving person in their lives...they are going to need the both of you...