No No No and absolutely not would be my response. I would not be ok with that and you have every right to be upset. Maybe you should tell him that during the time that his 'ex' wife is there, you will be staying somewhere else to give him "family time"... Tell him you'll be staying at an ex-boyfriends/husbands because you (like the ex wife) have nowhere else to go. See how he feels about that!
-------------------- "I am not afraid. I was born to do this..."