If you don't make plans to switch week-ends or time beforehand, then, yes, you lose the time, especially if you are the one who canceled.
Now if she took the kids out of town during your time, you need to make it clear beforehand that it is your time and you aren't forfeiting it. If she wants to take the kids during your time, she is going to have to make up that time.