NO - it is not their choice to visit him. It will be a court order and the biggest responsibility you have as the custodial parent is to support the kids visiting him no matter where he lives (unless dangerous) no matter how you feel about it.

They need their dad and yes what he has done is wrong and morally terrible. But your kids need to build their relationship with their dad and you need to foster and help that.

Think of it this way. If they were saying that about your mom's house - would you still say that they had a choice?