Practice ignoring him on his level...instead create a business like persona that you would have if you were having to get stinky cheese from the local deli.
I have to talk to the OW all the time, as my ex married her and lets her handle everything from finances to visitation...I get to listen to my son call her mom. It takes time, but you can get to a point where you feel nothing for the act they did. If they are idiots to boot, then that can sometimes make it easier to deal with too.