"The best advice I have gotten from my therapist is when someone threatens to hurt themselves the only thing to do is not react and say "It would make me sad if you chose to hurt yourself". I have done this several times. It is a way of saying I care but if you do so then it is your choice. It is a bullying tactic. Don't give in to it."
Regardless of his affairs, don't you suppose that if he makes such a statement, he must really love you? Are you really sure that, for all men, having an affair is always purely a considered act of willfull commission? I'm willing to bet that the therapist who advised you that "this is only a bullying tactic" is a woman. Will she give you free therapy if you leave him, he subsequently kills himself and you are then consumed with guilt for the rest of your life? Sigmund Freud once said that the most common cause of neurosis was guilt over the way the patient had treated someone who is deceased, because no apology or compensation to the departed person could ever be made!
If modern history and human experience teach us anything, it is that you don't do something if the possible consequences are both utterly unacceptable and completely irreversible. Yes, it is a decision that only you can make and, regardless of what anyone else says, only you will have to live with the consequences.