What is keeping you both from ending your marriages right now and starting a life together? I think if you can both have an honest conversation about that, you will be able to move forward accordingly. Are you certain that his marriage is in such a bad state? I ask because sometimes people are not entirely honest when trying to maintain two relationships at once. As for your marriage, under the circumstances, you would be perfectly justified in leaving and just remaining friends. Between all four of you, communication is the key.