First, I'd suggest speaking up for yourself, to him and his mom. Let them know how destructive they are, not only to you but the relationship in general. Second, I'd suggest finding things outside the house and away from them to do with your kids, like field trips, outings to the library or local recreation center. Start building a support network among people who have similar interests to you. Churches and volunteer groups are great places to meet people, and you won't feel so isolated.