First thing, he was rude to you by dumping you when you found out he was unfaithful in the relationship. Now that the relationship with his new love did not go well, he has come back and visits you to show you are still the one he loves.
Yes, you love him, but you need to take it slow. You need time to heal and to ponder or think things through. Would you want to continue in the relationship with a man who dumped you for another woman? Are you strong enough to continue in the relationship?
You need to stop seeing each other. You should stop communicating with him unless it's an emergency. You need to stop him from visiting you at your home or tell him to cease visiting you. This means you need to exercise no contact rule. When there is no communication between the two of you, it will allow the healing process to begin. You cannot get back in the previous relationship while still in pain. It will lead to more heartbreaks. You need time to heal and to be strong to stand on your two feet. Also, you need to know whether it is sensible to get back with him or forget about him and move on with your life.
Thus let him know you do not want to communicate with him for some time and the two of you should stop seeing each other for some time. If he asks why, tell him to think things through, to ponder and to heal from the wound he inflicted in your heart and to find the strength to move on. At the end of the no-contact period, a month is preferable, you would know which direction to head to, and the pain will have lessened.