It seems unusual for a man to withhold affection. Ask him if there is a problem, not with your relationship necessarily, but perhaps with him physically or mentally. Maybe he is depressed, doesn't feel well or has high blood pressure.
Marriage counseling can be intimidating, but you can get a lot out of it by going for yourself.
It is hard to feel so alone. If he won't even communicate what the problem is, it leaves you in a difficult place. I'd suggest professional counseling, just to help you manage your own experience and feelings.