Yes, it is certainly a mind game extraordinaire--and it can feel devastating, especially when it comes out of the blue: you are caught off-guard and it's hard to know how to respond and not undermine the other person in trying to defend yourself.
Calling out how inappropriate the other person's behavior is (in trying to undermine you), rather than attacking the other person is a far more effective approach. The underminer is trying to turn the tables on you (using something you've said or done to cast you in a bad light), but instead of attacking them, you can point out that their assessment of you wasn't sought out. In private you could ask them what part of them is so damaged that they have to undermine someone else to try to make themselves feel better. The last thing they wanted was THEIR behavior examined. So in a sense, you've now turned the tables on them but done so in an appropriate fashion.