I am not sure what you mean by casual interaction. Are you doing the things that dating people do, such as going out to dinner, etc., having sex and planning trips together? If so, that sounds like dating to me. When you say you have not expressed any romantic interest whatsoever, does that mean you never say romantic things, like 'I love you' or never indicate the two of you could have a future, or have you perhaps indicated that things might possibly change after six to nine months?
It sounds as though you may not have been clear enough. You need to tell them, 'We are dating casually.' I am not interested in a relationship.' 'I enjoy your company, but that is as far as it goes.' If you say all of that, and he or she still wants to 'non-date' you, then the onus is on them to settle for a casual relationship that is not particularly meaningful, but which may provide some fun and something to do to pass the time.
To answer your question, if they say you are 'too difficult to know, or 'too hard' that tells me they are not clear about the situation. Again, if you indicate that things might change after nine months, then naturally, some die-hard types will stick it out to the bitter end in the hope that things might change.
Consequently, you must tell them clearly you are not committed and then let them choose what to do next. As an aside, you are in a relationship, just not in a committed one.... it is strictly casual. Just tell them that.