WHOA! Sorry to hear about the heinous attack on your collection, guy.
I personally never had a problem along those lines, my Dad and I shared the hobby. My Mom & Sisters never really joked me too much, only when I deserved ribbing for something else I had done!
I think they realized that I took it fairly seriously, and the fact that my Dad was also a modeler just lent further "legitimacy" to the hobby in their eyes. Plus, I make it known (sometimes unpleasantly so) that I am not pleased when someone does something that shows total disregard for the physical condition of anything I own, including the models.
I don't tolerate folks who don't know how to handle things with care. A couple of times when I was younger I reamed my Sisters out for accidentally damaging my built-ups. I felt bad after, 'cuz I usually had them in tears, but I got my point across. Nowadays, I'm more diplomatic and have learned to use words rather than simply shouting at someone.
I suggest that you try to explain in a very stern but calm manner that regardless of what they think, it's YOUR stuff. Another favorite tactic I've used on people is to turn their argument around on them. Find out if they have a hobby and start to trash talk 'em about it. When they get all hot and bothered, reverse again and say that your models are no different.
Show them the work of some REAL masters out there. The guys who do this stuff for a living. Whether or not YOU aspire to reach the same level that some of these guys have done, it may convince them that it's not just a hobby for kids. It's also an outlet for creativity and artistry, just like any other artist or craftsman.
Finally, if you cannot get them to see it your way, then they simply have a problem of some sort and there's nothing you can do. Any rational individual will accept any and all non-detrimental behavior (no matter how weird!) of any sort on the part of their loved ones, including "playing with their toys".
Fade to Black...