Good question Larry. So far, it hasn't been a problem for me. I use one of those dusting feathers to lightly remove dust (the real, house dust) from the models. Didn't affect the chalk pastel yet :)
When I apply the chalk pastel dust, I generally tend to really work it in, then blow off the excess. It tends to adhere to a flat finish very well, and the light force the dusting feather applies is nothing close to what I do with a brush, so I guess that's why it's stayed on so far. My first one that I used chalk was made over a year ago, and I tend to dust once a month at least. It's still perfectly fine.