That's odd....
I have a pastel set that has a few sponge type applicators with it (they also have them in the womens cosmetic isle, like a reusable swab on a plastic stick. I don't even have to grind any off, just rub the applicator in the color chalk I want and apply to the model. It does come off when touched so, it should be the very last technique applied to the model. Some pastels will even disappear if a dull coat is shot on top as a sealer.
Are you using a brush to apply the chalks? Maybe they need something a bit more stiff to get them to rub into the piece. I would really suggest trying a different applicator.