Yes. I think (though I have no scientific evidence) it happens when the Milliput is old. (You can't tell how old your package is; there's no way to tell how long it sat on a dealer's or distributor's shelf before you bought it.)
Another possible problem: water. The instructions suggest using a little water when you're smoothing the surface. That's a good idea, but if you overdo it the Milliput may (a) take forever to dry, and (b) chip off easily when it does.
I only have occasion to use Milliput once or twice a year. When I do, I start by mixing up a blob and see if it's dry in four hours. If not, I throw the package out and buy another one.
All that said, I really like the stuff. I have a sailing ship model with Milliput "carvings" that I put on it in 1979, and they haven't deteriorated at all.