When I owned an airplane and had it painted by a pro shop, I was able to observe most of the process. The paint used was Imron, a polyurethane, which appeared to be a rather thin viscosity, it flowed out to set up as extremely glossy and smooth.
I noticed they kept touching the last coat, in areas where it was on the masking. In just several minutes, they felt that the paint had set up enough to pull the masking tape.
So, like you observed, if the paint is allowed to set up too much it becomes a bit brittle, and pulling the mask away does result in chipping and flakes. They explained that when the paint on the mask feels just about to turn tacky and not stick to your finger, it has set up enough to not run away from the mask edge, but will allow the paint to remain.
Hope it helps.
Patrick