What thinning ratio did you use, and how much retarder did you use?
It sounds like maybe not enough solvent, and too much retarder. Retarders are strange critturs in that they slow down polymerization for a time—after which they accelerate it.
OR—did you add the retarder to the distilled water and mix that thoroughly before adding it to the paint? Adding retarder directly to the paint can, and usually does, create severe problems.
Also, always strain the paint before it goes into the airbrush cup. I prefer a 100 mesh screen, but a fine tea strainer will do. Some people use old nylons, but they are rather coarse mesh for hobby paints, especially if stretched.
As far as masking and airbrush technique, there is a "simple" (Ha!) answer: practice, practice, practice, experiment, practice, practice, practice, experiment, practice…
But no doubt you get my drift.