Acrylic paints are the fastest drying of all the paint types out there, so perhaps the paint is drying and being reworked and forming clumps. What you might have to do is pick up the pace and really think about what you are doing, and keep the brush strokes down to a minimum. Keeping your brush clean also helps so if you are drybrushing a lot of the same colour you might have to use a 2nd brush or clean the the existing one several times through the process. Also be aware of where you are wiping off the excess paint - if its on paper or some kind of kleenex tissue, you could be picking up fibers.
