I don't know if this is what's happened in your case, but it may not be residue from the masking tape.
Sometimes, especially when masking over Tamiya or Gunze gloss acrylics (using any type of masking tape) even when the base layer being masked over has been apparently dry for a week, you can get a reaction from the tape adhesive actually softening the underlying paint, and leaving a "textured" effect in the paint surface. This can be worse if the tape is left on for a longer period. eg. several days to a week or more. I find the best way to minimise this is to paint and then remove the masking as soon as practically possible.