Silvering isn't the only issue.
When using a decal setting solution, when it works properly the first thing that starts to happen is that the decal will shrivel and appear to wrinkle up on the model's surface. As if by magic, as it dries, it smooths out again and snuggles down tightly onto the surface.
A gloss surface is smooth and slick and offers little resistance to the decal as it dries.
A flat or matt surface (covered in microscopic pits) offers far more resistance to the decal as it settles, which can prevent the decal from settling down completely, resulting in the decal drying with creases and wrinkles which are impossible to remove or repair.