Some of that tooth as you call it will show through with different paints. If you're doing a flat camo job or a high gloss colored one chances are you'll fill those scuffs with no problem. But if you are doing a thin skinned NMF or an acrylic painted scheme they might stick out like a sore thumb.
If you need tooth...then prime.
I use nail sanding and polishing sticks, rarely do I dig out the papers.