Luis is right. You are missing some steps in your sanding technique.
You should start out with a rough grit but you have to work your way
down to a finer grit. I usually use 3 or 4 different kinds of sanding
grits.
It sounds like you are using files. Try sandpaper or sanding sticks.
You should also check out your local bueaty shop or grocery store.
Women use these sanding sticks for thier nails. They come in all
different grits, and the concept is the same.