Sounds like a paint drying problem. If you're using enamels, you have to really let the paints dry thoroughly before handling the finish. This is one of my BIG problems in building a kit. I get impatient and try to rush from one thing to the next (sometimes only a couple of hours after I've applied paint). Mistake, mistake, mistake... always have fingerprints.
Keeping your hands clean is also a good idea though. I've got way too many examples of times where the tiniest bit of silver paint on my hand/finger (that I thought was dry), got transfered/smudged onto an area it had no business being. I have a little bottle of soapy water and a towel near by that I use to rinse my hands frequently. When I'm in the final detailing/polishing stages, I have some cotton gloves that I wear. Works well for me.
Murray