If you paint with enamels, there is little (usually) consequence to not washing, as most organic solvents will dissolve and neutralize skin oil. However, acrylics are another story.
Before priming, I use either alcohol (isopropyl or ethyl) or a surfactant cleaner, or Simple Green (which needs to be rinsed with water).
Between coats, just soap and water. I use either Woolite or Dr. Bronner's, as they do not leave residue. Any detergent that does not have "skin conditioners/softeners" will also work, provided that you rinse it off completely.
But even with enamels, cleanliness is next to…just do it!