An addtitional step I learned when painting copper and aluminum is to rub your PE frets with steel wool, then wipe it with vinegar and then the laquer thinner. Wash well and then wipe down with alcohol to pull out any left over moisture. Don't worry about any scratches because the primer will fill them.  

Morrison's lacquer thinner suggestion is good for helping to remove the skin oils from fondling the PE.   IPA (not the beer kind) might work some but you may be better served with denatured alcohol from the hardware store.   An overnight soak (or longer) in vinegar will help etch the brass surface to add 'tooth'.

Wash the PE well with dish soap.  Rinse well and allow to air dry.   Etch with vinegar if desired, again allow to air dry, then prime with a good primer. Tamiya is good.  There are some good products for fantasy figures. I will use auto primers sparingl.  Applied heavily they tend to clog and obscure PE details.  

Lacquer thinner.

What do you guys clean it with? I've tried apple cider vinegar and 91ipa so far. 

Exactly what Don said, a good cleaning just to make sure theres no contaminates left on the surface, then use a self etching primer.  Part of the reason if you just paint PE with regular paint and tape will easily pull it off is because PE's surface is so smooth, the paint has nothing to really bond with.

common problem!

Most metals do nnot take paint well.  you must clean the metal very well before applying primer, and use an etching primer like an auto body primer.