Gen. Lee had a shall we say "laz-ze-fair" attitude toward the uniform. In fact he was rarely seen other than the colonel's uniform and did not wear the Austrian Knots ("chicken guts") on his sleeves. While he did wear the dark blue trousers ascribed for staff by the 1861 Regs, he also did not ascribe to the "buff" colored collar and cuffs also ascribed by regulation. There is a practical side to this, as union sharpshooters tended look for officers first.
Also this probably had more to do with the Confederacie's problems fielding any kind of a standard uniform to the troops during the early part of the war.
At one point General Lee was asked your exact question, and his reply was something to the effect of "It's the only rank I feel I am entitled to wear." In reference to his rank in the US Army.
Regards,
Chuck
Able Audacious Army Aviation Above All!