Hi Bryan:
My first word of advice would be to enjoy the building/painting process. You're the main audience for the finished product.
Now if you want to be moderately accurate, then you'll have to modify your painting and marking choices. All US M3 Lees in N Africa were painted OD. Some had mud smeared on them to give them a partial earth tone/partial OD. Otherwise, the photographic evidence is that they were dusty OD.
The British had the M3 Grant on hand in N Africa. A minute few M3 Lees were sent in the early batches, I recall. These would have been repainted completely desert camo schemes. I don't have this info handy. This would require some more research.
Again, take this advice is you want to be historically accurate. If you're just building for fun's sake, go ahead w/the Afrika Mustard (which is a German AFV color BTW).