Wallboard compound (drywall compound or spackling mud) makes a nice sacle "plaster" or concrete. Its easy to work, adheres well and doesn't shrink when it dries. (Also very inexpensive) It tools well and can be painted, sanded, and in some instances will make good mud, rubbles, water, etc. depending on how you paint it.
For mud I use white glue, plaster, paint (the color of my mud) and very fine sand and or dust. A stiff paintbrush or a ver controlled "flick" of the thumb on a toothbrush and you'll get great splattering and build up under wheel wells, back of soldiers on bikes, etc. Splatter where needed... Celuclay also makes good "chunky mud" like Russia in the spring... ahhhhhh Russia in the spring, the mud, the flies, the partisans (sigh).
I use foam core like warren and also a product called gator board. Rather than foam center it has an encapsulated cell and it makes it a little stiffer.
I aslo use luan or doorskin which is really just a thin plywood. 1/8"-1/4" thick. The 1/4" works out perfect for wall thicknesses on houses for 1/35 scale. Anything bigger like medieval ruins, large commercial buildings etc, I use a thicker ply of plywood or manufacture my own wall material from a combination of meterials to get the desired thickness.
A good method of making brick walls is to lay your wall flat (prior to assembly) and coat it with wall board compound. Once it starts to dry, lay a straight edge on it horizontaly and run a scribing tool along it to make your mortar lines. I use a rounded tip of different size nails, key stock, plastic swizzle sticks, pieces of scrap wood etc, dependent on the size and texture of the mortar line you want and frankly, whatever works. Once I have my courses scrobed in, I then go back and make the edes of the bricks. This way you can off set them, line them up, make patterns, etc. Once dry, you can paint and assemble your walls. Then you can touch up seams with more wall compound or make decoractive edging to present decorative brickwork like you see on the edges of urban buildings, to go over and disguise your seams.
Kitty litter works good for rubble and I've also taken plaster and poured it into a tray about 1/4" maybe 1/8" thick. Again, as this sets up you score it like you would the wall compound and make some deeper lines this time in the proportion of your bricks. Then place the whole thing in a bag, go outside and drop it in your driveway a few times. Instant bag-o-rubble that you can paint and add. Not only do you get single bricks but wall sections as well. Sure its a little work up front, but go out and price some fo the rubble that is commercial available.... Time to modelers is cheap. If you separate out your single bricks and put them in a bag and shake them up you will get some nice worn bricks to go along with your crisper edged bricks. Nice project between builds to fill out the down time.
Cassi's suggestion for kitty litter is A#1. Been using it for years and have always liked it for ground material and rubble. In a pinch I'll even grind it up to get some fine dust. Afterall, the cheaper kitty litter is really only clay.
good luck
Mike