Foamcore is a good choice. This is a product witth two cardstock faces between an extruded foam center. It's relatively inexpensive from a crafts store. It is also fairly easy to cut, and join together. It's (usually) thick enough to represent the core of a building's walls.
Fow thick is thick enough? Well, let's break 132 scale down a tad. 1/32 is also 3/8" = 12". From this we can state that 3/16" = 6"; 3/32" = 3"; and thus, 1/32" = 1" to scale. European walls run to about a foot thick on the first floor, or about 4" wide brick against an 8" wide block inner wall.
If a person were using one of the pressed paper brick veneers, you'd want foamcore about 10/32" (5/16") thick, which would allow for a thin coat of spackle on the inner face to represent plaster. If a person wer using a 4/32" (1/8") thick product for brick veneer, then the foamcore could be 8/32" (1/4") thick. Or, a thick coat of spackle could be built up and scribed to represent brick.