You don't actually want thick, heavy plywood for the base.
1/4" plywood, masonite (it's a dark brown color and smooth on one side, typically) will suffice. All you need is some rigidity. Especially with foam used as a base.
Yes, the blue (good) or the pink (better) foam sheets at the home center are good stuff. Fine yellow foam is likely to be Isocyanurate, which is really good for roofing, but lousy for dioramas (it does not cut with a hot wire for one).
You might want to invest in a wire cutter from Foam Factory.
Many things at the home center are useful. Grout mix is a very stable "dirt" mixture. Premixed drywall compound and patch material are handy (as is cellutex, a paper-mache product [in Eurpoe sold as Sculptamold] which is more an art store product).
While in the home center, they also stock rolled wood veneer (or ought to). This is a handy thing to apply to finished edges of dioramas. You can get moulding shapes used for door & window trim as well.
Let me recommend Luke Towen and Kathy Millet on YouTube for having ecellent how-to videos for diorama videos. Another good video creator is Laser Creation World--but that's a very German site in terms of materials used (they market a bunch of supplies, themselves, too--which can cause tool envy of great proportion).
Tomas at LCW has a weathering technique that is fine-art based, rather than more modeler-based, and is fascinating to watch.