Brush a thin wash of water/PVA (wood glue) onto the plaster and let it set. Thinning it lets the water/glue mix penetrate the plaster surface. When it's dry, fix the plaser part to the diorama with neat PVA - it sticks very, very well onto the thin wash which was applied earlier.
Hope this helps.