Microscale makes a product called Micro Kristal Klear. This is specifically for making small windows. It looks like white glue but dries perfectly clear and thin. Just apply a drop or two accross the openning with a toothpick and leave it overnight. You will be amazed how good it looks the next morning.
A perfect fit, every time! Much better than trying to make a window from clear plastic sheet.
You will find Micro Kristal Klear in most good hobby shops in the same location where they have the Micro Set, Micro Sol, Micro Mask, etc.
Pete