This is a controversial subject among ship modelers. You'll get different - and quite emphatic - answers from different people.
I think there's a general consensus among people who've been at it for a while that making truly set sails (i.e., sails with wind blowing in them) is extremely difficult if not impossible unless the scale is either very large (so every individual piece of the sail can be represented accurately) or very small (so nobody will notice that it isn't). The most successful attempts at set sails that I've seen have been on small-scale waterline models set in diorama-type bases. (Maybe this is a personal idiosyncrasy, but I've got a problem with ships on stands in glass cases with billowing sails. How did the wind get into that case - and why isn't the ship moving?)
If you can, get hold of a copy of Donald McNarry's book,
Ship Models in Miniature. He's better at this particular aspect of modeling than anybody else I know of. He never works to a scale larger than 1/16"=1', andmakes his sails out of tissue paper. On small scales I think that's probably the best idea - either tissue or fine drafting vellum. When the light shines from behind a real sail, the spars and rigging behind it appear in silhouette. If your material duplicates that effect, you're on your way to success. You can represent the seams on the sail with pencil. (A 0.5 mm mechanical pencil works well. Good art supply stores sell packages of brown lead.) If the scale is small enough you can use the pencil to represent the reef points and other pieces of rigging attached to the sail as well.
Maritime museums have lots of examples of good set sails on large scales (say, 1/48 up), generally made of linen. That's quite a project for a mortal model maker, though one who could handle a sewing machine with a very fine stitch setting might be able to pull it off. I've never tried it.
I personally am partial to furled sails on models. To my notion they make a nice compromise; they demonstrate how the rigging works without making the model look like a fish out of water. To my eye a good model with furled sails, and its yards in the lowered positions, projects a look of latent power and action that's far more effective than phony-looking set sails. Over quite a few years I've worked out a technique for making furled sails that, I think, works pretty well.
The basic material is silkspan tissue. (Hobby shops that cater to flying airplane modelers carry it in various weights; you want the finest you can get. For small sails, lens tissue from a camera store works great.) Make up a simple wood frame (balsa will do) that's bigger than the sail in all dimensions and tape a piece of the tissue over it. Then mix up a batch of water-soluble hobby paint (I like PolyScale; I imagine other brands of acrylic would work too) in a very pale grey with a slight hint of beige. Next comes the key to the trick: before you apply the paint to the tissue paper, squirt a considerable amount (the proportion isn't critical; 50/50 will work) of Elmer's Glue into the paint. Next dillute the paint/glue mix with water till it's easy to handle with a brush, and paint the tissue with it. Let it dry completely.
The tissue is thin, but still too thick for the scale. One of the most common mistakes is to make the "bundle" of a furled sail too fat. It ought to be just about as big in diameter as the yard, gaff, or boom to which it's fastened - maybe even a little thinner. The bundle made by a square sail furled to a yard should, generally speaking, be thinner at the ends than in the middle. So mark out on the painted tissue paper a trapezoid shape, its width at the top the same as that of the finished sail, its width at the bottom about 1/4 that dimension, and its depth maybe about half that of the full sail. (That's far harder to describe than it is to draw.) Add onto each side a strip about 1/16" wide, which will form a hem. Cut the sail out.
Fold the hems on all four sides over, trapping a piece of fine thread of the same color as the running rigging. This is the bolt rope, and it's crucial to the success of the operation. (The hem over the boltrope isn't authentic, but nobody will notice.) Leave a small loop sticking out at each corner. Glue the hem with Elmer's, and let the whole thing dry. You now have before you a decidedly weird-looking conglomeration of stiffened tissue paper, paint, glue, and thread. Patience.
Mount the "sail" to the yard, using a needle and thread and making use of the two loops at the top corners. (If you're a stickler for accuracy, do some reading to find out how it was fastened on the prototype. If the ship dates from prior to about 1810, you can sew it to the yard with a single, long thread called a roband going around the yard in a spiral. If the ship is post-1810, the sail will be laced to the jackstay.) The boltrope will keep the roband thread from ripping through the edge of the sail.
Add the sheets, clewlines, buntlines, etc., but leave them slack for the time being. Don't fasten their ends to the belaying pins.
Now comes the real trick. Dip a paint brush in water and touch the sail with it. The Elmer's glue will soften up - but, astonishingly (and I don't have a scientific explanation for this phenomenon) the paint won't. The sail will get soft and rubbery. Take a pair of tweezers and bundle it up against the yard, being careful how you fold it. (Old photos and paintings are a big help for this stage.) The glue and paint have made the material surprisingly tough, but flexible. You'll find that the sail will furl up into a nice, tidy bundle that can then be secured with thread gaskets, just like the real thing. Tie off the various pieces of rigging and you're done.
This takes at least five times as long to read about as to do. The most time-consuming part of the process is waiting for the paint/glue mixture to dry. I've used this system on quite a few models; one of them is twenty-five years old now, and its furled sails look as good as new.
Other modelers will offer different opinions about putting sails on models; this is one person's. Take all of us with a shaker full of salt; we're all nuts. But you might want to give furled sails a try. Good luck.