The subject is indeed a little complicated, but if you're dealing with an old sailing ship you're fairly safe in assuming that the flag will be blowing in more-or-less the same direction as the wind. If the wind is blowing from behind, the flag will blow forward. If the wind is bloing from starboard, the flag will blow out to port. Etc. The laws of physics will make the precise angle of the flag vary somewhat, but if somebody questions how you've arranged a flag on a model you can always explain that it's flapping and you stopped it in mid-flap.
All this assumes, of course, that the ship doesn't have an engine. (If it does, all bets are off; its progress isn't directly related to the speed or direction of the wind.) Modern yacht designers, geniuses that they are, have figured out how to make a sailboat move faster than the wind under some circumstances, but that certainly didn't happen prior to the twentieth century.
The shaping of the flags can have quite a significant effect on the overall appearance of a model. If you want to show flags blowing in the breeze, it's worth spending a little time studying what they actually look like. Pick a sunny, breezy day, find a flagpole with a good-sized flag on it (in front of a school, post office, or car dealership maybe), set your camera (film or digital) to a high shutter speed, and take twenty or thirty pictures. (I got some weird looks once when I spent half an hour docmenting the movements of the flag in front of the local library.) You may be surprised at some of the shapes that flag assumes.
On a vaguely related subject, I've sometimes been puzzled by the "flag sheets" that come in plastic ship kits. In many cases some anonymous but highly skilled artist has taken a great deal of time drawing pictures, in perspective, of flags with ripples and creases in them. If a flag has an elaborate design, that must be quite a project. (The flags in the old Revell Santa Maria come to mind.) The modeler is supposed to cut the picture out, fold it in half, stick it to itself, and apply it to the model. The result is a perspective drawing of a flag with ripples and creases in it attached to the model, and looking utterly ridiculous. Why in the name of heaven would it ever occur to anybody to do such a thing? It surely would be easier for that artist to draw the flag as a flattened-out rectangle, and anybody with sufficient physical dexterity to dress himself can put genuine, three-dimensional ripples in a flag in a matter of seconds. Ah, the strange foibles of human nature.
Youth, talent, hard work, and enthusiasm are no match for old age and treachery.