Before tackling this question - and at the risk of sounding preachy - I really feel obligated to make point. I venture to think that virtually every experienced ship modeler would agree: a seventeenth-century ship-of-the-line is NOT a good project for learning how to rig a ship model.
In the first place, such a ship involves a tremendous amount of repetition. One of the trickiest parts of rigging is setting up deadeyes and lanyards. A two-masted schooner has two or three pairs of shrouds on each mast - a total of perhaps two dozen deadeyes and a dozen lanyards. A seventeenth-century ship-of-the-line has about ten pairs of shrouds each on the fore and main lower masts, six pairs on the mizzen lower mast, five each on the fore and main topmasts, four on the mizzen topmast, and three on the spritsail topmast. Throw in the various backstays, bowsprit shrouds, and a few others, and you're talking about over two hundred deadeyes and a hundred lanyards. A two-masted schooner's ratlines may require the tying of between a hundred and two hundred knots; a ship-of-the-line requires well over a thousand. (Though I've never gone through the depressing experience of counting them.) Seventeenth-century rigging was also characterized by what a more modern sailor would call unnecessarily complicated leads of lines. Such pieces of rigging as crowsfeet, bowlines, and the leads of braces were more complex than they were in later centuries. Setting up such gear with just the right amount of tension, so as to keep the lines realistically taut (and the spars in alignment) is quite a project.
The first suggestion I always make to people breaking into the hobby is: get your feet wet with a small ship - preferably on a fairly large scale. Small doesn't mean ugly, or insignificant. Start with a schooner or a sloop, as a means of - literally - learning the ropes. In a month or two you'll have a fine looking model on the mantle, and the time you've invested will pay off many times over on later, more elaborate projects. Elsewhere in this Forum a skilled but not-very-experienced modeler is posting in-progress photos of his fishing schooner Gertrude L. Thebaud. She's a beautiful, historically-important ship, the model shows every sign of being a fine one - and, a couple of months after the first photos were posted, it's almost finished.
There are several good books on the subject - including a couple fairly recent ones that I've missed. (The Naval Institute Press has one titled Rigging Period Ship Models, the author of which I fear I've forgotten, that might be just the thing, but I haven't seen it.) Dr. Anderson's The Rigging of Ships In the Days of the Spritsail Topmast was written for modelers, but it's about 75 years old. In terms of information about rigging it's the best source available, but the techniques it describes are seriously out of date. (Incidentally, if you do go looking for that book, make sure you get the title right. Anderson did a revised version, titled simply Seventeenth-Century Rigging, a few years later, which concentrates entirely on English ships. The earlier work discusses French practice as well.)
I generally recommend three books to people breaking into the hobby. The first, and cheapest, is The Neophyte Ship Modeler's Jackstay, by George Campbell. It's an old paperback published back in the 1950s by Model Shipways, and still sold by Model Expo for about $15.00. It was intended as a guide for building the company's solid-hull wood kits, so some of the techniques described in it aren't relevant to plastic models. But it contains a vast amount of information about hulls, decks, equipment, and rigging, all presented in non-intimidating fashion and accompanied by Mr. Campbell's clear, attractive drawings. Anybody who learned everything in that little book would be well on the way toward being a knowledgeable ship modeler.
Just a few years ago Model Expo published another book for newcomers: How To Build First-Rate Ship Models From Kits, by Ben Lankford. This one also concentrates on wood kits, but much of the material in it is just as applicable to plastic ones. (When it comes to the rigging, there really isn't much difference.) In terms of such things as materials, adhesives, and techniques, Lankford obviously is much more up-to-date than Campbell.
A good, general text on ship modeling is Historic Ship Models, by Wolfram zu Mondfelt. Lots of modelers make it their first book acquisition. It's a reliable, comprehensive overview of the hobby - though Mondfelt, like many other authors, barely acknowledges the existence of plastic kits. For purposes of projects like this one it has a couple of other weaknesses. One - its coverage is so generalized that it doesn't really offer much practical, detailed advice on any one aspect of the hobby. Two - it's heavily continental European in its approach. Some of the terminology, and the references to tools and materials, can be a little confusing to British and American readers. (The "walnut" over which Mondfelt enthuses for so many purposes is, I think, "French" or "European" walnut, which is quite different from the walnut one finds at American lumber dealers.)
All three of those books are available through Model Expo - and various other places as well. (I imagine used copies can be found pretty cheaply on the web.) Somewhere or other there's a website containing extensive excerpts from Mondfelt's book, including many of the sections on rigging. I don't have that site written down, but I'll bet some other Forum member knows it.
Hope that helps a little. Good luck.