This is a tough one, for a couple of reasons. In the first place, the usage of such terms has changed quite a bit over time. In the second place, even at a given moment in history they weren't by any means used consistently.
For what it's worth, here are some definitions from the glossary of The Line of Battle: The Sailing Warship, 1650-1840, a volume in the excellent Conway's History of the Ship series. This is a modern, well-researched volume - as good a place to start as any.
"barge...A long, light pulling boat, usually carvel-built, often associated with pomp and ceremony, as with the City of London livery company boats; a warship's barge was normally the transport of a senior officer or admiral.
"cutter....Ship's boat, usually clinker-built (qv), seaworthy and handier under sail than oars.
"dinghy. Originally an Indian local craft but the name was adopted for a warship's smallest boat.
"gig. A light, narrow ship's boat, better under oars than sail.
"jollyboat. Often the smallest ship's boat, the all-purpose hack for light duties. The term was common until the end of the seventeenth century but thereafter diedout in naval usage until a century later, when it was often applied to the smallest cutter.
"launch. A large relatively flat-sheered boat, initially employed in dockyard duties, but taken to sea from the 1780s as a substitute for the more seaworthy but less capacious longboat (qv).
"longboat. The largest ship's boat, used for heavy duty carrying; seaworthy and strongly built but often complained of as too difficult to stow. Replaced by the launch (qv) from the 1780s.
"pinnace. In the sixteenth century a small fast vessel, often used as a scout, that could usually be rowed as well as sailed but was decked like a ship. This usage survived into the mid-seventeenth century, in parallel with its application to a ship's boat; in the latter sense, a pinnace was a fairly seaworthy boat of up to 35 feet proportioned like a longboat, but by the eighteenth century it had become a narrower, lighter craft and in many respects a smaller version of a barge (qv), for the use of junior officers."
"Whaler" and "galley" aren't in that particular glossary. I'm accustomed to seeing "whaler" used to describe a ship used for whaling; such a ship carries "whaleboats." The U.S. Navy used the term "motor whaleboat" for a long time (and maybe still does) to describe a medium-sized, double-ended boat, with either a wood or metal hull, that was standard equipment on board most warship types during WWII. With the demise of the sailing whaler/whaleship, the term "whaler" seems to have turned into a synonym for "whaleboat."
I'm accustomed to seeing the word "galley" used to refer to a sizeable ship (propelled at least partially by oars). The smallest galleys I can recall encountering are the ones built by Benedict Arnold's force for use on Lake Champlain during the American Revolution. (One of them the Philadelphia, is preserved in the Smithsonian. She's also referred to as a gundalow (which term isn't in the Conway glossary).
That's about how things stood during the great age of the sailing warship - in the English language. A twentieth-century glossary undoubtedly would define some of those words differently. And I suspect if you asked several eighteenth-century British naval officers to label a particular boat, they might well give you a couple of different answers.