In fact, the use of "flags" as true identifiers of nationality was virtually unthinkable until the Napoleonic era. Flags, which were almost always referred to as "colors" at sea, were constantly changing prior to roughly 1780. The colors that we might recognize - the tricolour of France shown above, for instance - were unknown in the 18th century and unheard of prior to that. As an example, French ships were generally flying the pavillon blanc which was nothing more than a white flag perhaps kissed with an embroidered fleur-de-lis in one corner. Even close by it would be nearly impossible for an untrained eye to tell whether the ship was a French merchant, French man-of-war, or a freebooter using a ruse.
Another argument against routinely showing one's colors was expense and handling. Flags were made of hand loomed wool originally and in heavy weather or high winds they would unravel alarmingly. The cost of flying one's flag regularly was so prohibitive that even war ships were restricted to flying their colors when engaging an enemy or at anchor in a neutral port. Then, too, these flags were necessarily large - sometimes the size of the main and course sails on a mizzen mast - so they added weight, befuddled rigging and robbed a ship of speed and maneuverability.
Before the 19th century, ship's flags were called "ancients" (from which the English word "ensign" for an identifying flag is descendant) and - in theory at least - ships were to fly the ancient of their nation, home port or merchant company. Even in honest circumstances, however, this could turn into a confusing quagmire fairly quickly. Just as an example, here are some pendants and ancients from the Dutch merchant service of the 18th century (courtesy of the Heilongjiang Archives):
All of these colors would have been perfectly legal for the appropriate Dutch merchant to fly, but trying to discern the subtle differences at any reasonable distance would have been maddening. When one considers that the spy glass was rarely taken into the tops due to it's extreme cost, the impossibilities are clear. Add the fact that a pirate carried several flags aboard her to run up as ruses to catch prey and you can throw colors over the taffrail as far as identifying another ship. Your best bet was to know what types of ships came from what countries and watch the handling of the ship. Merchant seamen were - due to routinely small crews - unfortunately lubberly in the handling of their ships.
As another example, here is the evolution of the British Union Jack:
Some or all of these flags would have been flown at the same time and the designs were never in the main body of the flag itself. The field of the flag would have been red or blue while the canton (where the stars are on the U.S. flag, for instance) would have held the precious identifying symbol. Honestly, gentlemen, why bother?
Pirate flags as separate entities did not appear until the early 18th century. Most began simply as the red flag, pavillon rouge or sometimes jolie rouge which was understood as a sign that no quarter was expected and none would be given. Fancy flags, some black (originally a signal of plague aboard the ship) and some still red, with skulls and swords and the like began to develop during the Golden Age of Piracy and we've talked about them before. Even so, many pirates chose to sail under the legal ancient of a country or countries. Bartholomew Roberts flew a Dutch flag routinely and Michele de Grammont only ran up the French pavillon blanc if he bothered with an ancient at all.
As the age of nationalism dawned in the early 19th century, it became more common for ships to fly the flag of their country. Improvements in textiles made it possible to unfurl your ancient more frequently and the new countries in the Americas - born of hard fought revolutions and proud of their accomplishments - embraced and honored their colors as symbols of freedom. Francis Scott Keye's poem about seeing the U.S.'s Star Spangled Banner from the British prison hulk where he was being kept during the bombardment of Washington D.C. is just one example of the feeling a country's colors could inspire in it's people.
This new nationalism changed the way that the ruse was used in naval engagements and by the 1850s it was no longer considered honorable to sail under false colors. Such things didn't bother the pirates and/or privateers of the era, however. The Laffite brothers and their associates, for instance, happily sailed under whatever colors would buy them a hint of legitimacy. During the Galveston days they raised the ancient of the Congress of Mexico - a red and green checkered flag. The Congress, unfortunately, existed for only a few months and never actually issued a flag but why worry about details, mon frere?
My advice to you, mate: keep a young man with good eyes in the tops and trust no one out there. To paraphrase Nelson: all are foes until proven friends. If it's good enough for Britain's God of War, it's good enough for me.