The actual origin of the term is pretty hard to get your fist around - no pun intended. Webster's doesn't even give an original language or colloquialism. I've an idea though if you will indulge me.
Beginning in the Middle Ages, seafarers from Brittany in France fished out of small, two masted and square rigged vessels that were called "bugalets" by the English. Doubtless this is a corruption of a French term now lost to the ages (chime in on the comments if you know!) but by the 17th century even the French called such a boat a bugalet.
The name for the little boat was shortened to "bug" by the Elizabethan era. This held a derogatory connotation that could not be missed so it began to be used more broadly to indicate a vessel that was bigger than its capability. In other words a bug was a ship that looked impressive but was only marginal in maneuverability and speed.
In particular the derision was cast onto the Spanish galleon. These were originally caravels, like the painting shown above, that grew larger and more rounded in design until they became the Spanish Armada kind of galleon shown at the header of this post. They were handsome, could carry a grand compliment of men, cannon and treasure, but they weren't going anywhere fast and turning on a dime was right out. When Sir Francis Drake first encountered this type of ship at Cadiz he called them "mere great bugges". Nothing to worry about, in other words.
Of course, we all know that the enemy - whoever they are - is more prone to vile and unthinkable habits than we will ever be. Therefor, it wasn't much of a stretch to imagine that those aboard the bugs were buggers prone to buggery. The idea and the word stuck until, by the 18th century, a man could be flogged or even hanged for such an offense with the term being use specifically in the court documentation.
Is that how we really got the word? I can't say for sure but I will say that it is very likely indeed. Happy Saturday, Brethren. And keep your mouth clean.