buff (n.1) Look up buff at Dictionary.com
kind of thick, soft leather, 1570s, buffe leather "leather made of buffalo hide," from Middle French buffle "buffalo" (15c., via Italian, from Latin bufalus; see buffalo (n.)).

The color term "light brownish-yellow" (by 1788) comes from the hue of buff leather. Association of "hide" and "skin" led c. 1600 to the sense in in the buff "naked." Buff-colored uniforms of New York City volunteer firefighters since 1820s led to the meaning "enthusiast" (1903).
The Buffs are men and boys whose love of fires, fire-fighting and firemen is a predominant characteristic. [N.Y. "Sun," Feb. 4, 1903, quoted in OED]
buff (adj.) Look up buff at Dictionary.com
1690s, "of the nature of buff leather;" 1762, "of the color of buff leather;" see buff (n.1). Meaning "well-built, hunky" (of physically fit persons) is from 1980s, from buff (v.) "to polish, make attractive."
buff (n.2) Look up buff at Dictionary.com
"a blow, a slap," early 15c., probably from buffet (n.2).
buff (v.) Look up buff at Dictionary.com
"to polish, make attractive," 1849, from buff (n.1), either in reference to the treatment of buff leather or to the use of buff cloth to polish metals, etc., with a buff-wheel or a buff-stick (1881). Related: Buffed; buffing.