Huffman was born in Bedford, New York, into a wealthy family, the daughter of Grace Valle (née Ewing; 1921–2009) and Moore Peters Huffman (1910–1987), a banker and partner at Morgan Stanley. Her parents divorced a year after her birth, and she was raised by both of them. When she was a young teenager, she discovered that her biological father was Roger Tallman Maher, who was a family friend. She has six sisters, Mariah, Betsy, Grace, Isabel, Jessie, Jane (1949–2013), and a brother, Moore Jr. In the 1970s her mother left New York and bought property in Snowmass, Colorado, where Felicity and her siblings spent their youth. Her great-grandfather was Gershom Moore Peters (1843–1919), founder of the Peters Cartridge Company and prominent Baptist minister, author of The Master. Another great-grandfather, Frederick Berthold Ewing, graduated from Yale University and became a prominent St. Louis businessman. Her great-great-grandfather was Joseph Warren King (1814–1885), founder of the King Mills Powder Company. She has German, English, Scots-Irish, Scottish, French-Canadian, and Irish ancestry.