Taylor was born in 1856 to a Quaker family in Germantown, Philadelphia, Pennsylvania. Taylor's father, Franklin Taylor, a Princeton-educated lawyer, built his wealth on mortgages. Taylor's mother, Emily Annette Taylor (née Winslow), was an ardent abolitionist and a coworker with Lucretia Mott. His father's ancestor, Samuel Taylor, settled in Burlington, New Jersey, in 1677. His mother's ancestor, Edward Winslow, was one of the fifteen original Mayflower Pilgrims who brought servants or children, and one of eight who had the honorable distinction of Mister. Winslow served for many years as the Governor of the Plymouth colony.