The September 2014 Journal of American History has an Interchange on the History of Capitalism. In the Interchange Scott Marler states that
“The problem arises when historians assert that the slave South was “a flexible, highly developed form of capitalism” (as Robert Fogel does). The evidence for such characterizations is thin and usually hinges on questionable interpretations. For example, some will emphasize the careful attention given to profit among that minority of big planter–slave owners, despite the facts that the majority of slaves were held on small units, using roughly five or fewer slaves, and that three-fourths of white households held no slaves on the eve of the Civil War. This is why definitions of capitalism matter. The relationship between master and slave was, at bottom, a nonmarket relationship, redolent of precapitalist relations between lords and serfs—not an economic one, as with the qualitative changes apparent in fast-growing wage-labor societies elsewhere.”
I am not going to get into the issue of whether slavery in the United States was capitalist or not, but Marler bases his conclusion on “the facts that the majority of slaves were held on small units, using roughly five or fewer slaves, and that three-fourths of white households held no slaves on the eve of the Civil War.” All of the sources I know of do indicate that the vast majority of southern families did not own slaves. On the other hand, Gavin Wright estimated that the majority of slaves (nearly 80 percent in the Cotton South in 1860) lived on plantations with 16 or more slaves. Marler cites Kolchin’s American Slavery as a reliable source on the demographics of southern slavery. Kolchin (Appendix Table 4) claims that about 70 percent of slaves lived on farms with 10 or more slaves in the South as a whole; the figure was 80 percent for the Deep South. The majority (about 75 percent) lived on plantations with less than 50 slaves. Overall, the estimates in Wright and Kolchin are pretty consistent.
The availalbe evidence does suggest that the majority of slaveholders had five or fewer slaves, but that is not the same as saying that the majority of slaves lived on farms with five or fewer slaves. In other words, the typical southern farm owner would have looked around his farm and seen few if any slaves. The typical slave, on the other hand, would have looked around the farm he worked on and seen more than a dozen other slaves.