Sex differences in adipose tissue distribution and the metabolic, endocrine, and immune functions of different anatomical fat depots have been described, but they are incompletely documented in the literature. It is becoming increasingly clear that adipose depots serve distinct functions in males and females and have specific physiological roles. However, the mechanisms that regulate the size and function of specific adipose tissues in men and women remain poorly understood. New insights from mouse models have advanced our understanding of depot differences in adipose growth and remodeling via the proliferation and differentiation of adipose progenitors that can expand adipocyte number in the tissue or simply replace dysfunctional older and larger adipocytes. A limited ability of a depot to expand or remodel can lead to excessive adipocyte hypertrophy, which is often correlated with metabolic dysfunction. However, the relationship of adipocyte size and function varies by depot and sex. For example, femoral adipose tissues of premenopausal women appear to have a greater capacity for adipose expansion via hyperplasia and hypertrophy; although larger, these gluteal-femoral adipocytes remain insulin sensitive. The microenvironment of specific depots, including the composition of the extracellular matrix and cellular composition, as well as cell-autonomous genetic differences, influences sex-and depot-dependent metabolic and growth properties. Although there are some species differences, studies of the molecular and physiological determinants of sex differences in adipocyte growth and function in humans and rodents are both needed for understanding sex differences in health and disease.