Univariate analyses of structural neuroimaging data have produced heterogeneous results regarding anatomical sex- and gender-related differences. The current study aimed at delineating and cross-validating brain volumetric surrogates of sex and gender by comparing the structural magnetic resonance imaging data of cis- and transgender subjects using multivariate pattern analysis. Gray matter (GM) tissue maps of 29 transgender men, 23 transgender women, 35 cisgender women, and 34 cisgender men were created using voxel-based morphometry and analyzed using support vector classification. Generalizability of the models was estimated using repeated nested cross-validation. For external validation, significant models were applied to hormone-treated transgender subjects ( n = 32) and individuals diagnosed with depression ( n = 27). Sex was identified with a balanced accuracy (BAC) of 82.6% (false discovery rate [p FDR] < 0.001) in cisgender, but only with 67.5% (p FDR = 0.04) in transgender participants indicating differences in the neuroanatomical patterns associated with sex in transgender despite the major effect of sex on GM volume irrespective of the self-identification as a woman or man. Gender identity and gender incongruence could not be reliably identified (all p FDR > 0.05). The neuroanatomical signature of sex in cisgender did not interact with depressive features (BAC = 74.7%) but was affected by hormone therapy when applied in transgender women ( P < 0.001).