Reduced empathy and elevated alexithymia are observed in autism spectrum disorder (ASD), which has been linked to altered asymmetry in brain morphology. Here, we investigated whether trait autism, empathy, and alexithymia in the general population is associated with brain morphological asymmetry. We determined left–right asymmetry indexes for cortical thickness and cortical surface area (CSA) and applied these features to a support-vector regression model that predicted trait autism, empathy, and alexithymia. Results showed that less leftward asymmetry of CSA in the gyrus rectus (a subregion of the orbitofrontal cortex) predicted more difficulties in social functioning, as well as reduced cognitive empathy and elevated trait alexithymia. Meta-analytic decoding of the left gyrus rectus annotated functional items related to social cognition. Furthermore, the link between gyrus rectus asymmetry and social difficulties was accounted by trait alexithymia and cognitive empathy. These results suggest that gyrus rectus asymmetry could be a shared neural correlate among trait alexithymia, cognitive empathy, and social functioning in neurotypical adults. Left–right asymmetry of gyrus rectus influenced social functioning by affecting the cognitive processes of emotions in the self and others. Interventions that increase leftward asymmetry of the gyrus rectus might improve social functioning for individuals with ASD.