Children born very preterm (<33 weeks of gestation) are at a higher risk of developing socio-emotional difficulties compared with those born at term. In this longitudinal study, we tested the hypothesis that diffusion characteristics of white matter (WM) tracts implicated in socio-emotional processing assessed in the neonatal period are associated with socio-emotional development in 151 very preterm children previously enrolled into the Evaluation of Preterm Imaging study (EudraCT 2009-011602-42). All children underwent diffusion tensor imaging at term-equivalent age and fractional anisotropy (FA) was quantified in the uncinate fasciculus (UF), inferior fronto-occipital fasciculus (IFOF), inferior longitudinal fasciculus (ILF), and superior longitudinal fasciculus (SLF). Children’s socio-emotional development was evaluated at preschool age (median = 4.63 years). Exploratory factor analysis conducted on the outcome variables revealed a three-factor structure, with latent constructs summarized as: “emotion moderation,” “social function,” and “empathy.” Results of linear regression analyses, adjusting for full-scale IQ and clinical and socio-demographic variables, showed an association between lower FA in the right UF and higher “emotion moderation” scores (β = −0.280; p < 0.001), which was mainly driven by negative affectivity scores (β = −0.281; p = 0.001). Results further showed an association between higher full-scale IQ and better social functioning (β = −0.334, p < 0.001). Girls had higher empathy scores than boys (β = −0.341, p = 0.006). These findings suggest that early alterations of diffusion characteristics of the UF could represent a biological substrate underlying the link between very preterm birth and emotional dysregulation in childhood and beyond.