In a cross sectional study of 89 infertile women, we explore a relationship between aspects of psycho-social stress and ovarian reserve parameters. Questionnaires assessed general health and mood (profile of mood state) were administered. Serum (cycle days 1-3) was collected for biomarkers of ovarian reserve (follicle stimulating hormone (FSH), Mullerian Inhibitory Substance, Inhibin B) and stress (Cortisol). Multivariable regression analyses evaluated associations between parameters of interest (dysphoric mood, morning serum cortisol levels reflecting current stress; personal history of abuse, family and/or personal history of substance abuse reflecting chronic stress), with ovarian reserve biomarkers and with the likelihood of being diagnosed with diminished ovarian reserve (DOR). Women with DOR were almost four times more likely to acknowledge personal history of recreational substance use (0.023) and family history of early menopause (p = 0.018). Adjusted analyses demonstrated advancing age, family history of early menopause, body mass index and chronic psycho-social stressors as independent correlates to serum FSH levels; age, family history of early menopause and chronic stress were predictive of likelihood for DOR. No demonstrable relationship was observed between ovarian reserve and current stress. Our findings identify aspects reflecting 'chronic' lifetime psycho-social stressors (i.e., personal history of abuse and of recreational drug use and/or family history of drug use) rather than 'current' stress (as reflected by dysphoric mood score and morning serum cortisol level) as detriments to ovarian reserve (i.e., were predictive of higher FSH levels and of an enhanced likelihood for DOR).