Prostate cancer is the main causes of cancer associated mortality in men worldwide, cancer patients often suffer serious side effects when treated with chemotherapy or radiotherapy, therefore novel drugs are in high demand to treat prostate cancer. In the present study, a pentapeptide (Ile-Leu-Tyr-Met-Pro; ILYMP) with a molecular weight of 635.71 Da was isolated from the protein hydrolysate of Cyclina sinensis via ultrafiltration and chromatographic methods, and subsequently named Cyclina sinensis pentapeptide (CSP). The activity of CSP was first investigated in prostate cancer (PCa) DU-145 cells. CSP was demonstrated to significantly inhibit DU-145 cell proliferation at a half-maximal inhibitory concentration of 11.25 mM at a 72 h time interval. In addition, the results of acridine orange/ethidium bromide double staining, scanning electron microscopy and flow cytometry analyses suggested that CSP inhibited DU-145 cell proliferation via the induction of apoptosis. Following treatment with CSP, Bcl-2-associated X (Bax), cleaved caspase-3 and cleaved caspase-9 protein expression levels were enhanced in DU-145 cells; whereas B-cell lymphoma 2 expression was suppressed in DU-145 cells. In conclusion, to the best of the authors' knowledge, this is the first study to investigate the effects of an anti-proliferative peptide derived from Cyclina sinensis on DU-145 cells, and the results suggested that CSP may represent a therapeutic nutraceutical agent for the treatment of patients with PCa.