Raman spectroscopy was explored to diagnose normal, benign, and malignant human breast tissues based on principal component analysis (PCA) and support vector machine-recursive feature elimination (SVM-RFE), and SVM-RFE results were compared with PCA. 1800 Raman spectra were acquired from fresh samples of human breast tissues (normal, fibroadenoma, adenosis, galactoma, and invasive ductal carcinoma) from 168 patients. After set up the SVM-RFE and PCA models, Mahalanobis distance, spectral residuals, sensitivity, specificity, and Matthews correlation coefficient (MCC) were used as the discriminating criteria for evaluating these two methods. The comparison shows that SVM-RFE based on the selection of characteristic peaks better reflects the nature of biopsy and it produces better discrimination, sensitivity, specificity, and MCC for normal (1, 1, 1), malignant (0.93, 0.97, 0.91), and benign (0.95, 0.97, 0.91) breast tissues. The Raman spectroscopy combined with SVM-RFE opens great future in the clinical applications of mammary disease diagnosis.