Cross-validation and bootstrapping are unreliable in small sample classification

General studies of microarray gene-expression profiling have been undertaken to predict cancer outcome. Knowledge of this gene-expression profile or molecular signature should improve treatment of patients by allowing treatment to be tailored to the severity of the disease. We reanalysed data from the seven largest published studies that have attempted to predict prognosis of cancer patients on the basis of DNA microarray analysis. The standard strategy is to identify a molecular signature (ie, the subset of genes most differentially expressed in patients with different outcomes) in a training set of patients and to estimate the proportion of misclassifications with this signature on an independent validation set of patients. We expanded this strategy (based on unique training and validation sets) by using multiple random sets, to study the stability of the molecular signature and the proportion of misclassifications. The list of genes identified as predictors of prognosis was highly unstable; molecular signatures strongly depended on the selection of patients in the training sets. For all but one study, the proportion misclassified decreased as the number of patients in the training set increased. Because of inadequate validation, our chosen studies published overoptimistic results compared with those from our own analyses. Five of the seven studies did not classify patients better than chance. The prognostic value of published microarray results in cancer studies should be considered with caution. We advocate the use of validation by repeated random sampling.

Many cases of hereditary breast cancer are due to mutations in either the BRCA1 or the BRCA2 gene. The histopathological changes in these cancers are often characteristic of the mutant gene. We hypothesized that the genes expressed by these two types of tumors are also distinctive, perhaps allowing us to identify cases of hereditary breast cancer on the basis of gene-expression profiles. RNA from samples of primary tumor from seven carriers of the BRCA1 mutation, seven carriers of the BRCA2 mutation, and seven patients with sporadic cases of breast cancer was compared with a microarray of 6512 complementary DNA clones of 5361 genes. Statistical analyses were used to identify a set of genes that could distinguish the BRCA1 genotype from the BRCA2 genotype. Permutation analysis of multivariate classification functions established that the gene-expression profiles of tumors with BRCA1 mutations, tumors with BRCA2 mutations, and sporadic tumors differed significantly from each other. An analysis of variance between the levels of gene expression and the genotype of the samples identified 176 genes that were differentially expressed in tumors with BRCA1 mutations and tumors with BRCA2 mutations. Given the known properties of some of the genes in this panel, our findings indicate that there are functional differences between breast tumors with BRCA1 mutations and those with BRCA2 mutations. Significantly different groups of genes are expressed by breast cancers with BRCA1 mutations and breast cancers with BRCA2 mutations. Our results suggest that a heritable mutation influences the gene-expression profile of the cancer.