A number of published studies compare adverse event rates for drugs on the basis of reports in the US FDA Adverse Event Reporting System (AERS). While the AERS data have the advantage of timely availability and a large capture population, the database is subject to many significant biases, and lacks complete patient information that would allow for correction of those biases. The accuracy of comparative AERS-based data mining has been questioned, but has not been systematically studied. To determine whether AERS could be used as a data source to accurately compare the adverse event rates for pairs of drugs, using pre-defined, stringent criteria to dictate whether a given pair of drugs was considered eligible for such a comparison. The Fisher's Exact test was utilized to detect differences in adverse event rates between such pairs of drugs. Concordance was determined between statistically significant AERS-based adverse event rate differences, and adverse event rate differences published in the literature from clinical trials and case-control studies. The conditions for validity included (i) data that are free of 'extreme duplication' in AERS reports; (ii) drugs used in similar patient populations; (iii) drugs used for similar indications; (iv) drugs used with the same spectrum of concomitant medications; and (v) drugs not widely disparate in time on the market. For 19 drugs studied, a total of 36 evaluable adverse event rate comparisons were identified. Comparisons were classified as favouring 'drug A', favouring 'drug B' or detecting no difference. Concordance for the resulting 3x3 table (AERS vs literature) gave a kappa statistic of 0.654, indicating moderately good agreement. In only two cases was there absolute discordance, with AERS designating one drug as having a lower rate, while the published study designated the other drug as having a lower rate, with respect to a given adverse event. This pilot study encourages further research regarding the use of spontaneous report databases such as AERS, under stringently defined conditions, to compare adverse event rates for drugs. While not hypothesis proving, such estimates can be used for purposes such as generating hypotheses for controlled studies, and for designing those studies.