Few meta-analyses of randomised trials assess the quality of the studies included.
Yet there is increasing evidence that trial quality can affect estimates of intervention
efficacy. We investigated whether different methods of quality assessment provide
different estimates of intervention efficacy evaluated in randomised controlled trials
We randomly selected 11 meta-analyses that involved 127 RCTs on the efficacy of interventions
used for circulatory and digestive diseases, mental health, and pregnancy and childbirth.
We replicated all the meta-analyses using published data from the primary studies.
The quality of reporting of all 127 clinical trials was assessed by means of component
and scale approaches. To explore the effects of quality on the quantitative results,
we examined the effects of different methods of incorporating quality scores (sensitivity
analysis and quality weights) on the results of the meta-analyses.
The quality of trials was low. Masked assessments provided significantly higher scores
than unmasked assessments (mean 2.74 [SD 1.10] vs 2.55 [1.20]). Low-quality trials
(score < or = 2), compared with high-quality trials (score > 2), were associated with
an increased estimate of benefit of 34% (ratio of odds ratios [ROR] 0.66 [95% CI 0.52-0.83]).
Trials that used inadequate allocation concealment, compared with those that used
adequate methods, were also associated with an increased estimate of benefit (37%;
ROR=0.63 [0.45-0.88]). The average treatment benefit was 39% (odds ratio [OR] 0.61
[0.57-0.65]) for all trials, 52% (OR 0.48 [0.43-0.54]) for low-quality trials, and
29% (OR 0.71 [0.65-0.77]) for high-quality trials. Use of all the trial scores as
quality weights reduced the effects to 35% (OR 0.65 [0.59-0.71]) and resulted in the
least statistical heterogeneity.
Studies of low methodological quality in which the estimate of quality is incorporated
into the meta-analyses can alter the interpretation of the benefit of intervention,
whether a scale or component approach is used in the assessment of trial quality.