Cluster randomized trials with a small number of clusters: which analyses should be used?

In this paper, we propose an alternative covariance estimator to the robust covariance estimator of generalized estimating equations (GEE). Hypothesis tests using the robust covariance estimator can have inflated size when the number of independent clusters is small. Resampling methods, such as the jackknife and bootstrap, have been suggested for covariance estimation when the number of clusters is small. A drawback of the resampling methods when the response is binary is that the methods can break down when the number of subjects is small due to zero or near-zero cell counts caused by resampling. We propose a bias-corrected covariance estimator that avoids this problem. In a small simulation study, we compare the bias-corrected covariance estimator to the robust and jackknife covariance estimators for binary responses for situations involving 10-40 subjects with equal and unequal cluster sizes of 16-64 observations. The bias-corrected covariance estimator gave tests with sizes close to the nominal level even when the number of subjects was 10 and cluster sizes were unequal, whereas the robust and jackknife covariance estimators gave tests with sizes that could be 2-3 times the nominal level. The methods are illustrated using data from a randomized clinical trial on treatment for bone loss in subjects with periodontal disease.

Evidence suggests that cluster randomized trials are often poorly designed and analysed. There is little recent research on the methodologic quality of cluster randomized trials and none focuses on primary health care where these trials are increasingly common. We conducted a systematic review of recent cluster randomized trials in primary health care, searching the Cochrane Controlled Trials Register. We also searched for unpublished trials in conference proceedings, and the UK National Research Register. We assess methodologic quality using a checklist, articulate problems facing investigators conducting these trials, and examine the extent to which carrying out a cluster randomized trial (as opposed to an individually randomized trial) in primary care may reduce power. We found 367 trial reports. Many trials were reported more than once. We characterize 152 independent cluster randomized trials in primary health care published between 1997 and 2000, and briefly describe 47 trials unpublished at December 2000. The quality of design and analysis was variable. Of published trials reporting sample size calculations 20% accounted for clustering in these calculations, 59% of published trials accounted for clustering in analyses. Unpublished trials were more recent and of higher quality. Reporting quality was better in journals reporting more cluster randomized trials. Many trial investigators reported problems with adherence to protocol, recruitment and type of intervention. Methodologic quality of cluster randomized trials in primary health care is variable and reporting needs improvement. The use of cluster randomization should be indicated in the title or abstract so these kinds of trials are easier to identify. Communicating appropriate methodology to health care researchers continues to be a challenge. Cluster randomized trials should always be piloted and information from pilots and unsuccessful trials shared more widely.

The sandwich estimator of variance may be used to create robust Wald-type tests from estimating equations that are sums of K independent or approximately independent terms. For example, for repeated measures data on K individuals, each term relates to a different individual. These tests applied to a parameter may have greater than nominal size if K is small, or more generally if the parameter to be tested is essentially estimated from a small number of terms in the estimating equation. We offer some practical modifications to these robust Wald-type tests, which asymptotically approach the usual robust Wald-type tests. We show that one of these modifications provides exact coverage for a simple case and examine by simulation the modifications applied to the generalized estimating equations of Liang and Zeger (1986), conditional logistic regression, and the Cox proportional hazard model.