A comparison of statistical methods for combining event rates from clinical trials

We compare two statistical methods for combining event rates from several studies. Both methods treat each study as a separate stratum. The Peto‐modified Mantel—Haenszel (Peto) method estimates a combined odds ratio assuming homogeneity across strata and provides a test for heterogeneity. The DerSimonian and Laird modified Cochran method (D&L) produces a weighted average of rate differences, where the weights allow for among‐study variability. We analyse 22 meta‐analyses from ten reports by both methods. The pooled estimates are divided by their standard errors to produce a Z‐statistic. A t‐test comparing Z‐statistics from all 22 studies suggests that the D&L method tends to be more conservative [d(Peto—D&L) = 0.29, t = 2.53, p = 0.02]. For a subset of 14 non‐heterogeneous studies, the difference is smaller and non‐significant (d = 0.09, t = 0.72, p = 0.49). The results from the methods correlate well (r = 0.66 for all 22 studies, r = 0.95 for 14 non‐heterogeneous studies). Thus, the presence of heterogeneity influences our conclusion. We discuss the statistical and scientific implications of these findings.