The heterogeneity statistic I2 can be biased in small meta-analyses, von Hippel, 2015

Background

Estimated effects vary across studies, partly because of random sampling error and partly because of heterogeneity. In meta-analysis, the fraction of variance that is due to heterogeneity is estimated by the statistic I2. We calculate the bias of I2, focusing on the situation where the number of studies in the meta-analysis is small. Small meta-analyses are common; in the Cochrane Library, the median number of studies per meta-analysis is 7 or fewer.

Methods

We use Mathematica software to calculate the expectation and bias of I2.

Results

I2 has a substantial bias when the number of studies is small. The bias is positive when the true fraction of heterogeneity is small, but the bias is typically negative when the true fraction of heterogeneity is large. For example, with 7 studies and no true heterogeneity, I2 will overestimate heterogeneity by an average of 12 percentage points, but with 7 studies and 80 percent true heterogeneity, I2 can underestimate heterogeneity by an average of 28 percentage points. Biases of 12–28 percentage points are not trivial when one considers that, in the Cochrane Library, the median I2 estimate is 21 percent.

Conclusions

The point estimate I2 should be interpreted cautiously when a meta-analysis has few studies. In small meta-analyses, confidence intervals should supplement or replace the biased point estimate I2.

https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-015-0024-z

 

The Cochrane review of exercise, when comparing exercise to usual care, reported:

“The meta‐analysis of fatigue at follow‐up was also associated with heterogeneity (Analysis 1.2). Exclusion of Powell 2001 from the meta‐analysis resulted in a smaller SMD of −0.27 (95% CI −0.54 to 0.00) and reduced heterogeneity (I² = 49%, P = 0.16).”

This analysis included only 3 studies (counting the exclusion of Powell) and has a moderately large estimated I2, so according to van Hoppel’s paper, it may be that the actual I2 is underestimated and could perhaps be around 60% or above.

The authors state:

“We assessed heterogeneity in accordance with the recommendations of the Cochrane Handbook for Systematic Reviews of Interventions (I² values of 0% to 40%: might not be important; 30% to 60%: may represent moderate heterogeneity; 50% to 90%: may represent substantial heterogeneity; 75% to 100%: show considerable heterogeneity; Deeks 2011).”

There may thus have been substantial (50-90%) heterogeneity in this analysis instead of moderate heterogeneity. In other words, the studies differ too much to estimate the effect of exercise vs usual care at follow-up accurately. The certainty of this finding as assessed with GRADE was already very low, though.

 

The data on long-term follow-up is based on 3 studies, or 4 if Powell is included.

The post-treatment data is based on 7 studies (including Powell).