NCPA - National Center for Policy Analysis

Flaws Found In Most Meta-Analytical Studies

February 25, 2000

Most systematic reviews or meta-analyzes of other studies published in peer reviewed journals or funded by industry have serious methodological flaws that limit their value to guide decisions, say medical researchers.

A team of British medical researchers searched for meta-analyzes or systematic reviews of studies on the treatment of asthma published in peer reviewed journals, the Cochrane Library and medical databases, such as Medline and HealthSTAR.

Meta-analysis is the statistical combination of the results of several independent studies to produce a single estimate of the effect of a particular intervention or health care situation. In the case of asthma, for instance, meta-analysis has been used to evaluate studies on the effects of second-hand cigarette smoke on asthmatics.

The new study looked at the characteristics of the published studies, such as the type of data synthesis used and the quality of the methodology adopted by the studies' authors. Twelve of the reviews were published in the Cochrane Library and 38 were published in 22 peer reviewed journals. Three trained reviewers extracted data independently from the studies and differences were resolved by a fourth reviewer

  • Of the 50 systematic reviews and meta-analyses of asthma studies examined by researchers, 40 were judged to have serious or extensive flaws.
  • All six reviews funded by industry were in this group.
  • Seven of the 10 most rigorous reviews were published by the Cochrane Collaboration, a large international group.

Thus, they conclude, most systematic reviews and meta-analyses published in peer reviewed journals have methodological deficiencies that may limit their validity. And Cochrane reviews are more rigorous and better reported than those published in peer reviewed journals.

Source: Alejandro R Jadad, et al., "Systematic reviews and meta-analyses on treatment of asthma: critical evaluation," British Medical Journal, February 27, 2000.


Browse more articles on Health Issues