Assessing the methodological quality of network meta-analysis

from CEA Registry Team at on April 29, 2015 at 09:53PM

By James Chambers, Ph.D.

Clinical researchers and health care decision makers are increasingly using network meta-analysis to synthesize comparative effectiveness evidence of competing treatments that have not been studied in head-to-head clinical trials.  In a new paper published in PLOS ONE, my colleagues and I examine the methodological quality of the network meta-analysis literature.1

We systematically searched the medical literature for network meta-analyses evaluating pharmaceuticals, and identified 318 published papers through July 30th, 2014.  We assessed these network meta-analyses using objective criteria from the checklist of good research practices in the ISPOR guidance for interpreting and conducting network meta-analysis studies.2,3  Our assessment criteria pertained to the following:

1.    Methodological approach – for example, did the study report an assessment of the risk of bias in included trials, or an assessment of the consistency between direct and indirect evidence.
2.    Study transparency and reproducibility – for example, did the study report the search terms used, and were the extracted clinical trial data presented.
3.    Reporting of findings – for example, was a matrix of head-to-head comparisons presented.

To better understand the literature, we then used our assessment criteria to compare network meta-analyses published in journals with lower and higher impact factors, older and more recently published studies, and studies supported by the pharmaceutical industry and studies supported by non-industry sources or receiving no support.  The findings are presented in Table 1.

We found notable differences in the quality of network meta-analyses.  Studies published in higher impact factor journals and those that did not receive industry support performed better across our assessment criteria.  We found few differences between older studies and those published more recently.

Study findings highlight the variation in published network meta-analyses and underline the need for consensus among guidelines to guide their conduct and reporting. 

Click to view a larger image.


1. Chambers JD, Naci H, Wouters OJ, Pyo J, Gunjal S, Kennedy IR, Hoey MG , Winn A, Neumann PJ. An assessment of the methodological quality of published network meta-analyses: A systematic review PLOS ONE 2015 

2. Jansen JP, Fleurence R, Devine B, Itzler R, Barrett A, Hawkins N, Lee K, Boersma C, Annemans L, Cappelleri JC. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health. 2011 Jun;14(4):417-28.

3. Hoaglin DC, Hawkins N, Jansen JP, Scott DA, Itzler R, Cappelleri JC, Boersma C, Thompson D, Larholt KM, Diaz M, Barrett A.Conducting indirect-treatment-comparison and network-meta-analysis studies: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 2. Value Health. 2011 Jun;14(4):429-37.

Tags: network meta-analysis,PLOS ONE,Recently published studies