#HEORInsights

Systematic reviews (SRs) are incredibly crucial for healthcare decision-making, as they often provide a reliable summary of evidence on the comparison among healthcare interventions. They identify, assess, and combine the results of similar but individual studies and help to clarify the known and unknown benefits and risks associated with drugs, devices, and other healthcare interventions. SRs are helpful for clinicians to incorporate research findings into their daily practices, for patients to make informed choices about their care, and for professional medical organizations to develop clinical practice recommendations. (1)

Their importance in clinical decision-making has led to the rising number of published SRs. Consequently, the quality of published SRs is also being questioned. Findings of a cross-sectional study by Page et al. (2016) have observed poor quality of conduct and reporting of many SRs, despite the availability of the relevant reporting standards, such as the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement. (2) This is a big issue since poorly conducted and reported SRs often give away misleading conclusions that significantly affect decision-making. (2,3)

Cochrane SRs, published in the Cochrane Database of Systematic Reviews, are typically considered the ‘gold standard’ (4) and superior to other non-Cochrane SRs, are often contracted by policy-makers, and involve many safeguards against the potentially ‘misleading conclusions’.(3) Health Technology Assessment (HTA) is a specific area where an SR providing evidence on the clinical efficacy of a medical device/technology plays a crucial role in decision-making and would represent a different type of ‘non-Cochrane’ review. One such group of SRs is conducted for the UK National Institute of Health Research (NIHR) HTA programme (HTA-SRs). The full texts of HTA-SRs are published in the programme’s own journal, Health Technology Assessment (Winchester). (5)

The HTA-SRs show a remarkable comparison with Cochrane reviews as they have clear reporting standards and are not limited by word restrictions or the lack of online appendices, unlike many other non-Cochrane reviews published in conventional peer-reviewed journals. The nature and reporting of HTA-SRs, Cochrane, and non-Cochrane reviews are similar in many aspects but different in several other areas, such as, a more extensive range of included studies, the failure of HTA-SRs to report the total number of participants from included and instead reporting a range, the availability and registration of SR protocols, and a statement specifying adherence to respective guidelines while conducting the review. Whereas, concerning the conduct and reporting of the study selection, data extraction, and critical appraisal processes, the HTA-SRs have been observed to be better reported than non-Cochrane reviews.(3)

Predominantly, it has been observed that the reporting characteristics of HTA-SRs published in the Health Technology Assessment journal are analogous to Cochrane reviews and better than many other non-Cochrane reviews in several aspects. These include identification as an SR, review registration and protocol availability, conflicts of interest, mentioning the type of literature included, particulars of the strategies for database search, trial registry and grey literature search, the use of PRISMA flow diagrams, providing details of any excluded studies; and the reporting of limitations at the review level and of the included studies. All these factors impact informed decision-making, particularly the one that highlights the importance of sourcing unpublished data, (6,7) and explains uncertainties within the evidence-base and review itself.(3) Apparently, HTA-SRs have weaker reporting than Cochrane reviews across several characteristics, as reported earlier. HTA-SRs are also not expected to develop recommendations based on the quality of the evidence since that responsibility is on the other groups in the HTA process. (8)

The conduct and reporting of Cochrane reviews and those published in the UK Health Technology Assessment journal are thus of the same standard and usually better than many other non-Cochrane reviews. Accordingly, the HTA-SRs should arguably be considered equivalent to supposed Cochrane ‘gold standard’, and not be clubbed together with other non-Cochrane reviews. They are approved by regulators from the UK Department of Health (National Institute of Health and Care Excellence [NICE] and the NIHR) with a particular policy-making and decision-making objective and audience in mind.(5)  Also, it would not be wrong to say that any systematic reviews conducted for the purpose of HTA should follow the reporting guidelines of the HTA-SRs. The processes for conducting and reporting SRs for HTA have to be transparent, rigorous, and of the highest quality.  Bias in the review needs to be reduced, and the possibility of ‘misleading conclusions’ cannot be allowed in HTA-SRs because there is a lot at stake.(3)

Become A Certified HEOR Professional – Enrol yourself here!

References

1. Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press.
2. Page M, Shamseer L, Altman D, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med 2016; 13(5):e1002028.
3. Carroll C, Kaltenthaler E. Nature and reporting characteristics of UK health technology assessment systematic reviews. BMC Medical Research Methodology 2018; 18:35.
4. Cochrane library. Available at: https://www.cochranelibrary.com/about/about-cochrane-reviews
5. National Institute for Health Research (NIHR): Health Technology Assessment Programme. Available at: https://www.nihr.ac.uk/funding-and-support/funding-forresearch-
6. studies/funding-programmes/health-technology-assessment/.
7. Jones C, Keil L, Weaver M, et al. Clinical trials registries are underutilized in the conduct of systematic reviews: a cross-sectional analysis. Syst Rev 2014; 3:126.
8. Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ 2012; 344:d7202.
9. Rotstein D, Laupacis A. Differences between systematic reviews and health technology assessments: a trade-off between the ideals of scientific rigor and the realities of policy making. Int J Technol Assess Health Care 2004; 20:177–83.