#HEORInsights

Clinical prediction models (CPMs) are statistical models that use patient characteristics and clinical variables to estimate the probability of a particular health outcome, such as a disease or adverse event. CPMs can be diagnostic prediction models that aid in diagnosis, by predicting the likelihood that a person is currently having a particular health condition (for example, Wells score for pulmonary embolism). Another type of CPMs is the prognostic prediction models, which aid in prognosis by predicting the likelihood that a person will experience a particular health outcome over a specific period (for example, the Framingham Risk Score for cardiovascular disease). (1) These days CPMs have become an essential part of evidence-based clinical practice, and thus it becomes important that the CPMs provide accurate estimation of the disease condition for which they are used. (2, 3) Towards this direction, systematic literature reviews (SLRs) are often conducted to determine the quality and validity of CPMs, as well as to identify gaps in the literature to inform the development of new CPMs. (4)

The CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) checklist is a tool that has been developed to facilitate the critical evaluation and data extraction while performing SLRs of CPMs. First published in 2014, this checklist provides explicit guidance to help reviewers and users to frame the right review question. It also provides a data extraction list with explicit guidance on which items to extract from CPM studies for evaluating the risk of bias and applicability. It can be used to critically appraise all types of primary prediction model studies for all kinds of the target population, outcomes, and predictors, regardless of the statistical techniques used. (5) Thus, the CHARMS checklist helps the researchers in two critical areas of conducting an SLR: framing the review question, and critical appraisal of included articles.

The contents of the CHARMS checklist are arranged in two sections. First, there is a list of 7 key items to help the researcher frame a well-defined, proper, and focussed review questions; these key items include the intended scope of the review, type of prediction model studies, prognostic versus diagnostic prediction model, target population to whom the prediction model applies, outcome to be predicted, the period of the prediction, and the intended moment of using the model. Next, the checklist provides guidance for data extraction and critical appraisal of the included articles in the SLR by means of 35 key items that are organized in 11 domains: source of data, participants, outcome to be predicted, candidate predictors, sample size, missing data, model development, model performance, model evaluation, results, interpretation and discussion. (4, 5) 

Ever since its publication in 2014, the CHARMS checklist has been used by various SLRs of CPMs. (6-10) To further facilitate the easier application of the CHARMS tool, an Excel template for data extraction and risk of bias assessment of clinical prediction models has been recently published; this template makes it possible for the user to apply both the CHARMS and the PROBAST (Prediction model Risk Of Bias Assessment Tool) while conducting critical appraisal of SLRs of CPMs. It will also encourage more accurate and thorough reporting of these systematic reviews. (11)

Critical appraisal of included studies is an essential part of any SLR, and SLRs of CPMs is not an exception. It is crucial that the tool used for the critical appraisal is validated, asks the correct questions, is user-friendly, and gives accurate results. The CHARMS checklist, standing the test of the time, fits the bill perfectly, and along with the PROBAST tool, has become an invaluable resource in the conduct of SLRs of CPMs.

Become A Certified HEOR Professional – Enrol yourself here!

References:

1. Vogenberg FR. Predictive and prognostic models: implications for healthcare decision-making in a modern recession. Am Health Drug Benefits. 2009 Sep;2(6):218-22. 
2. Van Smeden M, Reitsma JB, Riley RD, et al. Clinical prediction models: diagnosis versus prognosis. Journal of clinical epidemiology. 2021 Apr 1;132:142-5.
3. Hendriksen JM, Geersing GJ, Moons KG, de Groot JA. Diagnostic and prognostic prediction models. J Thromb Haemost. 2013 Jun;11 Suppl 1:129-41. 
4. Damen JAA, Moons KGM, van Smeden M, Hooft L. How to conduct a systematic review and meta-analysis of prognostic model studies. Clin Microbiol Infect. 2023 Apr;29(4):434-440. Doi: 10.1016/j.cmi.2022.07.019. Epub 2022 Aug 4. 
5. Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PloS Med. 2014Oct 14;11(10):e1001744. 
6. Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. Bmj. 2020 Apr 7;369.
7. Viswanathan M, Patnode CD, Berkman ND, et al. Assessing the risk of bias in systematic reviews of health care interventions. Methods guide for effectiveness and comparative effectiveness reviews [Internet]. 2017 Dec 13.
8. Damen JA, Hooft L, Schuit E, et al. Prediction models for cardiovascular disease risk in the general population: systematic review. bmj. 2016 May 16;353.
9. Smith EE, Kent DM, Bulsara KR, et al. Accuracy of prediction instruments for diagnosing large vessel occlusion in individuals with suspected stroke: a systematic review for the 2018 guidelines for the early management of patients with acute ischemic stroke. Stroke. 2018 Mar;49(3):e111-22.
10. Meehan AJ, Lewis SJ, Fazel S, et al. Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges. Molecular Psychiatry. 2022 Jun;27(6):2700-8.
11. Fernandez-Felix BM, López-Alcalde J, Roqué M, Muriel A, Zamora J. CHARMS and PROBAST at your fingertips: a template for data extraction and risk of bias assessment in systematic reviews of predictive models. BMC Medical Research Methodology. 2023 Dec;23(1):1-8.