Contact Us

Home

About us

Services

AI Excellence

case studies

publications

HEOR Certificationheor Certification

AI for Medical Writing

Contact

  • Assessing the Reliability of Published Systematic Literature Reviews

    Assessing the Reliability of Published Systematic Literature Reviews

    Systematic Literature Reviews (SLRs) are the gold standard in evidence synthesis, occupying the pinnacle of the evidence pyramid.[1] Their trustworthiness is paramount, as SLRs frequently form the foundation of evidence-based guidelines and consensus statements.[2] SLRs differ from narrative reviews because the former aims to provide a comprehensive, unbiased summary of all relevant research on a specific question, considering all possible evidence, both favoring and opposing a particular topic of interest. SLRs are also quite helpful in identifying gaps in the current knowledge, thereby providing a direction in terms of future research efforts. Thus, it becomes essential that the methods employed in conducting an SLR are robust, authentic, and reliable so that the resultant evidence can be trustworthy.[1,2]

    Poorly conducted or reported SLRs can have wide-ranging negative effects. Despite their objective to provide an evidence-based synthesis, SLRs at times do not meet the rigorous standards expected. Critics argue that such SLRs of poor methodological quality or high degree bias contribute to research waste, and can be misleading or serve conflicted interests.[3] Many poor-quality reviews continue to be published, even though clear guidance has been available.[4]

    In an era of perverse academic incentives, the publication of redundant, overlapping, unreliable, or poor-quality SLRs is still relentless. Research has identified several issues, including redundancy, with multiple SLRs covering the same topic, often with similar conclusions. Methodological flaws are also prevalent, such as inadequate search strategies, incomplete data extraction, and poor statistical analyses. Furthermore, biased conclusions are a problem, with exaggerated or misleading interpretations of results. Poor reporting is another issue, with inadequate disclosure of methods, conflicts of interest, or funding sources. These flaws have been highlighted in various studies, but their impact is not being adequately addressed. Consolidating these findings is crucial to understanding the scale of the problem and pushing for improvements in SR quality.[5,6] A study conducted in 2023 found that between 2000 and November 2022, at least 485 articles documented issues with published SLRs, ranging from editorials highlighting concerns over specific reviews to rigorous analyses of issues with hundreds or thousands of reviews.[7]

    To ensure systematic reviews achieve their potential as reliable sources of evidence, it is essential to implement specific measures and maintain rigorous standards. SLRs should aim to include all relevant studies. Problems arise when relevant studies are missed or ignored, which can compromise the review’s validity. These issues can stem from overly stringent inclusion criteria, exclusion of grey literature, insufficient or outdated literature searches, and language restrictions. Additionally, appropriate methods must be used to ensure methodological soundness of the review. Errors in conducting the review or a lack of expertise can jeopardize the review’s internal validity. Issues including data extraction errors, flawed risk of bias assessments, limited quality assessment, and failure to incorporate risk of bias into conclusions can contribute to this.[7]

    To ensure the reproducibility of systematic literature reviews, it is essential to report their methods in sufficient detail. Poor reporting quality or inaccessible methods can hinder the ability of others to replicate the review’s findings. This is particularly problematic when reviews are used to inform important decisions. To address this issue, review authors should adhere to reporting guidelines like PRISMA 2020 (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and register their review protocols in databases such as PROSPERO.[7, 8]

    SLRs can become outdated over time due to the rapid pace of scientific research. New studies are constantly being published, and these can introduce new evidence that may challenge or modify the findings of existing SLRs. Additionally, the context in which SLRs are conducted can change, rendering previous findings less relevant or applicable. Living SLRs are a dynamic approach to evidence synthesis that addresses this limitation of traditional SLRs. Living SLRs are continuously updated as new research emerges. This ensures that the conclusions and recommendations of the review remain relevant and accurate over time.[8]

    Tools such as AMSTAR 2 (Assessment of Multiple Systematic Reviews) and ROBIS have been developed to assess the methodological quality and risk of bias of SLRs that have already been published. Such tools evaluate whether published SLRs have high quality in terms of internal validity, bias, and quality. Further, conducting double-checks of data and contacting statistical experts ensures results consistency and validates findings, increasing confidence in the review’s conclusions. Results should be interpreted with careful consideration of quality, risk of bias, and certainty, and any limitations or gaps in the evidence base should be acknowledged. Additionally, disclosing potential conflicts of interest and managing researcher bias are critical to ensuring that SR conclusions are not unduly influenced by conflicted parties, and that the review’s findings can be trusted by stakeholders.[8-12]

    In conclusion, the reliability of SLRs is crucial in guiding healthcare and policy decisions. As research continues to expand, ensuring the integrity and rigor of these systematic reviews is more important than ever. By following best practices, maintaining clarity, and properly applying methodological frameworks, the scientific community can safeguard the credibility of SLRs.

    Become A Certified HEOR Professional – Enrol yourself here!

    References:

    1. Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evid Based Med. 2016 Aug;21(4):125–7.
    2. Uttley L, Quintana DS, Montgomery P, Carroll C, Page MJ, Falzon L, Sutton A, Moher D. The problems with systematic reviews: a living systematic review. Journal of Clinical Epidemiology. 2023 Apr 1;156:30-41.
    3. Dr Jenny McSharry, What health evidence can we trust when we need it most? Cochrane news https://www.cochrane.org/news/what-health-evidence-can-we-trust-when-we-need-it-most.
    4. Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta‐analyses. The Milbank Quarterly. 2016 Sep;94(3):485-514.
    5. Uttley L, Montgomery P. The influence of the team in conducting a systematic review. Systematic reviews. 2017 Dec;6:1-4.
    6. Chapelle C, Ollier E, Bonjean P, Locher C, Zufferey PJ, Cucherat M, Laporte S. Replication of systematic reviews: is it to the benefit or detriment of methodological quality?. Journal of Clinical Epidemiology. 2023 Aug 28.
    7. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. bmj. 2021 Mar 29;372.
    8. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. bmj. 2017 Sep 21;358.
    9. Bristol U of. ROBIS tool [Internet]. University of Bristol. Available from: https://www.bristol.ac.uk/population-health-sciences/projects/robis/robis-tool/
    10. Dang A, Chidirala S, Veeranki P, Vallish BN. A Critical Overview of Systematic Reviews of Chemotherapy for Advanced and Locally Advanced Pancreatic Cancer using both AMSTAR2 and ROBIS as Quality Assessment Tools. Rev Recent Clin Trials. 2021;16(2):180-192.
    11. Uttley L, Quintana DS, Montgomery P, Carroll C, Page MJ, Falzon L, Sutton A, Moher D. The problems with systematic reviews: a living systematic review. Journal of Clinical Epidemiology. 2023 Apr 1;156:30-41.
    12. Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, Boutron I, Sarkis-Onofre R, Bjerre LM, Hróbjartsson A, Altman DG. Systematic review adherence to methodological or reporting quality. Systematic reviews. 2017 Dec;6:1-4

    MarksMan Healthcare

    , , ,

  • Date Range Restriction in Systematic Reviews: Navigating the Why, How, and When

    Date Range Restriction in Systematic Reviews: Navigating the Why, How, and When

    Systematic literature reviews (SLRs) play a pivotal role in evidence synthesis and informing healthcare decisions. One critical aspect of conducting an SLR is establishing a date range for the inclusion of studies. This temporal parameter, while seemingly straightforward, demands careful consideration. The decision to impose a date range restriction can significantly impact the comprehensiveness and reliability of the review.[1]

    The imposition of date range restrictions in SLRs serves several critical purposes. Firstly, it ensures the inclusion of the most recent and relevant evidence, acknowledging the dynamic nature of the healthcare landscape characterized by continuous advancements in research and technology. By setting a cutoff date, reviewers aim to focus on contemporary studies that accurately reflect the current state of knowledge and practice. Secondly, the application of date range restrictions is instrumental in managing the considerable workload associated with the SLR process. Given the vast volume of literature that the SLRs often entail, narrowing the focus to a specific timeframe becomes imperative. This strategic restriction not only streamlines the SLR process but also ensures that it remains feasible within the constraints of time and resources, enabling a comprehensive overview of the relevant literature. Additionally, date range restrictions play a crucial role in preventing the inclusion of outdated information. In healthcare, interventions, guidelines, and treatments undergo continuous refinement. Excluding older studies through a well-defined date range restriction becomes a deliberate choice aimed at eliminating information that may no longer be applicable or reflective of current best practices. This careful curation is essential for maintaining the relevance of SLR and its applicability to contemporary healthcare settings.[2]

    Effectively implementing date range restrictions in SLRs involves a thoughtful approach. Firstly, it is essential to clearly define the rationale behind the decision before setting a date range. Whether the goal is to capture the latest evidence, manage workload, or ensure relevance, a well-articulated rationale helps justify the chosen temporal parameters. Furthermore, the nature of the research question should be a guiding factor in establishing the date range. Different questions may require different temporal perspectives; some may necessitate a focus on recent developments, while others benefit from a historical context. Aligning the date range with the research question enhances the precision and relevance of the SLR, ensuring that the chosen timeframe aligns with the objectives of the study.[2]

    Transparency is of paramount importance in SLRs. Authors should explicitly report the chosen date range in the methods section, accompanied by a clear explanation of the rationale behind the decision. This transparency is crucial for aiding readers and reviewers in assessing the validity and applicability of the findings of the SLR, and in establishing trust in the methodology employed. In addition to transparency, the robustness of the SLR can be enhanced by conducting sensitivity analyses to assess the impact of the date range restriction. Comparing the results with and without restriction provides insights into the potential influence of the chosen temporal parameters on the overall findings of the SLR. This analytical approach adds an extra layer of validation, demonstrating the thoroughness and reliability of the methodology of the SLR.[3]

    Imposing an inaccurate date cutoff in SLRs can have significant consequences. Very important among them is the potential loss of valuable information, particularly when a crucial comparator or pivotal study was published just before the specified date. Excluding such pertinent studies may result in an incomplete representation of available evidence, compromising the robustness and validity of the conclusions drawn by the SLR. Furthermore, date range restrictions can introduce bias into comparisons between interventions or treatments. If comparators are subject to different date ranges, the SLR may inadvertently favor interventions with more recent evidence, leading to skewed conclusions. This bias has the potential to impact the overall assessment of interventions and treatments, hampering the reliability of the findings of the SLR. Moreover, the imposition of a rigid date range can result in the neglect of historical context. Certain research questions benefit from an exploration of older studies that provide valuable insights into the evolution of interventions or treatments. An inflexible date range may overlook these insights, impacting the depth and richness of the narrative. Therefore, the consequences of an inaccurate cutoff extend beyond mere omission, influencing the integrity and comprehensiveness of the findings of the SLR.[4]

    The consequences extend to meta-analyses, where date range restrictions can have unintended effects. If the chosen date range disrupts the balance of studies across interventions, the synthesis of data may be skewed. This imbalance can influence the overall effect size and draw potentially misleading conclusions from the meta-analysis, undermining the reliability of the quantitative synthesis. In assessing studies just outside the specified date range, their potential impact on the overall review’s robustness should be carefully reviewed. If the exclusion of these critical studies proves to compromise the completeness and reliability of the SLR significantly, it may prompt a re-evaluation of the initially set date range.[2]

    In rapidly evolving fields, where emerging trends or interventions may lack a substantial body of recent literature, strict date range restrictions can pose limitations on the SLR’s ability to capture the full spectrum of available evidence. Reviewers must be aware of this challenge and contemplate alternative approaches. One such approach could involve conducting separate analyses for emerging and established interventions, recognizing the unique dynamics and evidence landscapes within these categories. To further mitigate the risk of overlooking important evidence, systematic reviewers should consider the incorporation of continuous surveillance mechanisms and periodic updates. This proactive strategy enables the ongoing integration of new evidence without compromising the overall structure and methodology of the review. By staying attuned to emerging studies, reviewers can enhance the timeliness and relevance of their SLRs, ensuring that they remain reflective of the dynamic nature of the field.[4]

    In conclusion, the imposition of date range restrictions in SLRs is a crucial methodological decision that requires careful consideration. While there are valid reasons for setting temporal boundaries, the potential consequences of an inaccurate cutoff cannot be understated. The loss of valuable information, biased comparisons, and unintended impacts on meta-analyses underscore the importance of a nuanced approach to date range selection. Systematic reviewers must strive for transparency, align the chosen date range with the research question, and remain open to reconsideration when critical studies are just outside the specified timeframe. By navigating the complexities of date range restrictions with diligence and flexibility, SLRs can fulfill their role as reliable sources of evidence for informed decision-making in healthcare.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group*. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine. 2009 Aug 18;151(4):264-9.
    2. Helbach J, Pieper D, Mathes T, Rombey T, Zeeb H, Allers K, Hoffmann F. Restrictions and their reporting in systematic reviews of effectiveness: an observational study. BMC Medical Research Methodology. 2022 Dec;22(1):1-0.
    3. Aali G, Shokraneh F. No limitations to language, date, publication type, and publication status in search step of systematic reviews. Journal of Clinical Epidemiology. 2021 May 1;133:165-7.
    4. Sterne JA, Egger M, Smith GD. Investigating and dealing with publication and other biases in meta-analysis. Bmj. 2001 Jul 14;323(7304):101-5.

    MarksMan Healthcare

    , , ,

  • Including Conference Abstracts for Evidence Synthesis Through Systematic Reviews

    Including Conference Abstracts for Evidence Synthesis Through Systematic Reviews
    Including Conference Abstracts for Evidence Synthesis Through Systematic Reviews

    Systematic literature reviews (SLRs) serve as the gold standard for evidence synthesis, informing healthcare decisions. However, a pertinent question arises in the methodology of these reviews: should conference abstracts be included in the search for relevant studies? This query prompts a comprehensive examination of both the advantages and disadvantages, with the ultimate decision based on the specific goals of the SLR.[1]

    Conference abstracts, often preceding full publications, provide timely access to critical information. This timeliness is especially vital in rapidly evolving fields, ensuring that SLRs stay current and offer an up-to-date perspective on the existing evidence landscape. Further, considering that not all conference abstracts are published in peer-reviewed journals, excluding abstracts translates to the exclusion of valuable data, compromising the comprehensiveness of the SLR.[1]

    It is well known that SLRs that rely solely on published studies carry a risk of publication bias which can distort the overall treatment effect estimate. Publication bias arises out of the selective publication of studies with positive results. Including conference abstracts acts as a countermeasure to this, offering a more balanced representation of study outcomes, regardless of their direction. This inclusion contributes to a more nuanced understanding of the effectiveness of the intervention.[1]

    Excluding conference abstracts that report early trial findings particularly randomized controlled trials (RCTs) from the evidence synthesis process introduces ethical concerns as well. Patient participation is grounded in the expectation that their contribution will contribute to scientific knowledge. The non-fulfillment of this commitment is viewed as an ethical problem, emphasizing the importance of including conference abstracts to honor the contributions of study participants. This ethical dimension adds a layer of responsibility to systematic reviewers, urging them to consider the broader implications of excluding conference abstracts.[1]

    Conference abstracts frequently present findings with direct policy implications. Including them in SLRs ensures that policy decisions are informed by the latest and potentially impactful research. This alignment of SLR processes with real-world applications enhances the relevance and applicability of the synthesized evidence, making it a valuable resource for policymakers and healthcare practitioners. The advantages of including conference abstracts extend beyond the content they provide, contributing to a more dynamic and adaptive SLR process.[1]

    However, the process of identifying relevant conferences, extracting abstracts, and sifting through extensive collections can be resource-intensive. Despite advancements like the inclusion of conference abstracts in searchable databases such as EMBASE, challenges persist. Abstracts, characterized by brevity, may lack essential information for systematic reviewers to comprehensively assess study design, methods, bias risk, outcomes, and results. The concise nature of abstracts poses a challenge for reviewers seeking a thorough understanding of the included studies. Additionally, the non-standard format of conference abstracts presents an additional layer of complexity, further complicating the extraction and evaluation process.[3]

    Moreover, abstracts often lack peer review, and their preliminary nature may result in unreliable information. Discrepancies between abstracts and subsequent full publications, particularly in critical aspects such as methods and results, raise questions about the dependability of abstracts. While meta-epidemiologic studies have shown only minor differences in results between meta-analyses with and without conference abstracts, the inclusion of “gray” literature, including conference abstracts, may lead to shifts in significance levels or, in some cases, no substantial change.[3,4]

    In conclusion, the decision to include conference abstracts in SLRs should be carefully considered. The advantages, including timeliness, addressing publication bias, ethical considerations, and informing policy decisions, underscore the potential benefits. However, challenges related to resource intensity, information adequacy, and reliability necessitate a nuanced approach. By adopting a flexible and evidence-driven strategy, systematic reviewers can harness the benefits of including conference abstracts without compromising the integrity of their synthesis efforts.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Hackenbroich S, Kranke P, Meybohm P, Weibel S. Include or not to include conference abstracts in systematic reviews? Lessons learned from a large Cochrane network meta-analysis including 585 trials. Systematic Reviews. 2022 Dec;11(1):1-0.
    2. Murad H, Smith J, Singh G, Deber R. Methodological and reporting quality of conference abstracts: a systematic review. J Clin Epidemiol. 2013;66(7):705-15.
    3. Cohen AM, Glazner JE, Roeder K, Sandvik L. Does inclusion of published abstracts of randomized controlled trials in systematic reviews affect risk ratio estimates? J Clin Epidemiol. 2014;67(7):796-802.
    4. Egger M, Davey SG, Davey A. Bias in the selection of conference abstracts for presentation. Lancet. 1997;349(9055):1069-70.

    MarksMan Healthcare

    , , ,

  • Using Individual Participant Data for Network Meta-Analysis: Advantages and Challenges

    Using Individual Participant Data for Network Meta-Analysis: Advantages and Challenges

    The domain of healthcare research has witnessed remarkable advancements in the evaluation of various treatment modalities for specific diseases. One such pioneering approach, the Network Meta-Analysis (NMA), has proven instrumental in comparing multiple treatments assessed in randomized trials. Conventionally, NMAs primarily rely on Aggregate Data (AD) extracted from study publications, limiting the scope of analysis to the available summarized data. However, an innovative alternative that has gained traction in recent years is the integration of Individual Participant Data (IPD) into the NMA framework. This approach involves procuring raw information recorded for each participant in a study, allowing for a more comprehensive and nuanced evaluation of treatment effects.[1,2]

    A pivotal strength of incorporating IPD into NMA lies in the enhanced quality and scope of information available for each trial. Unlike AD, IPD circumvents the limitations posed by the published data, thereby facilitating a comprehensive analysis. This comprehensive approach fosters the standardization of outcome and covariate definitions across trials, reducing the potential for reporting biases. By harmonizing diverse definitions of outcomes, IPD enables a more robust analysis of continuous scales, yielding more precise and reliable conclusions. Additionally, the collaborative nature of IPD NMA projects, involving direct engagement with trial investigators, facilitates a more thorough identification of relevant trials, thereby reinforcing the reliability of risk assessments and data analyses.[2-4]

    Another compelling advantage of employing IPD for NMAs is the ability to examine and plot distributions of covariates across trials. This granular approach empowers researchers to discern systematic differences in variables that might influence measures of relative treatment effect. With access to a broader set of recorded covariates, IPD enables a more detailed comparison of how covariate distributions vary across trials, thus aiding in the assessment of their potential impact on network heterogeneity and inconsistency. Notably, this approach contributes significantly to the evaluation of the consistency assumption, a critical factor in ensuring the reliability of NMA results.[4]

    Furthermore, IPD allows for the standardization and improvement of the analysis of each trial, enabling researchers to define their own estimands of interest. By standardizing effect measures across trials, IPD facilitates the utilization of advanced analytical approaches, particularly in the modeling of time-to-event data and the handling of missing data. The utilization of a multivariate NMA approach becomes feasible, enabling the simultaneous comparison of treatments across multiple outcomes, while accounting for correlations among these outcomes. This multifaceted analysis not only enhances the precision of inferences but can also alter the ranking of treatments, thereby providing a more comprehensive perspective for clinical decision-making.[4]

    Using IPD in NMA can also allow adjusting for prognostic factors in the analysis of each trial. This aspect is instrumental in improving the power to detect treatment effects, especially in scenarios where stratified randomization has been employed. Moreover, the inclusion of treatment-covariate interactions facilitated by IPD, allows for a more comprehensive understanding of how treatment effects may vary across different patient populations. This nuanced analysis paves the way for tailored treatment recommendations, aligning with the evolving paradigm of personalized medicine and patient-centric care.[4]

    However, the integration of IPD into NMA projects is not without its share of challenges. The substantial time and effort required to procure, validate, harmonize, and synthesize IPD often pose significant logistical hurdles. Negotiating data-sharing agreements and maintaining effective collaboration with trial investigators demand meticulous planning and clear communication to ensure the smooth execution of the research. Additionally, the inclusion of data from diverse sources may introduce data heterogeneity. Performance of IPD NMA would necessitate expertise to handle both the IPD as well as perform NMA: thus, IPD NMA requires higher and complicated statistical analytic techniques, adding to the resource-intensiveness of the technique. Mitigating concerns regarding availability bias, stemming from the unavailability of IPD from certain studies, remains a persistent challenge that requires careful consideration and methodological adjustments. Furthermore, data privacy, confidentiality, and ethical considerations arise due to the transmission of sensitive information contained in the IPD, adding another layer of complexity.[4]

    To address the challenges discussed above, various innovative methodologies have emerged, such as multilevel network meta-regression, which extends the IPD NMA framework to incorporate both IPD and AD, ensuring robust and accurate estimates across diverse target populations. Additionally, matching-adjusted indirect comparison, simulated treatment comparison, and predictive-adjusted indirect comparison have been proposed as alternative approaches to account for limited IPD, although their applicability remains confined to specific study designs and populations.[4-7]

    In conclusion, the integration of IPD into NMA represents a paradigm shift in evidence synthesis, offering unprecedented insights into treatment effects and patient outcomes. Despite the challenges posed by the complex nature of IPD meta-analyses, the benefits of utilizing this approach far outweigh the obstacles, fostering a more holistic and informed approach to clinical decision-making and healthcare policy formulation. As the healthcare landscape continues to evolve, the judicious utilization of IPD in NMA is poised to redefine the trajectory of evidence-based medicine, leading to more effective and tailored healthcare interventions for diverse patient populations.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Dias S, Ades AE, Welton NJ, et al. Network meta-analysis for decision-making. John Wiley & Sons; 2018 Jan 8.
    2. Riley RD, Tierney JF, Stewart LA, editors. Individual participant data meta-analysis: a handbook for healthcare research. John Wiley & Sons; 2021 May 24.
    3. Riley RD, Phillippo DM, Dias S. Network Meta‐Analysis Using IPD. Individual Participant Data Meta‐Analysis: A Handbook for Healthcare Research. 2021 Apr 22:347-73.
    4. Riley RD, Dias S, Donegan S, et al. Using individual participant data to improve network meta-analysis projects. BMJ evidence-based medicine. 2023 Jun 1;28(3):197-203.
    5. Chaimani A. Conduct and reporting of individual participant data network meta-analyses need improvement. BMC medicine. 2020 Jun 2;18(1):156.
    6. Phillippo DM, Dias S, Ades AE, et al. Multilevel network meta-regression for population-adjusted treatment comparisons. Journal of the Royal Statistical Society Series A: Statistics in Society. 2020 Jun;183(3):1189-210.
    7. Phillippo D, Ades T, Dias S, et al. NICE DSU technical support document 18: methods for population-adjusted indirect comparisons in submissions to NICE.

    MarksMan Healthcare

    , , ,

  • Automation in Evidence Synthesis: Are We There Yet?

    Automation in Evidence Synthesis: Are We There Yet?

    In the ever-evolving landscape of scientific research and evidence synthesis, the demand for timely, comprehensive, and reliable information has never been greater. Decision-makers, healthcare professionals, and researchers seek up-to-the-minute insights to inform their actions and conclusions. In response to this need, the concept of living systematic literature reviews (SLRs) has emerged, ushering in a new era of continuous evidence updates. However, the question that looms large is whether automation in evidence synthesis has caught up with the pace of this dynamic endeavor.[1]

    Traditional SLRs have long been the gold standard for evidence synthesis. They involve a meticulous and often time-consuming process of gathering, appraising, and synthesizing data to provide a comprehensive overview of a particular topic. Yet, this approach is inherently static, lagging behind the ever-accelerating pace of scientific discovery. Living SLRs, on the other hand, offer a dynamic solution to this problem. These reviews are designed to evolve in parallel with the evidence being generated. They provide a continuous stream of up-to-date information, ensuring that stakeholders have access to the latest insights in real time. This approach is particularly invaluable in fields where the evidence base is rapidly changing, such as public health emergencies or emerging medical treatments.[2,3]

    While the concept of living SLRs is undoubtedly promising, it comes with its own set of challenges. Conducting and maintaining such reviews can be resource-intensive and time-consuming. Reviewers face the formidable task of constantly monitoring newly published research and integrating it into the evolving review. Moreover, the speed at which new studies are published can introduce challenges related to the quality and reliability of the evidence. To address these challenges, automation has emerged as a potential ally. Automation tools have the potential to streamline various aspects of the review process – both traditional SLRs as well as living SLRs. Automation tools can potentially help with multiple steps in an SLR process, including reference retrieval, literature screening, data extraction, quality assessment, data synthesis, and reporting.[2]

    One of the most time-consuming aspects of evidence synthesis is screening references for relevance from an initial pool of potential hits. Automation tools have introduced machine learning algorithms that actively prioritize relevant references. Many such tools implement these algorithms, expediting the screening process and reducing the workload. Additionally, automation tools employ machine learning and neural networks to extract data and predict the risk of bias for randomized controlled trials. These tools enhance the efficiency of data extraction and quality assessment, enabling reviewers to focus on the interpretation of results rather than the mechanics of data extraction. Automation also plays a crucial role in disseminating living evidence. [4-6]

    Crowd-sourcing platforms have the potential to alleviate the burden on reviewers by outsourcing specific review tasks to students, researchers, or interested citizens. In addition to these specific SLR steps, many automation tools have been developed as web-based applications to support SLR workflows. They can automatically search databases like PubMed, pulling in references at regular intervals based on user-defined search strategies. This feature alone significantly reduces the manual effort required for reference retrieval.[3-5]

    While automation tools have made significant headway in supporting the SLR process, challenges and opportunities lie ahead. Integration between tools to facilitate data synthesis remains a notable gap. Different review topics may require tailored synthesis methods, and interoperability between tools is crucial to ensure a seamless flow of data between stages. In addition to this integration challenge, there is a pressing need to develop automation methods that can retrieve evidence from a broader range of data sources, including preprint servers. Additionally, tools must be transparent and well-validated, instilling trust in their reliability. Moreover, the legal and ethical aspects of sharing raw data, especially before formal publication, present challenges. Ensuring the quality and clarity of preprints is essential to prevent misinformation.[1]

    The increasing complexity of automation in the SLR process can potentially hamper the reproducibility of the research: novel solutions are needed to mitigate this concern and ensure consistent and replicable reviews. It is also equally important to consider if automated tools can accurately match human reviewers’ discernment in terms of extracting data, especially when human reviewers can uncover subtle insights and biases. All said, the current capacity of these automation tools to produce human-comparable insights remains challenging, particularly in nuanced aspects such as interpreting conflicting study results and evaluating qualitative research quality, where human judgment adds critical context and depth. Furthermore, compatibility with established values, such as rigor and transparency, is essential, emphasizing the need to double-check automated outputs for reproducibility and ensure transparency for accountability. Lastly, there is a pervasive skepticism and mistrust surrounding automation’s ability to replicate human judgment and value-based decisions. This skepticism underscores the necessity for human oversight and control as automation capabilities evolve. [1]

    The journey toward achieving a harmonious synergy between automation and evidence synthesis is ongoing. With each step forward, we move closer to a future where decision-makers, healthcare professionals, and researchers can access the latest evidence at the speed of discovery. While we may not be there just yet, the path ahead holds great promise for the field of evidence synthesis and its ability to inform critical decisions in an ever-changing world of knowledge.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Arno A, Elliott J, Wallace B, et al. The views of health guideline developers on the use of automation in health evidence synthesis. Systematic Reviews. 2021 Dec;10:1-0.
    2. Simmonds M, Elliott JH, Synnot A, et al. Living Systematic Reviews. Methods Mol Biol. 2022;2345:121-134.
    3. Schmidt L, Sinyor M, Webb RT, et al. A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen. 2023 Aug 16.
    4. Van Altena AJ, Spijker R, Olabarriaga SD. Usage of automation tools in systematic reviews. Research synthesis methods. 2019 Mar;10(1):72-82.
    5. Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic reviews. 2019 Dec;8:1-0.
    6. Khalil H, Ameen D, Zarnegar A. Tools to support the automation of systematic reviews: a scoping review. Journal of Clinical Epidemiology. 2022 Apr 1;144:22-42.

    MarksMan Healthcare

    , , ,

  • Challenges of Single-Arm Trials in Network Meta-Analyses (NMAs)

    Challenges of Single-Arm Trials in Network Meta-Analyses (NMAs)
    Challenges of Single-Arm Trials in NMA

    Network Meta-Analysis (NMA) is a sophisticated statistical technique that allows researchers to simultaneously compare multiple treatments using a combination of direct and indirect evidence. While NMAs have revolutionized evidence synthesis, the inclusion of single-arm trials (SATs) within these analyses has sparked discussions and raised important questions about the potential challenges they bring.[1]

    NMAs typically rely on a wealth of data, including randomized controlled trials (RCTs) and observational studies. However, there are instances where patient-level data from published studies might be inadequate, leading researchers to explore alternative avenues for incorporating evidence. This is where SATs come into play, offering a unique approach to addressing data gaps in NMA. SATs can function as mutual control groups based on the presence of complete covariate profiles and differing interventions. These trials primarily provide insights into the safety and efficacy of a treatment within a specific population. They often play a crucial role in early-phase drug development, helping researchers understand the potential benefits and risks of a novel intervention.[2-4]

    The absence of a comparator group, which is the hallmark of SATs, is perhaps the greatest challenge of using these studies in NMAs because this precludes the construction of a connected network of treatments. NMAs depend on direct comparisons between treatments to establish relationships and estimate treatment effects accurately. The absence of head-to-head comparisons from SATs disrupts the seamless flow of evidence within the network. Moreover, the inclusion of SATs can potentially introduce bias and confounding. Without a control group, the observed outcomes in these trials might be influenced by factors beyond the intervention itself, leading to skewed estimates of treatment effects. This bias can jeopardize the validity and reliability of the overall NMA results.[5]

    SATs can also exacerbate heterogeneity, which is an inherent challenge in evidence synthesis through NMAs. The diversity in study design, patient populations, and outcomes measured across these trials can contribute to significant heterogeneity within the NMA, undermining the assumption of treatment effect consistency that NMAs rely upon. The precision and confidence in NMA results can also be compromised when single-arm trials with limited sample sizes are included. Since these trials contribute a smaller amount of data relative to comparative trials, their impact on the overall estimates can be disproportionate. This can lead to wider confidence intervals and reduced statistical power, which in turn affects the reliability of the conclusions drawn from the NMA.[5]

    To address these challenges, several strategies can be employed when considering the inclusion of SATs in NMAs. The incorporation of external evidence, such as historical controls, real-world data, or findings from related studies, can enrich the analysis by providing additional context and comparators, thereby enhancing the understanding of treatment efficacy and safety. Secondly, indirect comparisons can be leveraged, wherein treatments are linked through a common comparator, enabling the assessment of relative effects even in the absence of direct trial comparisons. Furthermore, the inclusion of matching controls entails identifying studies with analogous participant characteristics or baseline factors to the SAT, effectively generating a pseudo-comparator group that aligns with the trial’s attributes, thus mitigating potential bias. Moreover, the development of synthetic control groups by amalgamating data from multiple SATs that share comparable patient profiles and outcomes can be facilitated through statistical methods like propensity score matching or Bayesian approaches, enhancing the robustness of the analysis.[6-8]

    Thus, SATs can offer valuable information for conducting NMAs, but their inclusion in the NMA is associated with some genuine challenges. By implementing strategies such as sensitivity analyses, cautious interpretation, and judicious inclusion, researchers can navigate these challenges and make well-informed decisions about including SATs in NMAs.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Cipriani A, Higgins JP, Geddes JR, Salanti G. Conceptual and technical challenges in network meta-analysis. Annals of internal medicine. 2013 Jul 16;159(2):130-7.
    2. Evans SR. Clinical trial structures. J Exp Stroke Transl Med. 2010 Feb 9;3(1):8-18.
    3. Wang Z, Lin L, Murray T, Hodges JS, Chu H. Bridging randomized controlled trials and single-arm trials using commensurate priors in arm-based network meta-analysis. Ann Appl Stat. 2021 Dec;15(4):1767-1787.
    4. Schmitz S, Maguire Á, Morris J, et al. The use of single armed observational data to closing the gap in otherwise disconnected evidence networks: a network meta-analysis in multiple myeloma. BMC Med Res Methodol. 2018 Jun 28;18(1):66.
    5. Sachdev M. Using Single Arm Trials for HTA Submissions: Dr. Heeg and Maria Rizzo.
    6. Seeger JD, Davis KJ, Iannacone MR, et al. Methods for external control groups for single arm trials or long-term uncontrolled extensions to randomized clinical trials. Pharmacoepidemiol Drug Saf. 2020 Nov;29(11):1382-1392.
    7. Burcu M, Dreyer NA, Franklin JM, et al. Real-world evidence to support regulatory decision-making for medicines: Considerations for external control arms. Pharmacoepidemiol Drug Saf. 2020 Oct;29(10):1228-1235.
    8. Zhang J, Ko CW, Nie L, Chen Y, Tiwari R. Bayesian hierarchical methods for meta-analysis combining randomized-controlled and single-arm studies. Statistical methods in medical research. 2019 May;28(5):1293-310.

    MarksMan Healthcare

    , , ,

  • Elevating Evidence Synthesis: Unveiling the Research Integrity Assessment (RIA) Tool

    Elevating Evidence Synthesis: Unveiling the Research Integrity Assessment (RIA) Tool
    Research Integrity Assessment (RIA) Tool

    The bedrock of evidence-based decision-making rests on the integrity of systematic reviews, which rely on the credibility of the studies they encompass. While bias assessment and risk of bias evaluation are crucial steps in ensuring study quality, the need to go beyond these measures has become increasingly evident. Enter the Research Integrity Assessment (RIA) tool, a groundbreaking approach that aims to safeguard the authenticity and reliability of studies included in evidence synthesis.[1-4]

    The RIA tool is not a mere supplement to the traditional “Risk of Bias” assessment; it is a distinct and innovative framework designed to establish the integrity and authenticity of studies. RIA meticulously scrutinizes various aspects of study conduct, ranging from retraction notices and prospective trial registration to ethics approval, authorship, and the plausibility of methods and results. By doing so, RIA addresses concerns related to scientific misconduct, poor research practices, and potential biases that may distort evidence synthesis findings.[1, 5]

    Timing is crucial when implementing RIA within evidence synthesis. RIA is best employed early in the review process, particularly for randomized controlled trials (RCTs) that have passed the initial PICO (participants, intervention, comparator, and outcomes) eligibility screening. This proactive approach allows for the early exclusion of problematic RCTs, ensuring the integrity of the entire study pool and all subsequent analyses.[5]

    The workflow of RIA involves a hierarchical assessment through the six domains: Retracted studies, absence of prospective registration, inadequate ethical approval without informed written consent, discrepancies within the author group and study location, insufficient randomization, and implausible study results should result in the exclusion of an RCT. Should any concerns arise within any domain, the study is categorized as “awaiting classification,” prompting further scrutiny. If no concerns persist across all domains, or if issues are adequately addressed through correspondence with study authors, the RCT meets the criteria for inclusion in the review and can proceed to the next stages. In the context of living systematic reviews, both included RCTs and those labeled as “awaiting classification” need to be re-evaluated for potential retraction notices., culminating in a decision regarding a study’s eligibility.[5]

    For the RIA assessment, a collaborative and thorough approach is essential. Each study should be independently evaluated by two review authors, and any discrepancies should be resolved through discussions. A diverse team of researchers with expertise in clinical trial design, systematic review methodology, and clinical content should carry out the RIA assessment.[5]

    A hallmark of the RIA tool is its emphasis on transparency and documentation. An Excel-based format of the RIA tool contains critical signaling questions and columns for summarizing conclusions for each domain. The resulting table not only justifies the decision on research integrity and eligibility but also offers an accessible way to document the review authors’ assessments and judgments. This information should be made readily available through publication as a supplement to the systematic review or through online repositories.[5]

    The RIA tool represents a significant step towards a standardized approach for identifying and managing problematic studies within evidence synthesis. While the tool’s development was prompted by the challenges posed by the COVID-19 pandemic, its potential extends far beyond this context. As the systematic review landscape continues to evolve, the iterative refinement and validation of RIA offer a promising avenue for enhancing the credibility, reliability, and ethical soundness of evidence synthesis. To our collective commitment, the RIA tool stands as a testament, demonstrating our dedication to maintaining research integrity and furthering the quest for impartial knowledge, all for the advancement of society’s well-being.[5]

    In the ever-evolving landscape of evidence synthesis, the Research Integrity Assessment (RIA) tool emerges as a beacon of hope. By providing a systematic and comprehensive framework to assess research integrity, RIA adds a crucial layer of protection against distorted findings and compromised recommendations.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Ioannidis JPA. Hundreds of thousands of zombie randomised trials circulate among us. Anaesthesia. 2021 Apr;76(4):444-447.
    2. Soares-Weiser K, Lasserson T, Jorgensen KJ, et al. Policy makers must act on incomplete evidence in responding to COVID-19. Cochrane Database Syst Rev. 2020 Nov 20;11(11):ED000149.
    3. Avenell A, Stewart F, Grey A, Gamble G, Bolland M. An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature. BMJ Open. 2019 Oct 30;9(10):e031909.
    4. Higgins JP, Altman DG, Gøtzsche PC, et al. Cochrane Bias Methods Group; Cochrane Statistical Methods Group. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011 Oct 18;343:d5928.
    5. Weibel S, Popp M, Reis S, et al. Identifying and managing problematic trials: A research integrity assessment tool for randomized controlled trials in evidence synthesis. Res Synth Methods. 2023 May;14(3):357-369.

    MarksMan Healthcare

    , , ,

  • Matching-Adjusted Indirect Comparisons (MAICs): What, Why, and How?

    Matching-Adjusted Indirect Comparisons (MAICs): What, Why, and How?

    Meta-analysis is crucial in evidence-based medicine as it combines data from multiple studies for more precise treatment effect estimates. However, when head-to-head clinical trials directly comparing treatments are scarce, Indirect Treatment Comparisons (ITC) become valuable by offering insights through common comparators. The conventional approaches to ITCs hinge on aggregate data, assuming a uniform distribution of effect-modifying variables across trials. The emergence of the Matching-Adjusted Indirect Comparison (MAIC) methodology, which challenges these assumptions, is gaining momentum, particularly in submissions to reimbursement organizations.[1-3]

    MAICs are an extension of the traditional ITC method, developed with the aim of addressing some of the limitations of traditional ITCs, particularly the issue of confounding by patient characteristics. MAICs attempt to make the compared treatment groups more comparable by adjusting for patient-level characteristics that may influence treatment outcomes. MAICs offer a unique vantage point within Health Technology Assessment (HTA) submissions, amalgamating unadjusted ITC outcomes, even when relative treatment efficacy appears modest. This method aims to minimize bias, facilitating a fair and nuanced comparison of therapies akin to real-world scenarios.[4-6]

    MAICs are grounded in individual-level patient data (IPD) from an intervention trial (e.g., manufacturer’s product) and published aggregate data from the comparator’s trial, and seek equilibrium by reweighting IPD patient characteristics. Techniques such as propensity scores derived from moment methods or entropy balancing play a pivotal role in this equilibrium, ensuring the reweighted IPD outcomes are juxtaposed against published aggregate data to discern relative impact.[7]

    MAICs predominantly operate within an “anchored” framework, often relying on a shared comparator (e.g., placebo) to ground comparisons. This approach, common in connected networks that account for randomization, shields estimations from the sway of imbalanced prognostic factors. Nonetheless, empirical evidence or clinical insight must substantiate effect modification claims. Conversely, the “unanchored” MAIC takes center stage in disconnected networks lacking a common comparator, directly juxtaposing reweighted IPD outcomes and published aggregate data. Rigorous estimates of absolute effects and vigilant control of prognostic and effect-modifying factors are prerequisites for unanchored comparisons, while lurking unobserved confounding remains challenging due to a lack of randomization. Fundamentally, anchored MAICs illuminate treatment impact, whereas unanchored variants scrutinize outcomes across trials.[6,7]

    MAICs often have a lower risk of confounding because of the matching of patients based on key characteristics; for the same reason, potential bias from differences between the treatment groups in the original trials is also lower with MAICs. Further, since MAIC creates a more balanced comparison by aligning patient characteristics, treatment estimates are often more robust and reliable than conventional ITCs. However, MAICs also have certain limitations pertaining to the availability of suitable IPD, the potential of selection bias of patient data, quality and completeness of IPD, and challenges related to assumptions and extrapolations. While MAIC employs individual-level patient data (IPD) to mitigate observed differences, unobserved disparities can lead to residual confounding. Even when placebo-arm outcomes are balanced, unobserved factors affecting treatment outcomes but not placebo outcomes can bias comparisons. Practical challenges include the need for matched outcome definitions and inclusion/exclusion criteria and the inability to fit or calibrate propensity score models using aggregate data. Balancing multiple baseline factors relies on an adequate number of patients with IPD, which can reduce the adequate sample size. MAIC may be utilized for single-arm trials, but the absence of a common comparator limits validation. Irreconcilable differences in trial design or patient characteristics might exclude trials from analysis, necessitating a trade-off between evidence inclusion and reducing heterogeneity. Sensitivity analyses are crucial to assessing the impact of trial inclusion/exclusion on results.[8,9]

    In a landscape where clinical decision-making hinges on robust evidence, MAIC is a valuable tool, offering unique perspectives and cautionary lessons. As researchers, practitioners, and evaluators continue to explore the horizons of evidence synthesis, the pursuit of accuracy, transparency, and informed choices remains paramount. By embracing the insights and addressing the limitations of MAIC, we inch closer to a comprehensive understanding of treatment landscapes and forge a path toward more informed and patient-centered healthcare decisions.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Ahn E, Kang H. Introduction to systematic review and meta-analysis. Korean J Anesthesiol. 2018 Apr;71(2):103-112.
    2. Jansen JP, Fleurence R, Devine B, et al. Interpreting indirect treatment comparisons and network meta-analysis for health-care decision making: report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: part 1. Value Health. 2011 Jun;14(4):417-28.
    3. Veroniki AA, Straus SE, Soobiah C, et al. A scoping review of indirect comparison methods and applications using individual patient data. BMC Med Res Methodol. 2016 Apr 27;16:47.
    4. Phillippo DM, Ades AE, Dias S, et al. Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal. Med Decis Making. 2018 Feb;38(2):200-211.
    5. Phillippo DM, Dias S, Elsada A, et al. Population Adjustment Methods for Indirect Comparisons: A Review of National Institute for Health and Care Excellence Technology Appraisals. Int J Technol Assess Health Care. 2019 Jan;35(3):221-228.
    6. Thom H, Jugl SM, Palaka E, Jawla S. Matching adjusted indirect comparisons to assess comparative effectiveness of therapies: usage in scientific literature and health technology appraisals. Value in Health. 2016 May 1;19(3):A100-1.
    7. Petto H, Kadziola Z, Brnabic A, et al. Alternative Weighting Approaches for Anchored Matching-Adjusted Indirect Comparisons via a Common Comparator. Value Health. 2019 Jan;22(1):85-91.
    8. Signorovitch JE, Sikirica V, Erder MH, et al. Matching-adjusted indirect comparisons: a new tool for timely comparative effectiveness research. Value in Health. 2012 Sep 1;15(6):940-7.
    9. Jiang Y, Ni W. Performance of unanchored matching-adjusted indirect comparison (MAIC) for the evidence synthesis of single-arm trials with time-to-event outcomes. BMC Med Res Methodol. 2020 Sep 29;20(1):241.

    MarksMan Healthcare

    , , , ,

  • Collaborative Approach in Conducting Systematic Literature Reviews For Evidence Synthesis

    Collaborative Approach in Conducting Systematic Literature Reviews For Evidence Synthesis

    A well-conducted systematic review and meta-analysis can be invaluable to help clinicians stay up-to-date on current evidence-based medicine.(1) However, systematic reviews and meta-analyses often tend to be highly focussed on a specific research question, and as a result not broad enough to be equally useful for all stakeholders, especially in topics of broad public health importance with multiple facets involved in policy-level decision-making process. Furthermore, systematic reviews are frequently conducted by small teams of researchers, usually from a single or few institutions; while this can ensure quicker completion of the research, research resulting from a smaller team can suffer from disadvantages such as lack of diversity, limited expertise of team members, higher risk of bias, subjectivity and methodological errors, and an overall lack of generalizability.(2)

    A coordinated systematic review model called the “collaborative review model” proposed by Hayden et al can address the challenges posed by the process taken up for conducting conventional systematic reviews. The collaborative review model is a relatively new approach developed for areas with significant research material on a specific health condition.(3) This approach intends to divide a single systematic review topic into focused sub-reviews using homogeneous methods and tools and by sharing data among the team members. Collaborative input on method decisions is supported by comprehensive guidance documents shared across the network and multifaceted strategies for effective communication. Collaboration is supported by a well-defined project management structure, efficient communication strategies, and the collective harnessing of resources and skills.(3)

    The collaborative review model enables team coordination and collaboration, frequent expert discussions, coordinated literature searching across a broader topic, and consistency in data handling and analytic methods. The division of large reviews into smaller, focused sub-reviews allows for increased efficiency and faster completion of reviews. By involving multiple reviewers, these reviews can minimize the risk of bias and enhance the reliability of findings. Moreover, with the use of advanced comparative and multivariable analyses, including network meta-analyses, collaborative reviews provide a comprehensive understanding of treatment effects. These analyses can offer valuable insights into treatment options and comparative effectiveness.(3)

    The collaborative review approach ensures a more thorough and accurate assessment of the evidence by incorporating standardized data collection forms and consistent data handling to address discrepancies. Through task coordination and resource sharing, collaborative review approach optimizes the allocation of research resources and enhances the overall efficiency of evidence synthesis. By establishing standardized protocols, guidelines, and workflows, this approach ensures methodological consistency across review teams. Furthermore, collaborative reviews bring together the expertise of large international collaborators, promoting capacity-building and mentorship opportunities for new reviewers.(3)

    This collaborative review approach is not without challenges. The involvement of a large team requires steeper funding requirements for a well-coordinated review to handle large teams, systematic review tools, and good project management. Next, given the huge number of contributors involved in such a review model, maintaining a transparent process for authorship and acknowledgment of the multiple outputs poses another significant challenge.(3)

    The collaborative review model has the potential to address many barriers to getting evidence into policy by drawing on the strengths of pre-existing different approaches to evidence synthesis. The effectiveness of the collaborative review model with enhanced quality control measures provides a standardized approach to collating and summarising large volumes of evidence for policy makers for any policy topic area.

    Become A Certified HEOR Professional – Enrol yourself here!

    References:

    1. Tawfik GM, Dila KA, Mohamed MY, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Tropical medicine and health. 2019 Dec;47(1):1-9.
    2. Créquit P, Trinquart L, Yavchitz A, Ravaud P. Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer. BMC medicine. 2016 Dec;14(1):1-5.
    3. Hayden JA, Hayden JA, Ogilvie R, et al. Commentary: collaborative systematic review may produce and share high-quality, comparative evidence more efficiently. Journal of clinical epidemiology. 2022 Sep 28.

    MarksMan Healthcare

    , ,