Contact Us

Home

About us

Services

AI Excellence

case studies

publications

HEOR Certificationheor Certification

AI for Medical Writing

Contact

  • An Overview of Evidence Maps: a New Approach for Evidence Review Process

    An Overview of Evidence Maps: a New Approach for Evidence Review Process

    Systematic reviews (SRs) are the embodiment of the evidence-based approaches, and they have reformed clinical decision-making in almost all therapy areas. The approach of SRs was essentially developed to fulfil the need of the medical practitioners to obtain precise and consistent information about the efficacy and safety of a clinical intervention, diagnostic procedure, or a prognostic marker from a pool of evidence, which is apparently full of contradiction, heterogeneity and bias.(1) Although SRs and meta-analyses are robust and detail-oriented, they’re both resource intense, with a limited scope of outcomes.(2) In order to cater to a range of needs from the stakeholders, the SR approach has branched within the realm of evidence synthesis. For instance, rapid reviews provide for more urgent deadlines but may not follow all the methods of an SR,(3) scoping reviews include larger bodies of evidence, not requiring a detailed synthesis,(4) and realist reviews focus on the assessment of the functions of complex interventions, often comprising of evidence excluded from classic SRs.(5, 6)

    Given the resource intense nature of the SRs, it is essential to recognize the most informative research questions in order to maximize their value and efficiency in clinical and regulatory decision-making. It can be inefficient to invest resources in SRs barely as a means of addressing specific research questions, if data available to answer those questions is lacking. Therefore, decision-makers need to monitor and understand the evidence base as a whole, so as to quickly determine the emerging trends or issues of potential concern. This can, in turn, facilitate the development of proactive research questions by relevant stakeholders for SRs to answer.(1)

    Evidence mapping is a new approach for the evidence review process. This approach can potentially expedite evidence surveillance in a clear and reproducible manner, thus offering a broader understanding of the existing evidence base through interactive yields.(1) Evidence maps and evidence visualizations are systematic evidence synthesis approaches, which work by displaying visually the gaps in evidence or study characteristics, and, at times, summarize study quality or synthesized evidence from multiple studies. Such an interactive and visual representation provides a quick overview of the existing evidence base, thereby helping stakeholders and researchers to immediately understand research priorities.(7) For these reasons, evidence maps are excellent tools that help in guiding clinical investigators to set the agenda for future research.(8)

    Being a rather new concept, there has been no uniform definition of, or methodology for conducting, evidence maps yet. Mainly, evidence maps are referred to as tools of systematic organisation and illustration of evidence base with the intent to characterize the breadth, depth and methodology of relevant evidence, identifying gaps.(9) Another definition of evidence map is “an approach to providing a visual representation and critical assessment of evidence landscape for a particular topic or question”.(10) A more recent definition is developed from the published evidence maps, which turned out to be a systematic search of a broad field identifying gaps in knowledge and the needs for future research.(6) The last one thus takes evidence maps to be a user-friendly representation of evidence bases visually in a figure or graph, a table or a searchable database.(8)

    Nonetheless, due to the lack of a uniform definition, the stakeholders may not essentially know what to expect while warranting an evidence map or identifying existing maps. Moreover, lack of a repository for evidence maps makes them difficult to locate, thus making it less likely for authors to develop existing approaches further.(11)

    Essentially, evidence maps are primarily prepared by the relevant stakeholders (researchers, policy-makers, funders, and, most importantly, patients) by identifying the most important clinical questions to their context, and researching on the body of evidence that is available already. Next, the quality of the available evidence is assessed and conveyed to stakeholders. The final step includes the visual depiction of the most relevant data elements to the stakeholder; for e.g., focusing on the size of the body of evidence, comparisons made versus those avoided, populations studied versus those avoided, and risk of bias, among other factors.(8) At the end of this process, the gaps in the available evidence in the context of the original research question starts to become apparent, and this can be used to plan further research activities.

    In conclusion, evidence maps offer a robust and transparent methodological framework with which to assess the evidence landscape in a detailed manner, and aid clinical and regulatory decision-making. The broad scope of evidence maps, through efficient use of resources, can substantially streamline evidence synthesis by preventing unnecessary duplication of work. Additionally, future text mining and machine learning advancements will further possibly reduce the resource intensity of the methodology.(8)

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Wolffe TAM, Whaley P, Halsall C, et al. Systematic evidence maps as a novel tool to support evidence-based decision-making in chemicals policy and risk management. Environ Int. 2019; 130:104871.
    2. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010; 7(9):e1000326.
    3. Tricco AC, Antony J, Zarin W, et al. A scoping review of rapid review methods. BMC Med. 2015; 13(1):224.
    4. Colquhoun HL, Levac D, O’Brien KK, et al. Scoping reviews: time for clarity in definition, methods, and reporting. J Clin Epidemiol. 2014; 67(12):1291–4.
    5. Pawson R, Greenhalgh T, Harvey G, et al. Realist review–a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005; 10 suppl 1:21–34.
    6. Miake-Lye IM, Hempel S, Shanman R, et al. What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products. Syst Rev. 2016; 5:28.
    7. Evidence Maps and Evidence Visualizations. Patient-centered Outcomes Research Institute. Available at: https://www.pcori.org/impact/evidence-maps-and-evidence-visualizations
    8. Alahdab F, Murad MH. BMJ Evidence-Based Medicine. 2019.
    9. Katz DL, Williams AL, Girard C, et al. The evidence base for complementary and alternative medicine: methods of Evidence Mapping with application to CAM. Altern Ther Health Med 2003; 9:22–30.
    10. Bethan C, O’Leary PW, Kaiser MJ, et al. Evidence maps and evidence gaps: evidence review mapping as a method for collating and appraising evidence reviews to inform research and policy. Environmental Evidence 2017; 6.
    11. Evidence and gap maps: A comparison of different approaches. Oslo, Norway: The Campbell Collaboration. Retrieved from: campbellcollaboration.org/ DOI: https://doi.org/10.4073/cmdp.2018.2

    MarksMan Healthcare

    , ,

  • Additional Search Strategies to Identify Unpublished Research in Systematic Reviews

    Additional Search Strategies to Identify Unpublished Research in Systematic Reviews

    Systematic reviews (SRs) face a major challenge while identifying all the relevant research, including randomised controlled trials (RCTs) irrespective of their publication status.(1) Unpublished, selectively reported, or non-reported research can lead to poorer quality clinical trials, thus leading to suboptimal care delivery. It also overlooks the opportunities for potential scientific progress.(2) Such incomplete publication status is a significant problem for SRs, which aim to present a comprehensive and appropriate evidence pool.

    SR search strategies are typically designed to find all relevant evidence that answers a particular research question(3) and reduce bias.(4) However, it can get difficult to capture all the eligible studies for an SR through a bibliographic database search. This may be because of poor study indexing in databases, failure of constructing a comprehensive search strategy which includes all relevant terms, or studies getting buried in grey literature. Also, it can be difficult to access studies within a trial to look for unpublished findings. Unpublished studies are particularly often unidentifiable, and if excluded, can lead to incorrect estimation of effects.(5) Further contributing to bias in SR findings are the facts that findings of half of the RCTs never get published, and also, the publication status is often influenced by the nature and direction of the results.(6)

    Several initiatives can be undertaken to reduce publication bias. One of them is the trial registration policy initiated by the International Committee of Medical Journal Editors (ICMJE) in 2005,(7) followed by the mandate by the US Food and Drug Administration Amendments Act to post the study results at ClinicalTrials.gov no later than a year after the date of final data collection for the primary outcome, for all phase II to IV trials of drugs, devices, and biological treatments.(8,9) A noticeable increase was observed in the trial registration after these policies were implemented.(6,10) Consequently, the search of trial registries while conducting SRs is considered vital.(11)

    Another way of finding unpublished research is directly contacting trial units (CTUs) and investigators. Evidence from literature suggests that contacting investigators can prove beneficial in finding published and unpublished eligible studies, which are otherwise difficult to find for intervention SRs.(3,4) Brueton et al(12) in a survey of CTUs from the UK, communicated with colleagues working in RCT methodology, while also conducting standard searches of bibliographic databases, conference abstracts and reference list to look for additional studies for a Cochrane methodology SR of strategies to improve retention in RCTs. In this study, the authors found that principal investigators were willing to contribute results from unpublished RCTs, thus majorly contributing to the overall study findings.

    Personal communication has been used in the past while conducting methodology reviews to identify unpublished research. Evidence from literature even suggests that email communication is the best method of communication in such situations.(4) Furthermore, researchers can also consider exploring the Studies within a Trial (SWAT) database of evaluations of methods for RCTs to identify ongoing embedded methodology RCTs for future SRs of research methodology.(13)

    To conclude, for identifying unpublished research, as well as to reduce bias, it is essential for researchers to look for other sources than those routinely checked by default. Contacting trial units, and direct communication with the study investigators, are some of these sources that can help find important results that can help in critical decision-making, such as finding data on possible adverse outcomes.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Baudard M, Yavchitz A, Ravaud P, et al. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ. 2017; 356:j448.
    2. Knottnerus JA, Tugwell P. The potential impact of unpublished results. J Clin Epidemiol. 2013; 66(10):1061-3.
    3. Lefebvre C, Manheimer E, Glanville J. Searching for Studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. England: Wiley; 2008. Chapter 6 p. 95–150.
    4. Young T, Hopewell S. Methods for obtaining unpublished data. Cochrane Database of Syst Rev. 2011(11).
    5. Sterne J, Egger M, Moher D. Addressing reporting bias. In: Higgins JPT, Green S, editors. Cochrane Handbook for systematic reviews of Interventions. England: Wiley; 2008. Chapter 10 p. 297–333.
    6. Schmucker C, Schell LK, Portalupi S, et al. Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries. PLoS One. 2014; 9(12):e114023.
    7. De Angelis C, Drazen JM, Frizelle FA, et al. International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Ann Intern Med.2004; 141:477-8.
    8. United States Congress. (2007) Food and Drug Administration Amendments Act (FDAAA) of 2007: public law no 110-85. gpo.gov/fdsys/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf
    9. Groves T. Mandatory disclosure of trial results for drugs and devices. 2008; 336:170.
    10. Laine C, Horton R, DeAngelis CD, et al. Clinical trial registration: looking back and moving ahead. 2007; 298:93-4.
    11. Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. 2009; 339:b2535.
    12. Brueton V, Tierney JF, Stenning S, Rait G. Identifying additional studies for a systematic review of retention strategies in randomised controlled trials: making contact with trials units and trial methodologists. Syst Rev. 2017; 6(1):167.
    13. Education section – Studies Within A Trial (SWAT). Journal of Evidence-Based Medicine. 2012; 5(1):44–5.

    MarksMan Healthcare

    ,

  • The Impact Of PRISMA Statement On Systematic Review Publications

    The Impact Of PRISMA Statement On Systematic Review Publications

    Systematic reviews (SRs) are crucial in health and scientific research as they offer a thorough understanding of the findings from research. (1) They provide critical information on different aspects of research, such as answers to the questions not often addressed by individual studies, problems in primary research that need to be avoided in future studies, and theories on how’s and why’s. Therefore, they create a knowledge base for different types of stakeholders, viz. healthcare providers, patients, fellow researchers, as well as regulators and policymakers. (2)

    An SR is valuable only when it is prepared with transparent, comprehensive, and precise objectives. If the information presented in SRs is ambiguous, the readers may not interpret or reproduce the findings precisely. Additionally, it may also lead to the failure to implement the SR findings into clinical practice. (3) Adherence to clear, up-to-date reporting guidelines enables authors to achieve reliable, good-quality SRs. (4,5)

    The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement is a reporting guideline published in 2009, developed to curb poor reporting of SRs. (6) This statement comprises a checklist of 27 items recommended for reporting in SRs, as well as an “explanation and elaboration” paper that provides additional regulations for an individual item, along with examples of exemplar reporting. (7) As observed in 2017, the PRISMA 2009 statement reportedly had a very high uptake from the biomedical research community. However, not all published SRs were observed to cite the guideline. (8) As of 2020, the endorsement and adoption of the statement were more expansive, as observed through its co-publication in several journals, citation in over 60,000 reports, (9) commendation from nearly 200 journals, SR organizations, and uptake in various fields.(2)

    The PRISMA 2009 statement and extensions certainly helped in facilitating meta-research. Evidence also suggests that adherence to some PRISMA items was better than others. (3) Moreover, findings from observational studies have shown the use of the PRISMA 2009 statement to result in thorough reporting of SRs,(8) although with more scope to improve adherence to the same. (3) A scoping review was conducted to analyze the uptake and impact of the PRISMA 2009 statement, which considered 57 studies assessing adherence to the statement (reporting of SRs in line with PRISMA 2009 guidance).(3) Findings of this review showed, out of 57 studies considered, adherence to the statement was reported in 27 studies. However, lack of transparency was still an issue for many published SRs, which was shown with adherence shown to very few items from the statement among very few SRs.(3)

    In 2020, the PRISMA 2009 statement was updated and replaced by PRISMA 2020 statement. This update can be attributed to the perpetual changes in the health and life sciences research domain, such as technological advancements like natural language processing (NLP) and machine learning (ML) for robust identification of relevant evidence, development of novel methods for assessing the risk of bias, and transformation of publishing landscape, among others.2 The PRISMA 2020 statement provides updated reporting recommendations for SRs that consider advancements in methods to identify, select, and assess studies. This updated statement comprises a checklist of 27 items, an extended checklist detailing guidance for an individual item, the PRISMA 2020 abstract checklist, and amended flow diagrams for original and updated reviews.(2)

    It is believed that the use of PRISMA 2020 will potentially benefit many stakeholders, including authors, editors, peer reviewers for SRs, along with a varied range of SR audiences, including guideline developers, policymakers, healthcare providers, and patients among others. Furthermore, the uptake of the PRISMA 2020 guidance will lead to a more transparent, comprehensive, and precise SR reporting, thus enabling informed and evidence-based decision-making.(2)

    Become A Certified HEOR Professional – Enrol yourself here!

    References  

    1. Peričić T, Tanveer S. Why systematic reviews matter – A brief history, overview and practical guide for authors. Author’s Update. Elsevier. July 2019. Available at: https://www.elsevier.com/connect/authors-update/why-systematic-reviews-matter
    2. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021; 372:n71.
    3. Page MJ, Moher D. Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: a scoping review. Syst Rev 2017; 6:263.
    4. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007; 4:e78.
    5. Page MJ, Moher D. Mass production of systematic reviews and metaanalyses: an exercise in mega-silliness? Milbank Q 2016; 94(3):515-9.
    6. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009; 151:264-9, W64.
    7. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 2009; 62:e1-34.
    8. Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med 2016; 13(5):e1002028.
    9. Scopus Preview. August 2020. Available at: https://www.scopus.com/home.uri?zone=header&origin=

    MarksMan Healthcare

    , , ,

  • Reporting Characteristics Of Systematic Reviews For The UK NIHR HTA programme

    Reporting Characteristics Of Systematic Reviews For The UK NIHR HTA programme

    Systematic reviews (SRs) are incredibly crucial for healthcare decision-making, as they often provide a reliable summary of evidence on the comparison among healthcare interventions. They identify, assess, and combine the results of similar but individual studies and help to clarify the known and unknown benefits and risks associated with drugs, devices, and other healthcare interventions. SRs are helpful for clinicians to incorporate research findings into their daily practices, for patients to make informed choices about their care, and for professional medical organizations to develop clinical practice recommendations. (1)

    Their importance in clinical decision-making has led to the rising number of published SRs. Consequently, the quality of published SRs is also being questioned. Findings of a cross-sectional study by Page et al. (2016) have observed poor quality of conduct and reporting of many SRs, despite the availability of the relevant reporting standards, such as the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement. (2) This is a big issue since poorly conducted and reported SRs often give away misleading conclusions that significantly affect decision-making. (2,3)

    Cochrane SRs, published in the Cochrane Database of Systematic Reviews, are typically considered the ‘gold standard’ (4) and superior to other non-Cochrane SRs, are often contracted by policy-makers, and involve many safeguards against the potentially ‘misleading conclusions’.(3) Health Technology Assessment (HTA) is a specific area where an SR providing evidence on the clinical efficacy of a medical device/technology plays a crucial role in decision-making and would represent a different type of ‘non-Cochrane’ review. One such group of SRs is conducted for the UK National Institute of Health Research (NIHR) HTA programme (HTA-SRs). The full texts of HTA-SRs are published in the programme’s own journal, Health Technology Assessment (Winchester). (5)

    The HTA-SRs show a remarkable comparison with Cochrane reviews as they have clear reporting standards and are not limited by word restrictions or the lack of online appendices, unlike many other non-Cochrane reviews published in conventional peer-reviewed journals. The nature and reporting of HTA-SRs, Cochrane, and non-Cochrane reviews are similar in many aspects but different in several other areas, such as, a more extensive range of included studies, the failure of HTA-SRs to report the total number of participants from included and instead reporting a range, the availability and registration of SR protocols, and a statement specifying adherence to respective guidelines while conducting the review. Whereas, concerning the conduct and reporting of the study selection, data extraction, and critical appraisal processes, the HTA-SRs have been observed to be better reported than non-Cochrane reviews.(3)

    Predominantly, it has been observed that the reporting characteristics of HTA-SRs published in the Health Technology Assessment journal are analogous to Cochrane reviews and better than many other non-Cochrane reviews in several aspects. These include identification as an SR, review registration and protocol availability, conflicts of interest, mentioning the type of literature included, particulars of the strategies for database search, trial registry and grey literature search, the use of PRISMA flow diagrams, providing details of any excluded studies; and the reporting of limitations at the review level and of the included studies. All these factors impact informed decision-making, particularly the one that highlights the importance of sourcing unpublished data, (6,7) and explains uncertainties within the evidence-base and review itself.(3) Apparently, HTA-SRs have weaker reporting than Cochrane reviews across several characteristics, as reported earlier. HTA-SRs are also not expected to develop recommendations based on the quality of the evidence since that responsibility is on the other groups in the HTA process. (8)

    The conduct and reporting of Cochrane reviews and those published in the UK Health Technology Assessment journal are thus of the same standard and usually better than many other non-Cochrane reviews. Accordingly, the HTA-SRs should arguably be considered equivalent to supposed Cochrane ‘gold standard’, and not be clubbed together with other non-Cochrane reviews. They are approved by regulators from the UK Department of Health (National Institute of Health and Care Excellence [NICE] and the NIHR) with a particular policy-making and decision-making objective and audience in mind.(5)  Also, it would not be wrong to say that any systematic reviews conducted for the purpose of HTA should follow the reporting guidelines of the HTA-SRs. The processes for conducting and reporting SRs for HTA have to be transparent, rigorous, and of the highest quality.  Bias in the review needs to be reduced, and the possibility of ‘misleading conclusions’ cannot be allowed in HTA-SRs because there is a lot at stake.(3)

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press.
    2. Page M, Shamseer L, Altman D, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med 2016; 13(5):e1002028.
    3. Carroll C, Kaltenthaler E. Nature and reporting characteristics of UK health technology assessment systematic reviews. BMC Medical Research Methodology 2018; 18:35.
    4. Cochrane library. Available at: https://www.cochranelibrary.com/about/about-cochrane-reviews
    5. National Institute for Health Research (NIHR): Health Technology Assessment Programme. Available at: https://www.nihr.ac.uk/funding-and-support/funding-forresearch-
    6. studies/funding-programmes/health-technology-assessment/.
    7. Jones C, Keil L, Weaver M, et al. Clinical trials registries are underutilized in the conduct of systematic reviews: a cross-sectional analysis. Syst Rev 2014; 3:126.
    8. Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ 2012; 344:d7202.
    9. Rotstein D, Laupacis A. Differences between systematic reviews and health technology assessments: a trade-off between the ideals of scientific rigor and the realities of policy making. Int J Technol Assess Health Care 2004; 20:177–83.

    MarksMan Healthcare

    ,

  • The Importance of Grey Literature Search in Systematic Literature Reviews

    The Importance of Grey Literature Search in Systematic Literature Reviews

    During the course of any research, most of the relevant literature pertaining to a research question is retrieved though searching of recognised databases. However, in addition to this, searching of grey literature can add value to the depth of the research by providing information from varied sources. Grey literature search is an important, but often ignored, part of systematic literature review and data synthesis, especially in the medical research.

    Grey Literature: Definition, Types, and Sources

    The Grey Literature International Steering Committee (GLISC) defines grey literature as “Information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body”.[1] Grey literature is often self-published, and the sources of grey literature can include Government agencies, research institutions, organizations, companies, and associations.[2]

    Grey literature can be classified under various categories: [2,3]

    • Regulatory Information: this includes information available from archives of regulatory bodies such as the USFDA, CDSCO, EMA, and NICE. Regulatory information can also be sourced from stakeholder organizations and pharmaceutical companies, and include product information leaflets, white papers, internal documentations, SOPs, and procedure briefs. Other examples include company and industry wide repositories and financially driven investor service websites.
    • Government data: these include information available from government resources, such as notifications, guidelines, gazette notifications, judicial information, patent databases, policy briefs, etc.
    • Unpublished material from clinical trials: these include prospectively registered clinical trial protocols in repositories such as ClinicalTrials.gov in the USA and the Clinical Trial Registry of India (CTRI). Pre-prints which are not ultimately published due to various reasons, unpublished dissertations and theses,
    • Conference proceedings: abstracts, scientific sessions, and other conference proceedings provide a brief snapshot of contemporary research, which might not get published due to various reasons
    • Internet resources: With the increasing presence of social media, newer forms of grey literature have also surfaced, such as blogs, internet forums, wikis, video lectures, lecture slides and lecture notes, educational videos, personal websites, and information posted on the omnipresent social media.

    Importance of Including Grey Literature

    Commercial publishers are guided by interests and priorities, and all information which do not conform to these are often ignored and not published. This unpublished information forms the bulk of grey literature.

    A research which focuses solely on published material has a risk of missing out a comprehensive view of the topic under research. Grey literature provides valuable information about emerging or less popular research areas which are not published. Including a grey literature is also found to be useful in validating the results of a research-based literature search.[4]

    Grey literature bypasses the time-consuming peer-review process. Also, because of a quicker publication, the time delay between research and its formal publication is also bypassed. As a result, the information in grey literature can be more recent and up-to-date than in a formal publication. This is especially important in situations of public health emergency, such as the COVID-19 pandemic.

    There is a growing interest in using grey literature in systematic reviews and meta-analysis.[5] A study conducted by McCauley et al. analysed that 33% of meta‐analysis included some form of grey literature, accounting for 4.5% to 75% of studies in the meta‐analyses, and contributed significantly to the estimates of the intervention effects. The authors concluded that excluding grey literature from meta-analyses can result in a falsely exaggerated estimates of the effectiveness of intervention.[6] In several cases, information, and data disclosed at conference presentations is never published.[7] These observations have led to an opinion that the confounding effect of publication bias (where negative findings are more often not published) can be mitigated to a significant extent by including grey literature.[5-7]

    Challenges in Grey Literature Searching

    It is certain that grey literature search adds value to a research; however, it is also not possible to ascertain whether such a grey literature search has been done comprehensively or not. This is because grey literature is not well organized. In other words, there is no way to ‘define’ a proper grey literature search. It might be possible that a researcher has done grey literature search and included only data that is favourable, while excluding unfavourable data. This is in stark contrast to the published literature, which is often found to be well-organized. Thus, while it is possible to duplicate a database search strategy to verify if the search has been performed properly or not, such a luxury is not available with the quite unorganized grey literature. This is also the reason why inclusion of grey literature is more often than not entirely dependent on the choice of a researcher. Further, grey literature is not ensured to be peer-reviewed, which brings in an inherent bias.

    Conclusion

    The importance of transparency in research findings cannot be over emphasized. Publication bias still continues to be a huge problem in medical research. Grey literature search has a unique potential to improve transparency in medical research as well as offer a solution for publication bias, by including the unpublished information, thereby improving the comprehensiveness of research. With the increased ease of access of the internet, the access to grey literature has also become easier. Considering the amount of new information that grey literature can bring to research, all researchers must consider including grey literature search in their works. Efforts are required to organize the grey literature so that the credibility and validity of grey literature search can improve.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Grey Literature International Steering Committee. Guidelines for the production of scientific and technical reports: how to write and distribute grey literature. Available from: http://eprints.rclis.org/7469/1/nancy.pdf. Accessed on Jun 20th 2020
    2. Royal Roads University. Grey literature: what is it?: What is grey literature. Available from: https://libguides.royalroads.ca/greylit/what. Accessed on Aug 12th
    3. Citrome L. Beyond PubMed: Searching the “Grey Literature” for Clinical Trial Results. Innov Clin Neurosci. 2014 Jul;11(7-8):42-6.
    4. Benzies KM, Premji S, Hayden KA, et al. State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs. 2006;3(2):55-61.
    5. Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods. 2014;5(3):221-34.
    6. McAuley L, Pham B, Tugwell P, et al. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000;356(9237):1228-31.
    7. Hopewell S, McDonald S, Clarke M, et al. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007(2):MR000010.

    MarksMan Healthcare

    , , ,

  • Text Mining for Search Term Development Aiding in Conduct of Better SLRs

    Text Mining for Search Term Development Aiding in Conduct of Better SLRs

    Systematic literature reviews (SLRs) are widely used to pool and present the findings from multiple studies in a dependable way and are often used to inform policy and practice guidelines. (1) An important SLR feature is the application of scientific tools to find and curtail bias as well as error in the selection and treatment of studies. (2) However, the increasing number of published studies together with the rate of their publication makes it even more complicated and time-consuming to identify relevant studies in an unbiased way. (3)

    To reduce the impact of publication bias, reviewers usually try identifying all relevant research to include it in SLRs. This is challenging and laborious, but the challenge is growing due to increasing databases to search as well as the number of papers and journals being published. Furthermore, evidence suggests the existence of an inherent North American bias in several major bibliographic databases (e.g. PubMed). Therefore, a range of other smaller databases needs to be looked into to identify research for reviews aiming at maximising external validity. (4) This then requires a multi-layered approach to searching through extensive Boolean searches from electronic bibliographic databases and specialised registers and websites. (5)

    Unfortunately, sensitive electronic searches of bibliographic databases show low specificity. Consequently, reviewers often end up manually looking through many thousands of irrelevant titles and abstracts for identifying the much smaller number of relevant ones; which is known as ‘screening’. (6) Roughly, an experienced reviewer can take between 30 seconds and a few minutes to evaluate a citation, which is why 10,000 citations involved in the screening process is considerable. (7) On the other hand, reviews for informed policy and practice must be completed within timetables (often short) and limited budgets; also, this review must be comprehensive in order to be an accurate reflection of the state of knowledge in a given area.(5)

    Text mining has been suggested as a prospective solution to these practical issues, as automating some of the screening process can prove to be time-saving. (5)  Text mining is defined as, ‘the process of discovering knowledge and structure from unstructured data (i.e., text)’. (8,9) There are two particularly promising ways in which text mining can be used to support the screening in SLRs, viz. i) by prioritising the list of items for manual screening to include most likely to be relevant studies can be included at the top of the list; and ii) by manually assigning include/exclude categories of studies for further application of such categorisations automatically. (10) The prioritisation of relevant items may not lessen the workload, but identifying most of the relevant studies first can enable other members of the team to proceed with the next stages of the review, whilst the rest of the irrelevant citations are screened by others. This reduces the turnaround time, even if the total workload may not really reduce.(5)

    The benefits of text mining in case of SLRs cannot be denied when it comes to developing database search strings for topics described by diverse terminology. Stansfield et al. have recently suggested five ways in which the text mining tools can aid in developing the search strategy: (11,12)

    • Improving the precision of searches – Framing more precise phrases instead of single-word terms
    • Identifying search terms to improve search sensitivity – Using additional search terms
    • Aiding the translation of search strategies across databases
    • Searching and screening within an integrated system
    • Developing objectively derived search strategies

    The utility of these tools depends on their different competencies, the way they are used, and the text analysed. (11)

    Moreover, Li et al. have recently proposed a text mining framework to reduce the abstract screening burden as well as to provide high-level information summary while conducting SLRs. This framework includes three self-defined semantics-based ranking metrics with keyword, indexed-term and topic relevance. This framework has been reported to reduce the labour of SLRs to a large degree, while keeping comparably higher recall. (13)

    An array of different issues concerning text mining makes it difficult to identify a single, most effective approach for its use in SLRs. There are, however, key messages/toolsets for applying text mining to the SLR context. Future research in this area should aim at addressing the duplication of evaluations as well as the feasibility of the toolsets for use across a range of subject-matter areas.(5)

    Become a Certified HEOR Professional – Enrol yourself here!

    References 

    1. Gough D, Oliver S, Thomas J. An Introduction to Systematic Reviews. London: Sage; 2012.
    2. Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev 2012; 1(28).
    3. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 2010; 7(9).
    4. Gomersall A, Cooper C. Joint Colloquium of the Cochrane and Campbell Collaborations. Keystone, Colorado: The Campbell Collaboration; 2010. Database selection bias and its effect on systematic reviews: a United Kingdom perspective.
    5. O’Mara-Eves A, Thomas J, McNaught J, et al. Using text mining for study identification in systematic reviews: a systematic review of current approaches Syst Rev 2015; 4(1):5.
    6. Lefebvre C, Manheimer E, Glanville J. Searching for studies (chapter 6) In: Higgins J, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 510 [updated March 2011] Oxford: The Cochrane Collaboration; 2011.
    7. Allen I, Olkin I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA 1999; 282(7):634-5.
    8. Ananiadou S, McNaught J. Text Mining for Biology and Biomedicine. Boston/London: Artech House; 2006.
    9. Hearst M. Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL 1999): 1999. 1999. Untangling Text Data Mining; pp. 3–10.
    10. Thomas J, McNaught J, Ananiadou S. Applications of text mining within systematic reviews. Res Synth Methods 2011; 2(1):1–14.
    11. Stansfield C, O’Mara-Eves A, Thomas J. Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Res Synth Methods 2017; 8(3):355-365.
    12. Gore G. Text mining for searching and screening the literature. McGill. April, 2019. 
    13. Li D, Wang Z, Wang L, et al. A Text-Mining Framework for Supporting Systematic Reviews. Am J Inf Manag 2016; 1(1):1-9.

    Written by: Ms. Tanvi Laghate

    MarksMan Healthcare

    , , ,

  • How to Increase the Data Extraction Quality of Systematic Literature Review?

    How to Increase the Data Extraction Quality of Systematic Literature Review?

    Systematic literature reviews (SLRs) are the foundation of evidence-based healthcare. Explicit methods need to be implemented while conducting SLRs to minimize bias in order to provide more reliable findings, since reduction of bias may affect all steps of the review process. For instance, bias can occur while identifying/screening studies, selecting studies (e.g. due to unclear inclusion criteria), during data extraction process and also, during the validity assessment of included studies. (1,2)

    Data extraction or data collection is a critical step while carrying out SLRs. The process of data extraction can be defined as extracting any type of data from primary studies into any form of standardized tables.(2) Data extraction is one of the most time-consuming and critical tasks performed in order to validate the results of an SLR. (3) In reality, it typically takes between 2.5 to 6.5 years for a primary study publication to be included and published in a new SLR. (4) Moreover, almost 23 % such studies are out of date within 2 years of the publication of SLRs, because of lack of new evidence that might change the primary results of an SLR. (5)

    Evidence from literature further reports high prevalence of extraction errors, which may have only moderate impact on the results of an SLR.(2,6,7)  However, increasing data extraction errors indicate the importance of measures for quality assurance of data extraction in order to minimize the risk of biased results and wrong conclusions. (8) Therefore, there is a dire need of identifying ways to improve the quality of data extraction for SLRs.

    One of the first options is the use of two independent reviewers to extract the data, i.e. a process known as ‘double data extraction’. This process has been reported to result in fewer extraction errors; (9) however, it may not be always necessary, thus justifying the process of ‘reduced extraction’. Reduced extraction focuses on identification of critical aspects (e.g. primary outcomes) that form the basis of conclusion instead of emphasizing on the data extraction of lesser important parameters (e.g. patient characteristics, additional outcomes, etc.).2 This is also recommended by the Methodological Expectations of Cochrane Intervention Reviews (MECIR), wherein it is stated that “dual data extraction is particularly important for outcome data, which feed directly into syntheses of the evidence, and hence to the conclusions of the review”. (10) In addition, the Institute of Medicine (IOM) also states that “at minimum, use two or more researchers, working independently, to extract quantitative and other critical data from each study”. (11)

    Moreover, training the reviewer team in data extraction (e.g. using a sample) prior to performing the complete data extraction is essential to harmonize the end results as well as to clear up common misunderstandings, which would particularly reduce interpretation and selection errors as well as time and effort.6 The reduction of time and effort is especially useful in case of rapid reviews, which aim to deliver timely yet systematic results. (12)

    Automated data extraction has been recently proposed in order to reduce errors as well as for timely completion of SLRs. Natural language processing (NLP) is one such method that involves computerized data extraction to include new, previously unfound information by automatically extracting data from different written resources. (13) This process constitutes aspects of concept extraction/entity recognition, and relation extraction/association extraction. The technique of NLP has been used to automate extraction of genomic and clinical information from biomedical literature. However, the concept of automating data extraction process has not been explored completely yet. The techniques like NLP can initially be used to monitor manual data extraction (which is currently performed in duplicate); then to validate the same done by a single reviewer; then become the primary source for data element extraction to be validated by a human; and eventually completely automate data extraction to enable efficient and faster SLRs. (14)

    Having said that, there are no specific, established standards for data extraction, because the actual benefit of a certain extraction method (e.g. independent data extraction) or the specifications of the reviewer team (e.g. expertise) is not well proven. This warrants more comparative studies to further understand the influence of different extraction methods. Particularly, studies exploring the need of training for data extraction are vital owing to the lack of such analysis till date. More efficient utilization of scientific expertise can be achieved with the application of methods requiring less effort without threatening the internal validity. Finally, enhancing the knowledge base would also help in planning effective training strategies for new reviewers and students in the future.(2)

    Become a Certified HEOR Professional – Enrol yourself here!

    References

    1. Felson DT. Bias in meta-analytic research. J Clin Epidemiol 1992; 45(8):88-892. 
    2. Mathes T, Klaßen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: A methodological review. BMC Med Res Methodol 2017; 17(1):152.
    3. Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011.
    4. Elliott J, Turner T, Clavisi O, et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med 2014; 11:e1001603.
    5. Shojania KG, Sampson M, Ansari MT, et al. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 2007; 147(4):224-33.
    6. Haywood KL, Hargreaves J, White R, et al. Reviewing measures of outcome: reliability of data extraction. J Eval Clin Pract 2004; 10:329–337.
    7. Carroll C, Scope A, Kaltenthaler E. A case study of binary outcome data extraction across three systematic reviews of hip arthroplasty: errors and differences of selection. BMC research notes 2013; 6:539.
    8. Gøtzsche PC, Hróbjartsson A, Marić K, et al. Data extraction errors in meta-analyses that use standardized mean differences. JAMA 2007; 298(4):430–437. 
    9. Tendal B, Higgins JP, Juni P, et al. Disagreements in meta-analyses using outcomes measured on continuous or rating scales: observer agreement study. BMJ 2009; 339: b3128. 
    10. Higgins JPT, Lasserson T, Chandler J, et al. Methodological Expectations of Cochrane Intervention Reviews. London: Cochrane; 2016. 
    11. Morton S, Berg A, Levit L, Eden J. Finding what works in health care: standards for systematic reviews. National Academies Press; 2011.
    12. Schünemann HJ, Moja L. Reviews: rapid! Rapid! Rapid! …and systematic. Syst Rev 2015; 4(1):4.
    13. Hearst MA. Untangling text data mining. Proceedings of the 37th annual meeting of the Association for Computational Linguistics. College Park, Maryland: Association for Computational Linguistics; 1999. pp. 3–10. 
    14. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Syst Rev 2015; 4:78.

    Written by: Ms. Tanvi Laghate

    MarksMan Healthcare

    ,

  • Optimsing Database Combinations for Literature Searches in Systematic Reviews

    Optimsing Database Combinations for Literature Searches in Systematic Reviews

    Use of multiple databases together with additional search strategies is often suggested to search relevant references for systematic reviews. (1,2) For instance, the Cochrane Handbook recommends using at least MEDLINE and Cochrane Central as well as EMBASE, when available, to search randomized controlled trials (RCTs). (3) However, using multiple databases can be strenuous and time consuming owing to the database-specific syntax of search strategies and differences of field codes and proximity operators between interfaces. Another difficulty is the different thesaurus terms between databases that may hamper translation. In addition, it is inconvenient for reviewers to screen more and possibly irrelevant titles and abstracts. Last but not least, limited access and subscriptions make the process all the more tedious and challenging. (4)

    In contrast, some studies exist in the literature that investigate the value of using multiple databases for different topics. Some of these studies report no effect on the outcome by searching more than one databases, thus proving just one database to be sufficient. (5,6) While others have reported a single database to be insufficient to retrieve all references for systematic reviews. (7) Majority of articles on this topic base their conclusions on the coverage of databases, (8) while many have failed to identify an acceptable number of databases to be searched. (9) Having said that, the presence of an article in a database does not guarantee it will be found in a search in that database. Therefore, the ideal database or a certain number of databases to be searched for relevant references for a systematic review remains unclear.(4)

    A recent prospective study has done some research in this area with an aim to determine the combination of databases to be searched for systematic reviews to obtain efficient results by means of minimizing the burden for the investigators and not the validity of the research by missing relevant references.(4)  This study recommended the biomedical searches to be performed using a combination of the following four databases, viz. EMBASE, MEDLINE (plus Epub ahead of print), Web of Science Core Collection, and Google Scholar. Use of this combination showed 93% of the systematic reviews to obtain levels of recall to be considered acceptable (> 95%). Unique results from specialized databases that closely match systematic review topics indicated their use whenever applicable, for e.g. PsycINFO for reviews in the fields of behavioural sciences and mental health or CINAHL for reviews on the topics of nursing and/or allied health.(4)

    Similarly, researchers at Erasmus University Medical Center (MC) have developed a methodology for generating comprehensive search strategies. (10) This methodology encompasses all steps of the search process, starting with a question and resulting in thorough search strategies in multiple databases. Researchers believe that this can prove to be a robust method to create high-quality, vigorous searches in multiple databases in a relatively short time frame. (10)

    Another systematic review has also showed that searching Medline alone for systematic reviews of exercise or other unconventional therapies is likely to be inadequate, while additional specialised databases along with checking reference lists and contacting experts can prove to be most effective for including all relevant papers in the review. (11)

    However, skills and experience of the searcher is an important aspect that can play a role in the efficacy of the search strategies being used. (12) Non-structured searches and searches with lower recall may even miss out relevant references. This can be solved with additional efforts like hand/cursory searching, looking up references, and contacting key players, which might add the extra references in the search.(4)

    To sum it all up, there need to be ways to optimize multiple databases and/or combinations in order to include all the relevant references in systematic reviews. Some researchers might suggest a combination of a few databases, depending on the topic/area of research in a particular systematic review. Majority of evidence available in the literature states that searching only one database may prove to be insufficient, thus leading to missing references. In addition, checking reference lists as well as contacting key experts to include all the necessary information and insights is recommended.

    Become a Certified HEOR Professional – Enrol yourself here!

    References

    1. Levay P, Raynor M, Tuvey D. The contributions of MEDLINE, other bibliographic databases and various search techniques to NICE public health guidance. Evid Based Libr Inf Pract 2015; 10:50–68.
    2. Beyer FR, Wright K. Can we prioritise which databases to search? A case study using a systematic review of frozen shoulder management. Health Inf Libr J 2013; 30:49–58.
    3. Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions: The Cochrane Collaboration, London, United Kingdom. 2011.
    4. Bramer WM, Rethlefsen ML, Kleijnen J, et al. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev 2017; 6(1):245.
    5. Aagaard T, Lund H, Juhl C. Optimizing literature search in systematic reviews—are MEDLINE, EMBASE and CENTRAL enough for identifying effect studies within the area of musculoskeletal disorders? BMC Med Res Methodol 2016; 16:161.
    6. Rice DB, Kloda LA, Levis B, et al. Are MEDLINE searches sufficient for systematic reviews and meta-analyses of the diagnostic accuracy of depression screening tools? A review of meta-analyses. J Psychosom Res 2016; 87:7–13.
    7. Bramer WM, Giustini D, Kramer BMR. Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: a prospective study. Syst Rev 2016; 5:39.
    8. Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. The contribution of databases to the results of systematic reviews: a cross-sectional study. BMC Med Res Methodol 2016; 16:1–13.
    9. Ross-White A, Godfrey C. Is there an optimum number needed to retrieve to justify inclusion of a database in a systematic review search? Health Inf Libr J 2017; 33:217–24.
    10. Bramer WM, de Jonge GB, Rethlefsen ML, et al. A systematic approach to searching: an efficient and complete method to develop literature searches. J Med Libr Assoc 2018; 106(4):531–541.
    11. Stevinson C, Lawlor DA. Searching multiple databases for systematic reviews: added value or diminishing returns? Complement Ther Med 2004; 12(4):228-32.
    12. Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, et al. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. J Clin Epidemiol 2015; 68:617–626.

    Written by: Ms. Tanvi Laghate

    MarksMan Healthcare

    ,

  • The Importance of Rapid Review Methods in Healthcare Decision-making

    The Importance of Rapid Review Methods in Healthcare Decision-making

    Systematic literature reviews (SLRs) have always been amongst preferred tools for policy and decision makers owing to their quality of evidence as well as the ability to provide knowledge-base in terms of clinical practice guidelines as well as policy briefs. However, strict methodology followed during the conduct of an SLR can impact its duration, which may range anywhere from 0.5 to 2 years. (1) Along with the stringent guidelines, SLRs not only require two independent reviewers to execute all the necessary procedures of literature review, screening, data extraction and risk of bias appraisal; but also several technical experts, such as librarians, statisticians, subject matter experts, research coordinators and so on. (2)

    Healthcare decision makers frequently require well-timed access to information in order to make informed decision regarding treatment and care. Even though SLRs are a preferred choice, they require a great deal of resources. In addition, the longer duration as required by SLRs may not harmonize with the needs of the researchers requiring a quicker review. For instance, the mean number of hours to conduct SLRs have been estimated to be 1,139 hours (range 216–2,518 hours) with a budget of about $100,000. (3)

    ‘Rapid reviews’ then come into the picture, which simplify or omit the stages involved in conducting SLRs to give quick information. Agency for Healthcare Research and Quality (AHRQ, US) defines a rapid review as “a form of evidence synthesis that may provide more timely information for decision making compared with standard systematic reviews”. (4,5) Today, rapid reviews are increasingly being used to synthesize evidence, particularly for making informed yet promising decisions in healthcare.(1)

    Even though rapid reviews are emerging as a preferred tool for making urgent decisions, there is a dearth of evidence surrounding their methodology.(1,2) A recent scoping review of rapid review methods has reported their conduct to be complicated. Findings of this review show that many important steps in an SLR are often omitted during rapid reviews, such as protocol application, limited literature search, limited inclusion criteria, and lack of quality appraisal among others. Rapid reviews also involve limited interpretation of findings and may cause bias. Many articles in the literature poorly report the methodologies of rapid review, thus warranting further improvement.(2) Therefore, clear gaps exist in transparency in and knowledge about the reliability of rapid reviews.(1) However, there are some areas where rapid reviews can be helpful, such as broader PICO questions, novel or emerging research topics, updates on previous reviews, critical topics, further exploration of a policy or practice using some systematic review methods, and so on.(5)

    The increasing preference for rapid reviews and their application into urgent decision-making emphasizes the need to further explore their characteristics and uses. While researcher opting for rapid reviews are answerable about the time-sensitive needs of the health decision makers they work for, they must do it in line with the expected methodological rigor. For this purpose, methodological research and standard development is crucial. (6)

    In conclusion, not only the methodologies, but also their consequences should be further investigated in order to better implement rapid review methods for making timely decisions. A prospective study, which can compare the results of rapid reviews to those achieved from an SLR around the same topic, can be conducted (1) and will be interesting to look at.

    Become an Certified HEOR Professional – Enrol yourself here!

    References

    1. Khangura S, Konnyu K, Cushman R, et al. Evidence summaries: the evolution of a rapid review approach. Syst Rev 2012; 1:10.
    2. Tricco A, Antony J, Zarin W, et al. A scoping review of rapid review methods. BMC Medicine 2015; 13:224.
    3. Petticrew M, Roberts H. Systematic reviews in the social sciences: a practical guide. Malden, MA: Blackwell Publishing Co.; 2006.
    4. Agency for Healthcare Research and Quality (AHRQ).
    5. Systematic Reviews & Other Review Types. Temple University.
    6. Cochrane Rapid Reviews Methods Group (RRMG).

    Written By: Ms. Tanvi Laghate

    MarksMan Healthcare

    ,