Contact Us

Home

About us

Services

AI Excellence

case studies

publications

HEOR Certificationheor Certification

AI for Medical Writing

Contact

  • The Importance of Grey Literature Search in Systematic Literature Reviews

    The Importance of Grey Literature Search in Systematic Literature Reviews

    During the course of any research, most of the relevant literature pertaining to a research question is retrieved though searching of recognised databases. However, in addition to this, searching of grey literature can add value to the depth of the research by providing information from varied sources. Grey literature search is an important, but often ignored, part of systematic literature review and data synthesis, especially in the medical research.

    Grey Literature: Definition, Types, and Sources

    The Grey Literature International Steering Committee (GLISC) defines grey literature as “Information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body”.[1] Grey literature is often self-published, and the sources of grey literature can include Government agencies, research institutions, organizations, companies, and associations.[2]

    Grey literature can be classified under various categories: [2,3]

    • Regulatory Information: this includes information available from archives of regulatory bodies such as the USFDA, CDSCO, EMA, and NICE. Regulatory information can also be sourced from stakeholder organizations and pharmaceutical companies, and include product information leaflets, white papers, internal documentations, SOPs, and procedure briefs. Other examples include company and industry wide repositories and financially driven investor service websites.
    • Government data: these include information available from government resources, such as notifications, guidelines, gazette notifications, judicial information, patent databases, policy briefs, etc.
    • Unpublished material from clinical trials: these include prospectively registered clinical trial protocols in repositories such as ClinicalTrials.gov in the USA and the Clinical Trial Registry of India (CTRI). Pre-prints which are not ultimately published due to various reasons, unpublished dissertations and theses,
    • Conference proceedings: abstracts, scientific sessions, and other conference proceedings provide a brief snapshot of contemporary research, which might not get published due to various reasons
    • Internet resources: With the increasing presence of social media, newer forms of grey literature have also surfaced, such as blogs, internet forums, wikis, video lectures, lecture slides and lecture notes, educational videos, personal websites, and information posted on the omnipresent social media.

    Importance of Including Grey Literature

    Commercial publishers are guided by interests and priorities, and all information which do not conform to these are often ignored and not published. This unpublished information forms the bulk of grey literature.

    A research which focuses solely on published material has a risk of missing out a comprehensive view of the topic under research. Grey literature provides valuable information about emerging or less popular research areas which are not published. Including a grey literature is also found to be useful in validating the results of a research-based literature search.[4]

    Grey literature bypasses the time-consuming peer-review process. Also, because of a quicker publication, the time delay between research and its formal publication is also bypassed. As a result, the information in grey literature can be more recent and up-to-date than in a formal publication. This is especially important in situations of public health emergency, such as the COVID-19 pandemic.

    There is a growing interest in using grey literature in systematic reviews and meta-analysis.[5] A study conducted by McCauley et al. analysed that 33% of meta‐analysis included some form of grey literature, accounting for 4.5% to 75% of studies in the meta‐analyses, and contributed significantly to the estimates of the intervention effects. The authors concluded that excluding grey literature from meta-analyses can result in a falsely exaggerated estimates of the effectiveness of intervention.[6] In several cases, information, and data disclosed at conference presentations is never published.[7] These observations have led to an opinion that the confounding effect of publication bias (where negative findings are more often not published) can be mitigated to a significant extent by including grey literature.[5-7]

    Challenges in Grey Literature Searching

    It is certain that grey literature search adds value to a research; however, it is also not possible to ascertain whether such a grey literature search has been done comprehensively or not. This is because grey literature is not well organized. In other words, there is no way to ‘define’ a proper grey literature search. It might be possible that a researcher has done grey literature search and included only data that is favourable, while excluding unfavourable data. This is in stark contrast to the published literature, which is often found to be well-organized. Thus, while it is possible to duplicate a database search strategy to verify if the search has been performed properly or not, such a luxury is not available with the quite unorganized grey literature. This is also the reason why inclusion of grey literature is more often than not entirely dependent on the choice of a researcher. Further, grey literature is not ensured to be peer-reviewed, which brings in an inherent bias.

    Conclusion

    The importance of transparency in research findings cannot be over emphasized. Publication bias still continues to be a huge problem in medical research. Grey literature search has a unique potential to improve transparency in medical research as well as offer a solution for publication bias, by including the unpublished information, thereby improving the comprehensiveness of research. With the increased ease of access of the internet, the access to grey literature has also become easier. Considering the amount of new information that grey literature can bring to research, all researchers must consider including grey literature search in their works. Efforts are required to organize the grey literature so that the credibility and validity of grey literature search can improve.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Grey Literature International Steering Committee. Guidelines for the production of scientific and technical reports: how to write and distribute grey literature. Available from: http://eprints.rclis.org/7469/1/nancy.pdf. Accessed on Jun 20th 2020
    2. Royal Roads University. Grey literature: what is it?: What is grey literature. Available from: https://libguides.royalroads.ca/greylit/what. Accessed on Aug 12th
    3. Citrome L. Beyond PubMed: Searching the “Grey Literature” for Clinical Trial Results. Innov Clin Neurosci. 2014 Jul;11(7-8):42-6.
    4. Benzies KM, Premji S, Hayden KA, et al. State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs. 2006;3(2):55-61.
    5. Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods. 2014;5(3):221-34.
    6. McAuley L, Pham B, Tugwell P, et al. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000;356(9237):1228-31.
    7. Hopewell S, McDonald S, Clarke M, et al. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007(2):MR000010.

    MarksMan Healthcare

    , , ,

  • How Rapid Reviews Are Assisting in Healthcare Decision Making?

    How Rapid Reviews Are Assisting in Healthcare Decision Making?

    Traditionally, systematic reviews are considered as the gold standard to inform clinical practice and policy decisions. However, systematic reviews are resource and time intensive. The time factor has been identified as a barrier to implementing results from evidence synthesis, as a result of an incongruence between the time required to produce a full systematic review and the time within which policy and other decision makers must take decisions.1,2 Hence, there is an increasing demand and a rising interest from healthcare decision makers and knowledge users for a summary of high-quality evidence within a short time period to support their practice and policy decision-making.1,3

    Rapid reviews are viewed as valid forms of evidence synthesis products that provide sufficient information and advice to base clinical and policy decisions.3 A rapid review enables the provision of a concise summary of the evidence to answer specific policy or research-related questions. This recent interest in RRs could be due to the relatively large resource, time, and budget demands of well conducted systematic reviews.

    The term ‘rapid review’ (RR) has broad and varied definitions, and it is important to know how it differs from a systematic review. Broadly, a rapid review is a type of evidence synthesis product, which includes components of a systematic review, albeit in a simplified form that enables it’s completion in a timely fashion.3,4

    A rapid review differs from a systematic review in terms of the scope of the review question, comprehensiveness of the search, rigour and/or quality control, and the type of synthesis.3,4,5 Overall, a RR is similar to a systematic review in terms of how the evidence is identified, appraised, selected and synthesised. However, to enable the review to be completed within a short timeframe, certain steps in the systematic review process are altered or skipped or modified or omitted!

    In order to achieve best possible outcomes to facilitate the use of RRs in decision-making and overcome the barriers of lack of timely and relevant research, a range of methods have been developed, which involve modifications to the systematic review methods.1,5,6,7 According to Khangura et al (2012), limiting the scope of RRs or having a more targeted research question is probably the most efficient shortcut because of its impact on the number of articles, including full-texts to be retrieved, screened, assessed and synthesised (including data extraction).5 However, the authors do state that it is important to engage and collaborate closely with the knowledge end-users.1,5,8 Other modifications to the traditional systematic review method include: a reduced list of sources or databases to be searched, including limiting these to specialised sources (e.g. of systematic reviews), or by date, or by language; exclusion of grey literature; relying on existing systematic reviews; full-text review limitation; the use of only one reviewer for study selection and data extraction; providing minimal conclusions or recommendations; and limiting external peer review.1,5,8

    Currently, there is no formal established methodological guidance on conducting and reporting RRs, and there is some recent work that identified a variety of approaches to RRs.4 A project work related to the extension of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) reporting guidelines (for systematic reviews) for the conduct and reporting of RRs is currently underway and registered with the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network.9

    In a scoping review of RRs that compared the results of RRs to full systematic reviews (SRs) in four studies, it was found that the results reported in both the types of reviews were similar, with no incongruence.4 A rapid review in collaboration with clinical experts and/or knowledge end-users is thus a useful tool for assisting clinicians and policy decision-makers to identify evidence-based strategies for implementation into practice and to identify future research priorities.

    Become an Certified HEOR Professional – Enrol yourself here!

    References:

    1. Khangura S, Polisena J, Clifford TJ, Farrah K, Kamel C. Rapid review: an emerging approach to evidence synthesis in health technology assessment. Int J Technol Assess Health Care. 2014; 30(1):20-7.
    2. Featherstone RM, Dryden DM, Foisy M, Guise JM, Mitchell MD, Paynter RA, et al. Advancing knowledge of rapid reviews: an analysis of results, conclusions and recommendations from published review articles examining rapid reviews. Systems Review. 2015; 4(50).
    3. Munn Z, Lockwood C, Moola S. The Development and Use of Evidence Summaries for Point of Care Information Systems: A Streamlined Rapid Review Approach. Worldviews Evid Based Nurs. 2015 Jun; 12(3):131-8.
    4. Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC Medicine. 2015; 13(224).
    5. Khangura S, Konnyu K, Cushman R, Grimshaw J, Moher D. Evidence summaries: the evolution of a rapid review approach. Systems Review. 2012; 1(10).
    6. Polisena J, Garrity C, Kamel C, Stevens A, Abou-Setta AM. Rapid review programs to support health care and policy decision making: a descriptive analysis of processes and methods. Syst Rev. 2015; 4:26.
    7. Ganann R, Ciliska D, Thomas H. Expediting systematic reviews: methods and implications of rapid reviews. Implement Sci. 2010; 5:56.
    8. Haby MM, Chapman E, Clark R, Barreto J, Reveiz L, Lavis JN. What are the best methodologies for rapid reviews of the research evidence for evidence-informed decision making in health policy and practice: a rapid review. Health Res Policy Syst. 2016; 14: 83.
    9. The EQUATOR network 2015, PRISMA-RR 2017: an extension to PRISMA for rapid reviews, Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network, Centre for Statistics in Medicine, NDORMS, University of Oxford, London. 

    Written By – Dr. Sandeep Moola (Research Fellow, The University of Adelaide, Australia)

    MarksMan Healthcare

    , ,

  • How to Elicit Expert Opinion to Understand Missing Health Outcomes?

    How to Elicit Expert Opinion to Understand Missing Health Outcomes?

    Missing data are a big concern in any research project and are often unavoidable in spite of investigators’ best efforts. Missing outcomes have two effects: reduced precision and power, and bias. Also, the loss of precision is inevitable, except the possible use of the available data; e.g. to be sure not to exclude from the analysis individuals who dropped out before the end of the study but who nevertheless reported intermediate values of the outcome. However, the statistician can aim to reduce bias through suitable choice of an analysis. (1)

    Randomized controlled trials (RCTs) typically have missing outcome data for some participants. Patient-reported outcomes (PROs) such as health-related quality of life (QoL) are mostly prone to missing data due to patients failing to complete follow-up questionnaires. Assumptions are often applied in case of statistical analyses for missing data to explicitly specify the values of the missing data: e.g. missing values being failures, as in smoking cessation trials. Other assumptions are inherent statements about the similarity of distributions, such as ‘last observation carried forward’. (1,2)

    In the primary trial analysis, an approach is often proposed which is valid under plausible assumptions for studies with the missing data. Instead of assuming that the data are ‘missing completely at random’ (MCAR), the primary analysis should suppose them to be ‘missing at random’ (MAR), i.e. the probability of missing data does not depend on the patient’s outcome, after conditioning on the observed variables (e.g. the patients’ baseline characteristics). However, the MAR assumption is unlikely to be used in many settings; for example, patients in relatively poor health are less likely to complete the requisite questionnaires, thus making the outcome data ‘missing not at random’ (MNAR). (2)

    The US National Research Council (NRC) report on missing data in clinical trials advocates sensitivity analyses for recognizing the data to be MNAR, in accordance with general methodological guidance for dealing with missing data and previous specific advice for intention-to-treat (ITT) analysis in RCTs. (3) On the other hand, systematic reviews show that in practice RCTs do not handle missing data appropriately. (4) Sensitivity analysis can be approached with either statistical modeling of parameters that represent outcome differences between individuals with complete versus missing data and/or exploring varying inferences with respect to the ‘sensitivity parameters’ assuming specific values. (5) The final output, i.e. results and conclusion, can then be compared over a reasonable range of values, possibly including a ‘tipping-point’ when results change. However, this approach does have a set of shortcomings. (2)

    An alternative is to allow experts to quantify their views. This is not only more intuitive and attractive for them, but it also considers a fully Bayesian approach and properly captures and reflects expert opinion (and associated uncertainty) about the missing data in the subsequent estimate of the treatment effect and its credible interval. This is particularly useful for those needing a quantitative summary of the trial, such as systematic reviewers, decision makers and health providers, as it provides a quantitative synopsis of interpretation of results by experts, given the missing data. When reviewing the study, experts will implicitly ‘fill in’ the gaps created by the missing data to arrive at their conclusions. The proposed elicitation approach, together with a Bayesian analysis, allows the study to comprehensibly quantify the impact of incorporating expert knowledge through to the estimates of treatment effectiveness.(1,2)

    Sensitivity analyses using Bayesian approach require practical tools for easier expert elicitation, and recent research focuses on elicitation approaches within group meetings. Group level elicitation has benefits for training and clarification and facilitates behavioral aggregation for achieving consensus. (6) However, because of the ‘feedback’ loop, these approaches are costly in both money and time. Thus, in many RCTs, it may not be viable to elicit opinion from a sufficient number and range of experts. Easier uptake of recommended approaches for sensitivity analysis for missing data within RCTs requires more accessible, practical tools for eliciting and synthesizing expert opinion to be developed and exemplified. (2)

    Using open source software like face-to-face or online ones to elicit beliefs from reasonably large number of experts without imposing an undue burden is one option that has been recently suggested. With this tool, the elicited views can be converted into informative priors for the sensitivity parameters in a pattern-mixture model which will allow for correlation in the elicited values across the trial arms. After this, the trial data can be re-evaluated under different MNAR assumptions to explore the robustness of the results. These methods, along with the expected level of loss to follow-up, could provide an improved estimate of the probable impact of missing data on the trial’s results. Therefore, this approach can significantly help improve trial design, so that the study results are more robust to anticipated levels of missing data.

    Become an Certified HEOR Professional – Enrol yourself here!

    References

    1. Jackson D, White IR, Leese M. How much can we learn about missing data?: an exploration of a clinical trial in psychiatry. Journal of the Royal Statistical Society Series A, (Statistics in Society). 2010; 173(3):593-612.
    2. Mason AJ, Gomes M, Grieve R, et al. Development of a practical approach to expert elicitation for randomized controlled trials with missing health outcomes: Application to the IMPROVE trial. Clinical Trials 2017; 14(4):357–367.
    3. Little RJ, D’Agostino R, Cohen ML, et al. The prevention and treatment of missing data in clinical trials. N Engl J Med 2012; 367(14):1355–1360.
    4. Bell ML, Fiero M, Horton NJ, et al. Handling missing data in RCTs; a review of the top medical journals. BMC Med Res Methodol 2014; 14:118.
    5. Little RJA. A class of pattern-mixture models for normal incomplete data. Biometrika 1994; 81(3):471–483.
    6. O’Hagan A, Buck CE, Daneshkhah A, et al. Uncertain judgments: eliciting experts’ probabilities. 1st ed. Hoboken, NJ: John Wiley & Sons, 2006.

    MarksMan Healthcare

    , , , , , , , ,

  • Scoping Reviews: An Evolving Concept in EBM

    Scoping Reviews: An Evolving Concept in EBM

    Scoping reviews are exploratory projects that systematically map the literature on a topic, identifying key concepts, theories and sources of evidence. Scoping reviews are often conducted before full syntheses, and undertaken when feasibility of the research is considered to be a challenge, either because the relevant literature is thought to be vast and diverse (varying by methods, theoretical orientations and disciplines) and/or it is thought that little literature exists. In the scoping review, the same systematic, rigorous methodologies used by the systematic review are used to find studies and extract data. Analyses and syntheses are part of every scoping review but the depth and type of analysis are different.

    A scoping review (also scoping study) usually refers to a rapid gathering of literature in a given policy of clinical area where the aims are to accumulate as much evidence as possible and map the results. Scoping reviews provide an overview of the type, extent and quantity of research available on a given topic. By ‘mapping’ existing research, a scoping review can identify potential research gaps and future research needs, and do so by using systematic and transparent methods.

    In 2005, Arksey and O’Malley published the first methodological framework for conducting scoping studies. The term ‘scoping review’ does not seem to have a commonly-accepted definition but several researchers have attempted definitions. Scoping reviews can be an efficient way of indentifying themes and trends in high-volume areas of scientific enquiry. Generally, a scoping review is an interactive process whereby existing literatures identified, examined and conceptually mapped, and where gaps are identified. Thus, a scoping review could be considered as a first step in doing a systematic review or large study.

    In simple words, researchers can undertake a scoping study to examine the extent, range, and nature of research activity, determine the value of undertaking a full systematic review, summarize and disseminate research findings, in addition to identifying the gaps in the existing literature. As such, researchers can use scoping studies to clarify a complex concept and refine subsequent research inquiries. Scoping studies may be particularly relevant to disciplines with emerging evidence, such as rehabilitation science, in which the paucity of randomized controlled trials makes it difficult for researchers to undertake systematic reviews. In these situations, scoping studies are ideal because researchers can incorporate a range of study designs in both published and grey literature, address questions beyond those related to intervention effectiveness, and generate findings that can complement the findings of clinical trials.

    The literature search in a scoping review should be as extensive as possible, and include a range of relevant databases, hand searches and attempts to identify unpublished literature. Often, the underlying aim of a scoping review is to explore the literature as opposed to answering specific questions. The scoping review should also include locating organizations and individuals that are relevant to the domain and what those groups have published. In the social sciences, scoping studies are performed at an initial stage of doing research (i.e. program, project, process, or grant). Scoping reviews are used in some research areas to justify further investigation, time and resources.

    In evidence-based practice, scoping studies are undertaken as distinct research projects, and as precursors to other types of research. However, a scoping study may be requested as a search prior to the systematic review or preparatory to costing research projects. The interpretation, methodology and expectations of scoping reviews are variable and suggest that conceptually, scoping is not well-understood or defined. The distinction between scoping as an integral preliminary process in the development of a research proposal or a formative, methodologically rigorous activity in its own right has not been examined. Scoping studies in medicine are slowly evolving; their strength lies in their ability to summarize a body of evidence for quick but accurate synthesis. As with other approaches to evidence synthesis a standardized approach is always welcome. Full literature searching aimed at retrieving a maximum number of relevant studies or articles in a given discipline starts with a scope of a topic.

    Become a Certified HEOR Professional – Enrol yourself here!

    MarksMan Healthcare

    , ,