Contact Us

Home

About us

Services

AI Excellence

case studies

publications

HEOR Certificationheor Certification

AI for Medical Writing

Contact

  • Hybrid Study Designs: Combining Existing and New Data for Valuable Insights

    Hybrid Study Designs: Combining Existing and New Data for Valuable Insights
    Hybrid Study Designs

    Hybrid study designs are research methodologies that seamlessly blend elements from multiple study designs to address specific research questions comprehensively. Such a hybrid approach allows researchers to capitalize on the strengths of each study design while mitigating their individual limitations. As a result, hybrid designs usually provide a more comprehensive and holistic view of the research question.(1, 2)

    Hybrid study designs come in various forms, each tailored to specific research needs. One example for hybrid study designs are ambispective studies, which combine retrospective and prospective data, enabling the evaluation of research questions before and after specific episodes, making it particularly valuable for studying intervention impacts or rare events.(3, 4)

    The hybrid study design strategy has been used to enhance the robustness of traditional clinical trials, resulting in the evolution of hybrid clinical trials. While traditional clinical trials have long been the gold standard for generating evidence to support the safety and efficacy of new interventions, they have faced limitations in capturing the complexities of everyday practice.  Hybrid trials aim to generate evidence that considers both the controlled rigor of an RCT and the real-world applicability of observational studies. They can provide insights that are relevant to both research and clinical practice and can embrace a more inclusive approach. Additionally, with access to retrospective data, hybrid trials can evaluate interventions over extended periods, providing valuable insights into the long-term impact of treatments and interventions on patient outcomes.(5)

    Among hybrid clinical trials, one notable subtype is the Pragmatic Randomized Clinical Trial (RCT), which focuses on evaluating interventions under real-world conditions. By prioritizing applicability to routine clinical practice, pragmatic RCTs generate evidence that is directly relevant to everyday healthcare settings. This approach enables researchers to assess whether an intervention works in practical clinical scenarios, providing evidence that directly informs decision-making in clinical practice and healthcare policy. Embedded Pragmatic Randomized Trials take the concept of a pragmatic RCT even further. In an embedded trial, the intervention being studied is seamlessly integrated into routine patient care within the existing healthcare system. This integration ensures minimal disruptions to routine care for both healthcare providers and patients, making the trial highly feasible and practical.(6, 7)

    There are other examples for hybrid clinical trials. The ‘Stepped-Wedge Cluster Randomized Trial’ design evaluates new interventions over time by sequentially randomizing different clusters to receive the intervention. The ‘Observational Study With Nested Randomized Trial’ design identifies participants from an observational study for an RCT exploring specific interventions within an existing cohort. The ‘Sequential Multiple Assignment Randomized Trial (SMART)’ randomizes participants to different treatments in multiple stages based on previous responses. These various designs offer flexibility and adaptability to address diverse research questions effectively.(8-10)

    However, implementing hybrid study designs also presents some challenges. Integrating multiple data sources and research methods requires careful planning and coordination. Researchers must ensure the compatibility and coherence of the different data sets, establish rigorous data management procedures, and address potential biases or conflicts that may arise from the combination of data. Additionally, researchers need expertise in quantitative and qualitative methods to navigate the complexities of hybrid designs effectively.(2, 3)

    In conclusion, hybrid designs represent remarkable steps forward in evidence generation. By merging existing data with newly collected information, these methodologies provide a holistic view of research questions, enabling researchers to uncover valuable insights and long-term trends. As healthcare continues to evolve, hybrid designs and hybrid clinical trials will play pivotal roles in bridging the gap between research and clinical practice, shaping a future where evidence is not only robust but also reflective of the diverse and ever-changing realities of patient care.

    Become a Certified HEOR Professional – Enrol yourself here!


    References:

    1. Soon GG, Nie L, Hammerstrom T, et al. Meeting the demand for more sophisticated study designs. A proposal for a new type of clinical trial: the hybrid design. BMJ Open. 2011 Jan 1;1(2):e000156.
    2. Landes SJ, McBain SA, Curran GM. An introduction to effectiveness-implementation hybrid designs. Psychiatry Res. 2019 Oct;280:112513.
    3. Curran GM, Bauer M, Mittman B, et al. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012 Mar;50(3):217-26.
    4. Sessler DI, Imrey PB. Clinical research methodology 2: observational clinical research. Anesthesia & Analgesia. 2015 Oct 1;121(4):1043-51.
    5. Zhu M, Sridhar S, Hollingsworth R, et al. Hybrid clinical trials to generate real-world evidence: design considerations from a sponsor’s perspective. Contemp Clin Trials. 2020 Jul;94:105856.
    6. Koehler M, Donnelly ET, Kalanovic D, et al. Pragmatic randomized clinical trials: a proposal to enhance evaluation of new cancer therapies with early signs of exceptional activity. Annals of Oncology. 2016 Jul 1;27(7):1342-8.
    7. Ramsberg J, Platt R. Opportunities and barriers for pragmatic embedded trials: triumphs and tribulations. Learning Health Systems. 2018 Jan;2(1):e10044.
    8. Creswell JW, Clark VP. Mixed methods research. SAGE Publications. 2011.
    9. Hemming K, Haines TP, Chilton PJ, et al. The stepped wedge cluster randomized trial: rationale, design, analysis, and reporting. BMJ. 2015 Feb 6;350.
    10. Digitale JC, Stojanovski K, McCulloch CE, Handley MA. Study designs to assess real-world interventions to prevent COVID-19. Frontiers in public health. 2021 Jul 27;9:657976.

    MarksMan Healthcare

    , , , ,

  • The HARmonized Protocol Template to Enhance Reproducibility (HARPER) in RWE

    The HARmonized Protocol Template to Enhance Reproducibility (HARPER) in RWE

    Real World Evidence (RWE) is generated as a result of the real-world usage of drugs, and thus can complement the RCT (randomized controlled trial) evidence that is generated in controlled settings. RWE uses real-world data (RWD) including patient health status, health care, and outcomes routinely collected from various, usually unstructured, sources such as electronic health records (EHRs), insurance claims databases, patient registries, pharmacy data, laboratory data, data from wearables, and so on.  Therefore, it represents a broader and more varied distribution of patients, and for the same reason, regulators consider RWE as a crucial part of the evidence used in regulatory and Health Technology Assessment (HTA) decisions. However, because of this varied and unstructured nature of RWD, the resulting RWE is also found to be quite varying in terms of design, structure, flow, and content, all of which culminates to a low level of reproducibility of RWE studies.(1, 2)

    Recognizing this issue, the Professional Society for Health Economics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) convened a joint task force including members from both societies and international stakeholder groups to develop a uniform template to help collect RWD and conduct RWE studies. The result was the publication of the Harmonized Protocol Template to Enhance Reproducibility (HARPER) in 2022. This template provides a set of core recommendations for clear and reproducible RWE studies, and is intended to be used throughout the research process from designing a study, registration, its implementation and reporting on those implementations.(3)

    The template builds upon prior efforts to increase transparency on the design and conduct of studies using RWD (such as the EMA PASS template, STaRT-RWE template, and NEST protocol). It considers current insights on the degree of detail required to ensure study reproducibility. The joint task force’s core committee examined each section heading in the mapped table of the existing protocol templates to develop HARPER.(4-7)

    The development process of this template harbored an assessment of its internal and external validity. Five subteams evaluated internal validity by testing and developing example use cases with various designs and data sources. External validity was assessed by comparing existing protocol templates or guidance developed by international multi-stakeholder groups to ensure compatibility with agreed-upon scientific principles. These efforts resulted in a standard template with integrated advice and detailed instructions for each heading.

    The headers of the template are mostly similar to those of the EMA PASS template. There are nine main sections, each with structured free text, tables, or figures. Users are encouraged to provide context and rationale for the investigations and decisions in structured free text. Users can also provide details about how the study was carried out using free-text and organized tables. Further, to improve understanding and usability of the template, the authors have also provided sample protocols for various use cases to show how to utilize the template.(3)

    Alongside the benefits mentioned above of this template, the creators have highlighted various limitations. For example, some complex study designs may be wholly reasonable but may not fit within the HARPER structure. Moreover, HARPER does not cover every aspect of transparency over the lifecycle of a research study, which may involve sharing protocol, code, data, and results.(3)

    As this template is relatively recently launched, its acceptability by different organizations and stakeholders is yet to be gauged. The pilot studies conducted as a part of internal validation of this template provides suitable examples for its implementation in various study designs. It is possible that with time and with feedback from various stakeholders, the HARPER template may see further revisions to accommodate the evolution of new methods. The end users may be better able to assess the quality of RWE studies and, consequently, their usefulness for decision-making if significant difficulties in protocol registration are addressed.

    Become A Certified HEOR Professional – Enrol yourself here!

    References:

    1. Chodankar D. Introduction to real-world evidence studies. Perspect Clin Res. 2021 Jul-Sep;12(3):171-174.
    2. Kim HS, Lee S, Kim JH. Real-world Evidence versus Randomized Controlled Trial: Clinical Research Based on Electronic Medical Records. J Korean Med Sci. 2018 Jun 26;33(34):e213.
    3. Wang SV, Pottegård A, Crown W, et al. HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force. Pharmacoepidemiol Drug Saf. 2023 Jan;32(1):44-55.
    4. Guideline on good pharmacovigilance practices (GVP) Module VIII: post‐authorisation safety studies (rev 3) section VIII.B.2. Study registration. European Medicines Agency; 2017.
    5. European Medicines Agency. Guidance for the format and content of the protocol of non‐interventional post‐authorisation safety studies.
    6. Wang SV, Pinheiro S, Hua W, et al. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ. 2021 Jan 12;372:m4856.
    7. National Evaluation System for health Technology Coordinating Center (NESTcc) Methods Framework A Report of the Methods Subcommittee of the NEST Coordinating Center: An initiative of MDIC. 2020.

    MarksMan Healthcare

    , ,

  • The Role and Importance of SLRs and RWE in Drug Price Negotiations in the USA

    The Role and Importance of SLRs and RWE in Drug Price Negotiations in the USA

    The cost of prescription drugs is a significant burden on patients and the healthcare system, especially in countries such as the USA. High drug prices can strain government programs, such as Medicare and Medicaid, and private insurers, which can lead to higher premiums for consumers. Additionally, high drug prices are responsible for increased out-of-pocket expenses for patients, which further lead to medication non-adherence, and thus result in poorer health outcomes. On the other hand, the research and development activities in pharmaceutical industries depend on their profit from sales, and an extremely harsh reduction in drug prices can have adverse consequences in terms of a lack of incentive for innovation in the pharmaceutical industry. (1)

    All these factors make responsible and reasonable price negotiation an extremely important process in the market access cycle for drugs. By negotiating drug prices appropriately, the government and payers can help ensure that patients have access to affordable medications while also promoting competition and innovation in the industry. (1)

    Price negotiations for drugs in the US are typically done by insurers and government programs, such as Medicare and Medicaid. The negotiation process is complex, involves multiple factors, and largely depends on the payer and the drug; the typical steps and factors considered include formulary placement, rebates and discounts, value-based arrangements, price controls, and competitive bidding. (2) To this effect, evidence on clinical effectiveness becomes extremely important, and it is essential that there is a robust and ethical body of evidence to display that the new innovation is efficacious, safe, and brings about enough value to justify the premium that the patients and payers are asked to pay for accessing the intervention.

    The traditional sources for clinical effectiveness evidence for the purpose of price negotiation of drugs are the same as those for marketing approval, and are largely constituted by Randomized Clinical Trials (RCTs), which are often conducted by pharmaceutical companies to demonstrate the safety and efficacy of their drugs. These trials are designed to meet regulatory requirements and are often submitted to the FDA as part of the drug approval process. (3)

    However, it is being increasingly realized that evidence in addition to traditional RCTs can play a crucial role in determining the true extent of efficacy, safety, and value of an intervention in a particular therapy area or patient population. Specifically, increasing interest is being shown towards using evidence from systematic literature reviews (SLRs) and real-world evidence (RWE) for informing clinical effectiveness data for drug price negotiations. (3, 4)

    SLRs, being comprehensive evaluations of existing research studies, provide a balanced summary of the available evidence on a drug’s safety, efficacy, and cost-effectiveness, and thus can be an invaluable resource for drug price negotiations. However, using SLRs is associated with some challenges pertaining to the varying quality of evidence of studies included in the SLR, publication bias (by which there is an overestimation of a drug’s effectiveness due to non-publication of many studies with negative results), time and resource constraints, conflicts of interest, and lack of generalizability. (4)

    RWE coming from the analysis of different sources such as electronic health records, claims data, and patient registries, can provide insights into how drugs are used in actual clinical practice, including their safety and effectiveness over time. By demonstrating the value of a drug in real-world settings, RWE can provide details about the actual usage pattern of an intervention post its marketing, compared to RCTs which offer a view of clinical effectiveness from a restricted population, prior to marketing. However, using RWE is also associated with certain challenges, such as quality of data, reliability, lack of data standardization, data interoperability, privacy concerns, and concerns about the quality of data analysis leading to generation of RWE. (3)

    Health authorities worldwide have taken several initiatives to include Systematic Literature Review and RWE as key elements in market authorization and in price and reimbursement negotiations. Of more credit, is the fact that in the USA, the 21st Century Cures Act specified that RWD could be used to support the approval of a new indication for a drug that is already approved or to support or satisfy post-approval study requirements. (5-7)

    Interestingly, the Inflation Reduction Act (IRA) is the biggest landmark set by United States federal law to curb inflation by reducing the deficit and lowering prescription drug prices. (5) This Act established a drug price negotiation program within the department of Health and Human Services (HHS), enabling the Secretary to negotiate the prices of certain costly drugs within the Medicare program. The Centres for Medicare & Medicaid Services (CMS), through the U.S. Department of Health and Human Services (HHS), released initial guidance outlining the conditions and limitations of the new Medicare Drug Price Negotiation Program for 2026. The Medicare Drug Pricing Negotiation Program and other provisions in the new drug law will improve Medicare’s capacity to serve those enrolled in the program and future generations of Medicare recipients. (5-8)

    SLRs and RWE have an important role to play in generating clinical evidence for drug price negotiations, and the USFDA is in the process of regulating the steps needed for this by drafting the guidance document.  Together with other provisions in the new drug law, the Medicare Drug Price Negotiation Program will increase Medicare’s ability to serve current Medicare beneficiaries as well as future generations. (8)

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Papanicolas I, Woskie LR, Jha AK. Health care spending in the United States and other high-income countries. Jama. 2018 Mar 13;319(10):1024-39.
    2. Gruber J. Delivering public health insurance through private plan choice in the United States. Journal of Economic Perspectives. 2017 Nov 1;31(4):3-22.
    3. Pulini AA, Caetano GM, Clautiaux H, et al. Impact of Real-World Data on Market Authorization, Reimbursement Decision & Price Negotiation. Ther Innov Regul Sci. 2021 Jan;55(1):228-238.
    4. Tarsilla M. Cochrane handbook for systematic reviews of interventions. Journal of Multidisciplinary Evaluation. 2010;6(14):142-8.
    5. Levitt. The Inflation Reduction Act Is a Foot in the Door for Containing Health Care Costs, JAMA Health Forum. 3 (2022) e223575.
    6. A Turning Point for U.S. Climate Progress: Assessing the Climate and Clean Energy Provisions in the Inflation Reduction Act | Policy Commons. https://policycommons.net/artifacts/2649285/a-turning-point-for-us-climate-progress_inflation-reduction-act/3672158/
    7. Inflation Reduction Act Guidebook | Clean Energy | The White House. https://www.whitehouse.gov/cleanenergy/inflation-reduction-act-guidebook
    8. Sullivan SD. Medicare Drug Price Negotiation in the United States: Implications and Unanswered Questions. Value Health. 2023 Mar;26(3):394-399.

     

    MarksMan Healthcare

    , , ,

  • The STaRT RWE Template: Improving Reporting of RWE Studies

    The STaRT RWE Template: Improving Reporting of RWE Studies

    Improvements in healthcare digitalization and accelerated regulatory approvals of novel interventions have boosted the possibilities for gathering real-world data (RWD) and using the resultant real-world evidence (RWE) to support the generalizability, efficacy, and safety of interventions and medical devices, assisting healthcare decision-makers and policymakers.[1-5] RWE helps establish clinical guidelines, perform risk assessment for medication safety, improve market access, and undertake various epidemiological evaluations. However, RWE studies are typically collaborative, interdisciplinary, and involve multiple databases, and are generally collected for non-research purposes. As a result, such studies have considerable variation in study designs and analytical parameters. This brings in a challenge in terms of reliability of RWE, and calls for innovations to ensure the robustness of RWE research methodologies and outcomes.[6]

    The lack of a uniform structure in reporting RWE studies has also led to a requirement of significant efforts on the part of regulatory organizations to assess such studies. To facilitate critical evaluation of published RWE studies, and also to facilitate appropriate conduct of ongoing RWE studies, several checklists, methods, and guidelines have been developed. Notable examples include the Assessment of Real-World Observational Studies (ArRoWS) critical appraisal tool, the European network of centres for pharmacoepidemiology and pharmacovigilance checklist, and many more.[7-10] Nevertheless, these guidelines and checklists are generic, allowing ambiguity, presumptions, and incorrect interpretation while planning RWE studies. This prompted the development of the Structured Template and Reporting Tool for RWE (STaRT-RWE) template in 2021.[6] 

    The driving force behind the development of STaRT-RWE was the realization that reproducible research requires clear communication of scientific methods and results. This template was developed through the collaboration of the Professional Society for Health Economics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE).[6] The template has been structured in such a way that it can be adapted with a variety of RWD sources and RWE study designs. It allows for addressing the methodological problems in RWE studies. The template supports the  design, execution, and evaluation of RWE studies to aid in healthcare decision making.

    This template uses structured tables and figures to show the user where, what, and how to provide specifics of study implementation.[6] Structured tables facilitate communication by making it easy to locate relevant data on important study variables, which helps both research teams and end consumers. These tables also establish how these parameters were measured and set clear expectations for study implementation that must be reported. It also allows sharing code, algorithms, and data for computational reproducibility. 

    STaRT-RWE complements other available checklists by offering a framework to aid researchers in being explicitly thorough in research planning, implementation, and communication.[6] The template, unlike a checklist, minimizes unclear writing and the possibility of misinterpretation by employing tabular and graphic representations.

    There are certain limitations to the STaRT-RWE template. Although the template is meant to be flexible, the rigidity of the tables may not be a suitable fit for many studies. Only a portion of the STaRT-RWE tables may be put to use depending on the investigation. Using the study implementation template to assist research design does not ensure that the decisions will result in impartial results. However, clear explanations of how research results were obtained and what methods were employed to counteract potential biases can considerably aid reviewers in correctly interpreting study results. Furthermore, the template’s primary emphasis is on choices related to study implementation. Despite including a section on the data sources, the template fields do not fully capture the details required to determine whether the data are appropriate for the purpose.[6] This template is a user guide for reproducing RWE studies. It facilitates repeatability, validity evaluation, and evidence synthesis for efficient decision-making. 

    The STaRT-RWE template can potentially enable researchers to adhere to the standards established by professional organizations for conducting and publishing RWE research, thereby minimizing the ambiguity and misinterpretation of using non-standard terminologies in RWE studies. This in turn can allow regulators and decision-makers to use RWE to the fullest possible extent to improve patient access to safe and effective medicines.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Katkade VB, Sanders KN, Zou KH. Real-world data: an opportunity to supplement existing evidence for the use of long-established medicines in health care decision making. J Multidiscip Healthc. 2018 Jul 2;11:295-304. doi: 10.2147/JMDH.S160029.
    2. Jaksa A, Mahendraratnam N. Learning from the past to advance tomorrow’s real-world evidence: what demonstration projects have to teach us. J Comp Eff Res. 2021 Nov;10(16):1169-1173. doi: 10.2217/cer-2021-0166.
    3. Hampson G, Towse A, Dreitlein WB, Henshall C, Pearson SD. Real-world evidence for coverage decisions: opportunities and challenges. J Comp Eff Res. 2018 Dec;7(12):1133-1143. doi: 10.2217/cer-2018-0066.
    4. Corrigan-Curay J, Sacks L, Woodcock J. Real-World Evidence and Real-World Data for Evaluating Drug Safety and Effectiveness. JAMA. 2018 Sep 4;320(9):867-868. doi: 10.1001/jama.2018.10136
    5. Sherman RE, Anderson SA, Dal Pan GJ, et al. Real-World Evidence – What Is It and What Can It Tell Us? N Engl J Med. 2016 Dec 8;375(23):2293-2297. doi: 10.1056/NEJMsb1609216.
    6. Wang SV, Pinheiro S, Hua W, et al. STaRT-RWE: structured template for planning and reporting on the implementation of real world evidence studies. BMJ. 2021 Jan 12;372:m4856. doi: 10.1136/bmj.m4856.
    7. Kurz X, Perez-Gutthann S; ENCePP Steering Group. Strengthening standards, transparency, and collaboration to support medicine evaluation: Ten years of the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). Pharmacoepidemiol Drug Saf. 2018 Mar;27(3):245-252. doi: 10.1002/pds.4381.
    8. Coles B, Tyrer F, Hussein H, Dhalwani N, Khunti K. Development, content validation, and reliability of the Assessment of Real-World Observational Studies (ArRoWS) critical appraisal tool. Ann Epidemiol. 2021 Mar;55:57-63.e15. doi: 10.1016/j.annepidem.2020.09.014.
    9. Langan SM, Schmidt SA, Wing K, et al. The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology (RECORD-PE). BMJ. 2018 Nov 14;363:k3532. doi: 10.1136/bmj.k3532.
    10. Allen A, Patrick H, Ruof J,et al. Development and Pilot Test of the Registry Evaluation and Quality Standards Tool: An Information Technology-Based Tool to Support and Review Registries. Value Health. 2022 Aug;25(8):1390-1398. doi: 10.1016/j.jval.2021.12.018.

    MarksMan Healthcare

    , , , ,

  • Generating Clinical Evidence From Electronic Health Records and Patient Registries

    Generating Clinical Evidence From Electronic Health Records and Patient Registries

    Both electronic health records (EHRs) and patient registries store and use patient-related clinical information. However, they are conceptualized for different purposes. Both are a significant source of real-world evidence (RWE) as they gather a considerable amount of clinical information collected in the real-world setting.

    An EHR is an electronic record of health data generated during routine patient care delivered by healthcare providers.(1) EHRs are widely used to obtain information on several medical parameters from patients and maintain clinical workflows.(2) They usually comprise data on patients’ demographic, vitals, administrative, claims (medical and pharmacy), and clinical parameters. They also include other patient-related information, such as data from health-related quality-of-life instruments, home-monitoring devices, and caregiver assessments.(3)

    EHRs usually represent individual care structures, such as primary, emergency, and intensive care units, and are visit-oriented and transferable. They also offer data from integrated systems in single or linked hospitals.(4) As the use of EHRs becomes extensive in clinical research, it is only ideal that they are designed to enhance diagnosis and clinical care to improve their relevance further. The design of EHRs can also update with time as the technology advances or depending on external factors, such as changes in data type as per coding or reimbursement patterns.(1, 3)

    A patient registry is “an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition, or exposure and that serves one or more predetermined scientific, clinical, or policy purposes.”(1) Patient registries are crucial in research as an ultimate platform for focused information about patients with specific health conditions. They often also help answer questions otherwise not answered by randomized clinical trials (RCTs), owing to practicality or ethicality. Registry data also help reduce the time and cost of prospective data collection. RWD generated in registries enables hypotheses generation in research, thus helping descriptive studies and research in health services.(5) Registries are typically patient-oriented and goal-driven. They are designed to collate information on specific exposures and health outcomes. Patient registries can be population-based or hospital-based.

    Often data captured from EHRs are used to construct patient registries. Specifically, EHRs can facilitate certain functions for patient registries, such as collection, cleaning, and storage of data. Likewise, a registry can enhance the value of the information gathered in the EHRs, for instance, comparative effectiveness, safety, and value, population management, and quality reporting, among others.(6)

    Data from EHRs, either as stand-alone or as complementary information to the primary research or data from administrative databases, have been used to support observational studies. For instance, the Euro Heart Survey (7), the Eurobservational Research Program (EORP) which followed the survey,(8) and the AHA Get With the Guidelines (AHA GWTG) (9) show clinical information from EHRs on several cardiovascular diseases.(3) Moreover, the EU-ADR project connects eight databases from four European countries (United Kingdom, Italy, The Netherlands, and Denmark) to facilitate the analysis of specific target adverse drug reactions (ADRs).(10) The USFDA uses data from EHRs from various sources, including sentinel systems,(11) claims databases (Medicare and Medicaid Services), and Veterans Affairs, among others, to support safety investigations for products after marketing approvals.(12)

    Many clinical registries across the globe comprise patient data on acute and chronic stages of different diseases, such as cancer, cystic fibrosis, and multiple sclerosis, to name a few. For instance, countries like the US, Canada, Australia, Germany, Sweden, and Argentina, have registries to monitor and store patient data on acute stroke. The Cystic Fibrosis Foundation Patient Registry is a clinical quality registry (CQR), developed from an epidemiological and clinical research model.(13) The American Heart Association (AHA) recommends 5 key concepts in establishing patient registries: ensuring high quality data, linking registries with relevant supplemental data, integrating registries with EHRs, safeguarding privacy, and funding considerations.(14) CQRs have also been reported to contain extensive clinical information that can complement data from government-monitored registries. These data are vital for assessing the quality of care and research.(5, 15)

    Clinical data from both EHRs and registries can generate meaningful evidence to enhance trial efficiency and optimize novel research approaches. These RWD sources can help comparative effectiveness research while facilitating new trial designs to address unmet clinical needs. Their use seems hopeful. However, the technological advancements in these sources need to be looked at with applicable care measures to ensure data privacy and confidentiality.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Ehrenstein V, Kharrazi H, Lehmann H, et al. Obtaining Data From Electronic Health Records. In: Gliklich RE, Leavy MB, Dreyer NA, editors. Tools and Technologies for Registry Interoperability, Registries for Evaluating Patient Outcomes: A User’s Guide, 3rd Edition, Addendum 2 [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2019 Oct. Chapter 4. Available from: https://www.ncbi.nlm.nih.gov/books/NBK551878/
    2. Gliklich R, Dreyer N, Leavy M, eds. Registries for Evaluating Patient Outcomes: A User’s Guide. Third edition. Two volumes. (Prepared by the Outcome DEcIDE Center [Outcome Sciences, Inc., a Quintiles company] under Contract No. 290 2005 00351 TO7.) AHRQ Publication No. 13(14)-EHC111. Rockville, MD: Agency for Healthcare Research and Quality. April 2014. Available from: http://www​.effectivehealthcare.ahrq.gov
    3. Cowie MR, Blomster JI, Curtis LH, et al. Electronic health records to facilitate clinical research. Clin Res Cardiol. 2017; 106(1):1-9.
    4. Hayrinen K, Saranto K, Nykanen P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform. 2008; 77:291–304.
    5. Hoque DME, Kumari V, Hoque M, et al. Impact of clinical registries on quality of patient care and clinical outcomes: A systematic review. PLOS ONE. 2017; 12(9): e0183667.
    6. Gliklich RE, Dreyer NA, Leavy MB. Interfacing Registries With Electronic Health Records. Registries for Evaluating Patient Outcomes: A User’s Guide. 2. Third ed. Rockville, MD: Agency for Healthcare Research and Quality (AHRQ); 2014. p. 3–22.
    7. Scholte op Reimer W, Gitt A, et al. Cardiovascular diseases in Europe. Euro Heart Survey−2006. European Society of Cardiology, 2006.
    8. Ferrari R. EURObservational research programme. Eur Heart J. 2010; 31:1023–1031.
    9. Smaha LA. The American Heart Association Get with the Guidelines program. Am Heart J. 2004; 148:S46–S48.
    10. Trifiro G, Fourrier-Reglat A, Sturkenboom MC, et al. The EU-ADR project: preliminary results and perspective. Stud Health Technol Inform. 2009; 148:43–49.
    11. Ball R, Robb M, Anderson SA, Dal Pan G. The FDA’s sentinel initiative-A comprehensive approach to medical product surveillance. Clin Pharmacol Ther. 2016; 99:265–268.
    12. Staffa JA, Dal Pan GJ. Regulatory innovation in postmarketing risk assessment and management. Clin Pharmacol Ther. 2012; 91:555–557.
    13. Schechter MS, Fink AK, Homa K, Goss CH. The Cystic Fibrosis Foundation Patient Registry as a tool for use in quality improvement. BMJ quality & safety. 2014; 23(Suppl 1):i9–i14.
    14. Bufalino VJ, Masoudi FA, Stranne SK, et al. The American Heart Association’s recommendations for expanding the applications of existing and future clinical registries a policy statement from the American Heart Association. Circulation. 2011; 123(19):2167–79.
    15. Emilsson L, Lindahl B, Koster M, et al. Review of 103 Swedish Healthcare Quality Registries. Journal of Internal Medicine. 2015; 277(1):94–136.

    MarksMan Healthcare

    , ,

  • RWE Framework by NICE (UK) for Optimizing RWD Collection and RWE Generation

    RWE Framework by NICE (UK) for Optimizing RWD Collection and RWE Generation

    Evidence from randomized clinical trials (RCTs) continues to be the standard reference point for treatment efficacy across the world. However, RCTs enrol patients based on strict inclusion and exclusion criteria, and hence RCT evidence is often not generalizable and inadequate for contributing to the day-to-day clinical practice decisions. Consequently, researchers are more interested in using real-world data (RWD) to guide healthcare decisions.(1) Analysis of RWD that enables risk vs. benefit assessment while also providing data on the utility of medical intervention is called the real-world evidence (RWE).(2)

    With the growing interest in adopting RWE for decision-making, regulators are formally introducing recommendations for both collecting RWD and generating RWE. The latest is the National Institute of Health Care and Excellence (NICE, UK). In June 2022, NICE launched a real-world evidence (RWE) framework for optimizing RWD collection to fill the knowledge gaps and make innovative healthcare interventions easily accessible to patients.(3)

    The NICE defines RWD as data on patient healthcare delivery gathered from real-world sources not controlled by any eligibility criteria like those in RCTs. The NICE guidance suggests that RWD and RWE can be used for various purposes, such as to distinguish diseases, interventions, and patient outcomes, to design and endorse economic models, to validate digital health applications (cases of RWD used to develop clinical algorithms), to address health inequalities, and to assess the impact of interventions on care delivery, among others.(3)

    The NICE RWE Framework is a part of NICE Strategy 2021 to 2026,(4) a five-year strategic plan focusing on the RWE use to fill evidence gaps, enhance NICE’s decision making, and facilitate patient access to innovative health technologies. The Framework will help identify the need to use RWE to limit uncertainties and advance guidance. The Framework thoroughly defines the best practices for designing, conducting, and reporting RWE studies to improve the quality and transparency of the evidence.(3)

    The NICE RWE Framework outlines the role of RWE in health technology assessments (HTAs). It has a separate dedicated section on best practices for comparative effectiveness studies as they have more refined considerations requiring higher precision. The Framework also offers the Data Suitability Assessment Tool (DataSAT) for assessing the data’s relevance and validity for the intended purpose. The goal of DataSAT is to compel researchers to justify the choice of the RWD source. Like other frameworks provided by FDA and EMA, the DataSAT focuses on the origin, quality, and reliability of data. The Framework also recommends that researchers follow the ‘target trial’ approach while designing an RWE study to pursue all measures to limit bias, control the confounders, and assess the robustness of the findings.(5)

    Although the NICE RWE Framework is at par with the other guidance documents by the FDA and EMA, it still has some gaps. For instance, it does not define the minimal criteria for acceptance of RWE study elements. Also, not all the RWE study elements are mentioned; therefore, more detailed processes and checklists are required for more comprehensive guidance.(5)

    It is expected that the NICE RWE Framework will keep updating with evolving processes and methodologies. In addition to the existing frameworks by other regulators such as the USFDA,(2) the NICE RWE framework will be an excellent resource for developing RWE studies, particularly comparative effectiveness studies. NICE encourages the submission of RWE studies with an open dialogue with the pharmaceutical companies. Transparency and accountability are central components of the RWE Framework as it enables researchers to provide all the possible documentation and justification of the RWE study design and execution.(5)

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. McDonald L, Lambrelli D, Wasiak R, Ramagopalan SV. Real-world data in the United Kingdom: opportunities and challenges. BMC Med. 2016; 14(1):97.
    2. Framework for FDA’s Real-World Evidence Program. December 2018. Available at: https://www.fda.gov/media/120060/download
    3. The National Institute of Health and Care Excellence Real World Evidence Framework. June 23, 2022. Available at: https://www.nice.org.uk/corporate/ecd9/resources/nice-realworld-evidence-framework-pdf-1124020816837
    4. The NICE Strategy 2021 to 2026. Available at: https://www.nice.org.uk/about/who-we-are/corporate-publications/the-nice-strategy-2021-to-2026
    5. Jaksa A. RWE Guidance: NICE’s RWE Framework. April 2022. Available at: https://aetion.com/evidence-hub/rwe-guidance-nices-rwe-framework/

    MarksMan Healthcare

    ,

  • How to Improve Transparency with Real World Data to Support Market Access?

    How to Improve Transparency with Real World Data to Support Market Access?

    The USFDA defines real-world data (RWD) as ‘the data relating to patient health status and/or the healthcare delivery that is routinely collected from a variety of sources’, and real-world evidence (RWE) as ‘the clinical evidence regarding the usage and potential risks/benefits of a medical product obtained from analysis of RWD.’[1] RWD includes data from electronic health records (EHRs), administrative and medical claim databases, pharmacy data; data from product, patient, and disease registries, patient-generated data (including in-home use settings, social media data, patient forums, etc), and data gathered from other sources that can inform on health status.[1]

    RWD has been extensively used for pharmacovigilance in the form of phase 4 post-marketing surveillance studies. Lately, it is being realised that RWE and RWD can play additional role in the drug approval cycle. For example, the approval of Avelumab for the treatment of metastatic Merkel cell carcinoma (MCC) in March 2017 utilised electronic health record (EHR) data as historical control data for efficacy. Likewise, the approval of Lutetium Lu 177 dotatate for the treatment of certain neuroendocrine tumors in 2018 by the USFDA made extensive use of safety and efficacy data from the ERASMUS study, which was an expanded access study.[2]

    RWD has also been used for market access and to make reimbursement decisions in several countries. For example, in France, RWD collected during temporary authorization for use of a drug can be used to assess the price and level of reimbursement through health technology assessment (HTA). In the UK, RWD collected during early access to medicine scheme can be used as dossier of market access. In fact, the UK NHS has performed cost-effectiveness studies based on RWE and established a scheme to collect RWD for pharmacovigilance purposes. RWD are also used for pay-for-performance schemes in the USA.[3]

    Despite the raising importance of RWD in improving market access, a constant critique of RWD is that, since RWD is collected from routine healthcare, there is no set objective for RWD collection, unlike randomized controlled trials (RCTs) which generally have an objective and hypothesis. In other words, since RWE studies are secondary analysis of existing data that are collected unplanned (i.e., without a specific objective), they are susceptible to bias; this is not that prominent an issue for pre-planned studies of prospectively collected data (such as RCTs). Thus, there are high chances that RWE studies are more susceptible to results-driven design modifications than RCTs, thus bringing a question about transparency of RWD and the resulting RWE.[4]

    Data transparency improves the ability of decision-makers to assess the quality and validity of an RWE study by giving a deeper understanding of why and how the research was conducted and whether the results reflect pre-established questions and methods. It also facilitates the replication of results and an understanding of why findings of apparently similar studies differ. In line with these concepts, attempts are being made to improve RWD transparency, with an intention to strengthen the confidence that different stakeholders have on RWD, thereby further boost its usage for different aspects, including enhancing market access of drugs.[4]

    One such method for improving data transparency is by promoting registries for RWE studies. Though existing registries register and ClinicalTrials.gov) collect many features that are required for improving data transparency, they focus on primary data, and are not specific for RWD.[4] In 2017, International society for Pharmacoeconomics and Outcomes Research (ISPOR) and International Society for Pharmacoepidemiology (ISPE) formed a joint task force to identify good practices for addressing the concerns and to enhance confidence in evidence derived from RWE studies. The ISPOR-ISPE Special Task force recommends the researchers should declare the hypothesis to be tested, post study protocols and analysis publicly; and during publication attestation of any conformation or deviation from the initial study protocol.[4] In lines with the same, the ISPOR RWE registry was formally launched in October 2021, and represents a fit-for-purpose platform for registering RWE study designs prior to data collection, with an intention to facilitate RWD transparency and to elevate the trust in the study results; the RWE registry can be accessed at https://osf.io/registries/rwe/discover.

    Another method to improve RWD transparency is the concept of tokenization of healthcare data, by which different sources of patient-level RWD (for example; claims, EHR, registries, molecular biomarkers, and laboratories) can be linked to provide a complete, non-duplicate, and comprehensive understanding of the patient’s health. By providing an option to cross-check same patient data from different sources, tokenization can help improve the confidence in RWD from different sources.[5]

    RWD play enormous role in research and development process, they also help in estimating the risks and benefits of any treatment in real world scenarios. Enhancing RWD with transparency can go a long way in increasing its usage for market access.

    Become A Certified HEOR Professional – Enrol yourself here!

    References

    1. Framework for FDA’s real-world evidence program. Dec 2018. Available from: https://www.fda.gov/media/120060/download
    2. Feinberg BA, Gajra A, Zettler ME, et al. Use of real-world evidence to support FDA approval of oncology drugs. Value Health. 2020 Oct;23(10):1358-1365. 
    3. Pulini AA, Caetano GM, Clautiaux H, et al. Impact of real-world data on market authorization, reimbursement decision & price negotiation. Ther Innov Regul Sci. 2021;55(1):228-238.
    4. Orsini LS, Berger M, Crown W, et al. Improving transparency to build trust in real-world secondary data studies for hypothesis testing-why, what, and how: recommendations and a road map from the real-world evidence transparency initiative. Value Health. 2020 Sep;23(9):1128-1136. 
    5. Dagenais S, Russo L, Madsen A, et al. Use of real-world evidence to drive drug development strategy and inform clinical trial design. Clin Pharmacol Ther. 2022;111(1):77-89.

    MarksMan Healthcare

    , , , ,

  • Unlocking the Power of RWE: Changing Scenario in the Global Landscape

    Unlocking the Power of RWE: Changing Scenario in the Global Landscape

    There has been a shift in the global healthcare ecosystem from volume-based to value-based payment model, thanks to a surge in data availability, interoperability, advancing health technologies, cost and competitive pressures, scientific advances, and increasing adoption of personalized medicine. The resulting availability of a large quantity of real-world data (RWD) has made it possible to perform continual observation of disease epidemiology, treatment patterns, and outcomes in the real world. Analysing strong RWD generates strong real-world evidence (RWE), and the incredible power of RWE in the drug approval process, including prioritizing and streamlining drug development, is being realised by all stakeholders. RWE especially gains importance because randomized controlled trials (RCTs) cannot be applied to the entire patient population of a specific disease. Parallel to this, the value, usage, and acceptance of RWE in the pharmaceutical and biotechnology industries have also increased in recent years.[1]

    RWE is increasingly used by the regulators in the drug and device approval cycle, for safety evaluation, updating label claims, and for new usage approvals, as a supplement to RCTs for improved understanding of efficacy and safety of medical products and devices. However, the main concern in the usage of RWE lies in the robustness and quality of RWD. Since RWD is mined under uncontrolled settings from data often collected without a pre-defined objective, there is a possibility of data inconsistencies and spurious results, leading to a relatively lower quality of data compared to RCT data. Data quality has been defined to ensure conformance, completeness, and plausibility,[2] and to achieve high quality of RWD, there is a need for uniform regulatory guidelines and frameworks surrounding RWD collection and analysis.

    Globally, regulatory bodies are showing interest in adopting RWE as a component of the decision-making process to complement RCT evidence by strengthening the guidelines and framework for including RWD. For example, in the USA, the 21st Century Cures Act and Prescription Drug User Fee Act recommend the use of RWE, as a supplement to RCTs evidence, for regulatory decision-making and approval of drugs. In December 2018, the USFDA released a framework for the USFDA’s RWE program for evaluating the potential use of RWE for approval support to drugs and biologics.[3] The key considerations in the USFDA RWE program are: RWE must be ‘fit for use’; trial/study designs should provide adequate scientific evidence; and RWE must comply with the USFDA regulatory requirements. [4] In addition to the USFDA’s efforts, several other initiatives, such as the Clinical Trial Transformation initiative, Friends of Cancer research, are working to optimize RWD, developing new study methods, and refining RWD analytics.[3,5]

    In the UK, an RWD framework has been structured to ensure that the collected RWD is of relevance, provenance, and sufficient quality. The framework ensures relevancy, transparency at all levels of study planning, conduct, and reporting, and robust analytics to minimize bias and uncertainty. This living framework is being periodically updated based on user feedback and practice.[6] In Europe, the EMA launched the OPTIMAL (OPerational, TechnIcal, and MethodologicAL) framework in 2019 to explore the pertinent use of valid RWE for regulatory purposes.[7]

    Similarly, Health Canada is working with the Canadian Agency for Drugs and Technologies in Health (CADTH) and the National Institute of Excellence in Health and Social Services (INESS) to establish a joint document to optimize the use of RWE. In Japan, the Pharmaceuticals and Medical Devices Agency (PMDA) established regulatory guidelines in March 2021 for the use of registries, to ensure the reliability of RWD and RWE.[4]

    In addition to these efforts, the ICH-GCP has also established plans to harmonize global RWE and update its existing E6 (General considerations in clinical trials) and E8 (Guidelines for Good Clinical Practice) guidelines. The CIOMS (Council of International Organisations of Medical Sciences) is currently developing a consensus report and recommendations for the use of RWE in the regulatory decision-making process.[4]

    The shift from restricted uses of traditional evidence sources (RCT) to wider adaptation of newer modes (RCT + RWE) in different regions of the world is a positive sign showing a global increase in the value of RWE in research, practice, and policymaking. RWE undoubtedly serves as a key to a more robust, less expensive, and more inclusive approach to better healthcare through research. Undoubtedly there are gaps in the use of RWE at present: some countries have more acceptance and activity than others. There is a need for development of realistic and robust standards and best practices to ensure the quality of RWD used in RWE. Recommendations and uniform guidelines are needed across the world to shape, harmonize and generate reliable RWE. Nonetheless, the future holds a good promise on RWE.

    Become A Certified HEOR Professional – Enrol yourself here!

    References:

    1. Khosla S et al. Real world evidence (RWE) – a disruptive innovation or the quiet evolution of medical evidence generation? F1000Research. 2018;7:111.
    2. Kahn M et al. A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data. eGEMs (Generating Evidence & Methods to improve patient outcomes). 2016;4(1):18.
    3. Framework for FDA’s real-world evidence program. 2018. Available from: https://www.fda.gov/media/120060/download
    4. Burns L et al. Real-world evidence for regulatory decision-making: guidance from around the world. Clinical Therapeutics. 2022;44(3):420-437.
    5. Oncology Real World Evidence Program. USFDA 2021. Available from: https://www.fda.gov/about-fda/oncology-center-excellence/oncology-real-world-evidence-program
    6. The NICE strategy 2021 to 2026. NICE 2022. Available from: https://www.nice.org.uk/about/who-we-are/corporate-publications/the-nice-strategy-2021-to-2026
    7. Big Data Steering Group workshop. Available from: https://www.ema.europa.eu/en/documents/work-programme/workplan-hma/ema-joint-big-data-steering-group_en.pdf  

    MarksMan Healthcare

    ,

  • The Contribution of Wearables Towards Healthcare Evidence Generation

    The Contribution of Wearables Towards Healthcare Evidence Generation

    The health and fitness wearable market is on the rise since the last few years. Analysts expect that by 2020, almost half a billion smart wearable devices will have been sold. However, despite the fast-growing market, only 10% of those consumers are using the product daily. This is an opportunity for innovative life science companies to tap into the market and create value-added services for consumers. Demand for wearables which include wristbands, smart garments, chest straps, sports watches and other health monitors is being driven by consumer fascination in quantifying personal health metrics, but it also opens up a world of opportunities to the wider healthcare and pharmaceutical industries. Pharma wants to take wearables beyond fitness trackers to add value through disease diagnosis and monitoring solutions in the form of medical-grade wearables, while also generating evidence in the form of real-world data. 

    In line with the evolution of patient-centricity, wearables have enabled us to become personal data creators, with constant streams of fitness and medical information being generated of our own volition. The question is can this be harnessed by the healthcare industry to drive efficiencies and enhance products and healthcare services in the long term? Apple thinks so. It launched the Apple CareKit this year: software that makes it easier for individuals to keep track of care plans and monitor symptoms and medication, with the ability to share that information with doctors or family. It delivers the promise of empowering the patient and personalizing their care patient centricity at its very best.  Apple also offers a ResearchKit, providing a software framework for apps that enables medical researchers to gather robust and meaningful data. Interestingly in July 2016, GlaxoSmithKline launched a rheumatoid arthritis study, called PARADE, and an iPhone app using Apple’s ResearchKit, demonstrating the first time a drug maker has used the open source software framework to conduct clinical research.

    Wearables are being put to use wherever we look, including in the NHS, which has endorsed wearable use in its own trials. In partnership with Diabetes UK and Hewlett Packard it is deploying mobile health self-management tools (wearable sensors and software) for people with Type 1 and Type 2 diabetes to self-manage their condition. The devices feed data back via the internet to allow more timely intervention from peers, healthcare professionals, carers and social networks, should their help be required.

    Evidence to date suggests that wearable technology is plugging a gap in our knowledge by way of collecting real-world data directly from patients. Hospitals are deploying wireless monitoring so that patient’s vital signs are automatically and continuously fed back to clinicians. Furthermore, sensors are now available that automatically measure and store glucose readings removing the need for finger pricking. But can it really help us as market researchers to better understand the patient journey- is the question.

    Results from a recent survey of HCPs and pharma professionals, which was presented at the BHBIA Conference (May 2016), shows that an already quite apparent use of wearable devices especially amongst HCPs. Two thirds currently make use of biometric data for personal reasons outside of work, which is found to be surprisingly high. But when the use of standard health applications on mobile devices is factored in, this may help to explain things. Over 40% of HCPs use biometric data for work-related purposes, but this is much lower for pharma. Again this seems high, but perhaps this supports increasing examples of patients bringing in their data on their devices to help inform consultations.

    The question is how pharma can generate evidence from wearables? Biometrics collected from wearables could potentially offer an exciting future for qualitative research – to determine what is actually happening instead of what patients tell us is happening. For example, while carrying out a qualitative interview with a patient and discussing mobility, data from their wearable device could help verify whether the patient has been as mobile as they say they have been.   From a quantitative perspective, biometric data from wearables could feed into online segmentation studies. For example, does segment A exhibit a higher heart rate and greater sleep disturbance? In this instance biometric data could be collected to help facilitate the analysis process and test additional hypothesis that at present are very difficult to test. And of course, there is the potential for biometric data to be collected alongside our on-going tracking studies – to monitor how important metrics are changing over time and to identify periods in the day/ week/ month where patients exhibit peaks and troughs.

    It seems the application for data derived from wearables has increasingly more value in the healthcare research process as the technology improves and becomes more reliable. Their potential lies in their inconspicuousness – we forget we’re wearing them, and they provide access to an un-simulated world of responses to everyday events to either back up or refute what we think we know already.

    There could be a very bright future for wearables at the centre of healthcare research, but there are many issues to address first. The accuracy of the data from wearables must be assessed. Not to mention the issues surrounding use; at the very least does the patient own a device and have they worn it at all times without tampering or hindrance? There are ethical issues around the data, gaining approval and ownership of the information generated. And of course, if there’s a problem in the data, should it be reported or not? There are still many unanswered questions.

    Wearables are unlikely to ever entirely supersede traditional market research, but it’s clear their use for wider healthcare research purposes is on the increase. They currently provide a valuable tool to use in conjunction with more traditional methods of research by offering context, putting clinical data into a more relevant light and further personalizing the patient journey. Could the collection of biometric data provide researchers with a new, previously inaccessible, form of insight that may help in further understanding the patient experience? Only time will tell.

    Become A Certified HEOR Professional – Enrol yourself here!

    MarksMan Healthcare

    , ,