Original Article
AHRQ Series Paper 3: Identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care program

https://doi.org/10.1016/j.jclinepi.2009.03.008Get rights and content

Abstract

Objective

This article discusses the identification, selection, and refinement of topics for comparative effectiveness systematic reviews within the Agency for Healthcare Research and Quality's Effective Health Care (EHC) program.

Study Design and Setting

The EHC program seeks to align its research topic selection with the overall goals of the program, impartially and consistently apply predefined criteria to potential topics, involve stakeholders to identify high-priority topics, be transparent and accountable, and continually evaluate and improve processes.

Results

A topic prioritization group representing stakeholder and scientific perspectives evaluates topic nominations that fit within the EHC program (are “appropriate”) to determine how “important” topics are as considered against seven criteria. The group then judges whether a new comparative effectiveness systematic review would be a duplication of existing research syntheses, and if not duplicative, if there is adequate type and volume of research to conduct a new systematic review. Finally, the group considers the “potential value and impact” of a comparative effectiveness systematic review.

Conclusion

As the EHC program develops, ongoing challenges include ensuring the program addresses truly unmet needs for synthesized research because national and international efforts in this arena are uncoordinated, as well as engaging a range of stakeholders in program decisions while also achieving efficiency and timeliness.

Introduction

Globally, people are struggling with the reality of limited resources to address the breadth of health and health care needs. Evidence has been recognized as the “new anchor for medical decisions” [1], and many consider systematic reviews to be the best source of information for making clinical and health policy decisions [2]. These translational research products rigorously summarize existing research studies so that health and health care decisions by practitioners, policy makers, and patients are more evidence based. Yet, dollars for research—whether for systematic reviews, trials, or observational studies—are constrained, and are likely to be into the future. Effective prioritization is clearly necessary to identify the most important topics for synthesized research investment that may help the US health care system realize powerful and meaningful improvements in health status.

This article discusses the identification, selection, and refinement of topics for comparative effectiveness systematic reviews within the Agency for Healthcare Research and Quality's (AHRQ) Effective Health Care (EHC) program, which has been described in more detail elsewhere [3]. Briefly, AHRQ's EHC program was authorized in 2003 by the US Congress to conduct and support research on outcomes, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and health care services. This program uses the AHRQ Evidence-based Practice Center (EPC) program, with 14 designated centers throughout North America that conduct comparative effectiveness systematic reviews, among other research products of the program. AHRQ has designated a Scientific Resource Center (SRC) currently housed at the Oregon EPC to support the EHC program as a whole, with specific responsibilities including assisting AHRQ with all aspects of research topic development (see Fig. 1), providing scientific and technical support for systematic reviews and outcomes research, and collaborating with EHC stakeholder and program partners.

Selecting and developing good topics for research is not a simple process. Researchers' success depends, in large part, on their ability to identify meaningful questions, whereas funding agencies continually seek to maximize the return on their investment by funding research on important, answerable questions relevant to significant portions of priority populations. Although some have criticized how well funders have actually achieved these results [4], there is little guidance for successfully developing a research program that generates the type of evidence necessary to improve the public's health.

Section snippets

Guiding principles for identifying and selecting comparative effectiveness research topics in AHRQ's EHC program

To derive guiding principles for selecting important comparative effectiveness systematic review topics, we considered what others have done when trying to select priority topics for any health care–related activity. Over the last 18 years, the Institute of Medicine (IOM) and selected others have explored priority setting models and approaches [5], [6], [7], [8], [9], [10]. Across a diverse set of international health- and health care–related activities, including the development of guidelines

Processes for identifying and selecting comparative effectiveness systematic reviews in AHRQ's EHC program

As illustrated in Fig. 2, the current EHC program processes aim to allow the consistent, broadly focused development of a portfolio of relevant comparative effectiveness systematic reviews. These processes are more heavily focused now on engaging stakeholders than in the initial years of the EHC program, particularly during topic identification, but throughout the processes of research development and dissemination within the EHC program. We are using new and existing publicity avenues to

Principles and processes for refining selected topics for comparative effectiveness systematic review in AHRQ's EHC program

Once topics are selected for comparative effectiveness systematic review, they are further focused into research questions for systematic review. This process is designed to ensure that the research review results in a product that meets the needs of stakeholders. Key questions should reflect the uncertainty that decision makers, patients, clinicians, and others may have about the topic. Key questions guide the entire systematic review process, from the formulation of comprehensive search

Challenges in identifying, selecting, and refining topics for comparative effectiveness systematic reviews in AHRQ's EHC program

One of the main challenges we face as we move forward is to ensure that we have engaged the most important perspectives since, due to issues of timeliness and cost, we cannot engage all types of stakeholders at each step for every topic. Our goal is to continue to develop a system that fairly represents the range of interests of all stakeholders across all aspects of the program (Fig. 2), yet that results in timely and clear reports that are useful to decision makers and other audiences. For

References (24)

  • J. Slutsky et al.

    AHRQ Series Paper 1: Comparing medical interventions: AHRQ and the Effective Health-Care Program

    J Clin Epidemiol

    (2010)
  • D.M. Eddy

    Evidence-based medicine: a unified approach

    Health Aff (Millwood)

    (2005)
  • A. Laupacis et al.

    Systematic reviews: time to address clinical and policy relevance as well as methodological rigor

    Ann Intern Med

    (2007)
  • C.P. Gross et al.

    The relation between funding by the National Institutes of Health and the burden of disease

    N Engl J Med

    (1999)
  • R.N. Battista et al.

    Setting priorities and selecting topics for clinical practice guidelines

    CMAJ

    (1995)
  • Institute of Medicine

    National priorities for the assessment of clinical conditions and medical technologies: report of a pilot study

    (1990)
  • Institute of Medicine

    Setting priorities for health technology assessment: a model process

    (1992)
  • Institute of Medicine

    Setting priorities for clinical practice guidelines

    (1995)
  • Institute of Medicine

    Priority areas for national action: transforming health care quality

    (2003)
  • Institute of Medicine

    Knowing what works in health care: a roadmap for the nation

    (2008)
  • J.L. Gibson et al.

    Setting priorities in health care organizations: criteria, processes, and parameters of success

    BMC Health Serv Res

    (2004)
  • A.D. Oxman et al.

    Improving the use of research evidence in guideline development: 2. Priority setting

    Health Res Policy Syst

    (2006)
  • Cited by (74)

    • Facilitation of risk assessment with evidence-based methods – A framework for use of systematic mapping and systematic reviews in determining hazard, developing toxicity values, and characterizing uncertainty

      2020, Regulatory Toxicology and Pharmacology
      Citation Excerpt :

      This allows for both the utility and feasibility to be considered in determining specific objective(s) related to a risk assessment. Typically, problem formulation involves a multidisciplinary team (when possible) and an iterative series of interactions that consider stakeholder needs along with feasibility, ultimately resulting in the determination of the context, purpose, focus area, and format of an assessment (Whitlock et al. 2010). The planning processes to define the review question and the basis for determining how an assessment will be done directly impact the scope of subsequent evaluations.

    • A common framework of steps and criteria for prioritizing topics for evidence syntheses: a systematic review

      2020, Journal of Clinical Epidemiology
      Citation Excerpt :

      Most studies (n = 28; 97%) incorporated at least one step from the preprioritization phase, and around half (n = 15; 52%) incorporated at least one step from the postprioritization phase. Across the three phases, nine studies (31%) covered more than half of the steps involved in prioritization [14,20,24,25,29,31–33,37]. The steps of prioritization that were reported in most studies were generation of initial list of topics (n = 28; 97%), prioritization/ranking (n = 28; 97%), and research gap analysis (n = 24; 83%).

    View all citing articles on Scopus
    View full text