Skip to main content

The GRADE Evidence to Decision (EtD) framework for health system and public health decisions

Abstract

Objective

To describe a framework for people making and using evidence-informed health system and public health recommendations and decisions.

Background

We developed the GRADE Evidence to Decision (EtD) framework for health system and public health decisions as part of the DECIDE project, in which we simultaneously developed frameworks for these and other types of healthcare decisions, including clinical recommendations, coverage decisions and decisions about diagnostic tests.

Developing the framework

Building on GRADE EtD tables, we used an iterative approach, including brainstorming, consultation of the literature and with stakeholders, and an international survey of policy-makers. We applied the framework to diverse examples, conducted workshops and user testing with health system and public health guideline developers and policy-makers, and observed and tested its use in real-life guideline panels.

Findings

All the GRADE EtD frameworks share the same basic structure, including sections for formulating the question, making an assessment and drawing conclusions. Criteria listed in the assessment section of the health system and public health framework cover the important factors for making these types of decisions; in addition to the effects and economic impact of an option, the priority of the problem, the impact of the option on equity, and its acceptability and feasibility are important considerations that can inform both whether and how to implement an option. Because health system and public health interventions are often complex, detailed implementation considerations should be made when making a decision. The certainty of the evidence is often low or very low, but decision-makers must still act. Monitoring and evaluation are therefore often important considerations for these types of decisions.

We illustrate the different components of the EtD framework for health system and public health decisions by presenting their application in a framework adapted from a real-life guideline.

Discussion

This framework provides a structured and transparent approach to support policy-making informed by the best available research evidence, while making the basis for decisions accessible to those whom they will affect. The health system and public health EtD framework can also be used to facilitate dissemination of recommendations and enable decision-makers to adopt, and adapt, recommendations or decisions.

Peer Review reports

Background

Policy-makers or managers, often in groups or individually whilst advised by a group, have the mandate to set priorities and make health system and public health decisions on behalf of a population. Health systems require decisions about how services are delivered, financed and governed, and about implementation strategies [1,2,3,4]. Public (population) health decisions that are made on behalf of a population include decisions about whether to offer organised screening and other public health programmes, environmental and occupational health policies, injury prevention policies, and nutrition and food safety policies. Such decisions can be made at an international level (e.g. WHO guidelines), a national level (e.g. deciding on national regulations for physicians and other health professionals) or a local level (e.g. deciding whether to care for acute stroke patients in a specialised stroke unit or a general medical ward).

All healthcare decision-making is complex. Decision-makers may not have clear criteria, may sometimes neglect important criteria, give inappropriate importance to certain criteria, or may not use the best available evidence to inform their judgments. Structured and transparent systems for decision-making can help to ensure that all important criteria are considered and that the best available research evidence is used.

In this article, we present, and illustrate with an example, the GRADE Evidence to Decision (EtD) framework for people who make or use health system and public health recommendations and decisions. We will use the term ‘health system and public health decisions’ to indicate both recommendations and decisions, and we will refer to the group of people making a decision or advising decision-makers as the ‘panel’.

GRADE EtD frameworks

Building on the GRADE approach to making judgments about the strength of recommendations and EtD tables [5,6,7,8], the DECIDE project [9] developed EtD frameworks for different types of healthcare decisions, including clinical recommendations, coverage decisions, and health system or public health recommendations and decisions. This is described in detail elsewhere [10,11,12,13,14].

Figure 1 shows how EtD frameworks can fit into practical decision-making processes, who is involved, and who the target audience for the framework is. For any healthcare decision, policy-makers and their constituents identify and prioritise problems, then work with a technical team to formulate a question. The technical team searches for and summarises the evidence and populates the EtD framework. The framework is then used by a panel to facilitate and document their decision-making, and later by people who will either use or be affected by a decision, including policy-makers, health professionals, patients and the public. Additional file 1 is a glossary of terminology used in EtD frameworks for all types of healthcare decisions.

Fig. 1
figure 1

Evidence to Decision (EtD) conceptual map workflow

The purpose of the frameworks

The main purpose of EtD frameworks is to help panels use evidence in a systematic and transparent way to inform decisions [10]. EtD frameworks support panels by informing panel members about the relative pros and cons of the interventions or options being considered; ensuring that panel members consider all the important factors (criteria) for making a decision; providing panel members with a concise summary of the best available evidence about each criterion to inform their judgments; helping panels structure and document discussion; and helping panels identify reasons for disagreements, making the process and the basis for decisions transparent.

EtD frameworks support users of recommendations and those affected by decisions by assisting them to understand the judgments made by the panel and the evidence supporting those judgments; helping policy-makers and managers consider whether recommendations can and should be implemented in their own settings; and facilitating adoption of decisions and adaptation of recommendations to specific contexts.

Developing the GRADE EtD framework for health system and public health decisions

We aimed for consistency across the EtD frameworks for different types of decisions; most of the criteria are similar across the frameworks, as can be seen in Table 1, which shows the criteria for five types of decisions. However, different types of decisions and varying perspectives require different considerations. Consequently, the specific criteria for health system and public health decisions differ in some ways from the criteria for clinical recommendations, coverage decisions, and recommendations and decisions about tests.

Table 1 Criteria for Evidence to Decision (EtD) frameworks for five different types of decisions

We developed the EtD framework for health system and public health decisions using an iterative approach, as described in a previous publication [9]. Additional input came from experience with evidence-based policy briefs [15, 16]. We obtained feedback from an international stakeholder group including health system and public health researchers and policy-makers, and undertook an international survey of policy-makers’ perceptions regarding the criteria in the framework and of how best to summarise and present evidence to support health systems [17]. We applied the framework to a diverse set of examples [18,19,20,21,22,23], conducted workshops with guideline developers and policy-makers, and observed guideline panels using the framework. Finally, we undertook user testing [24] with members of guideline panels and with policy-makers.

Below we describe the EtD framework for health system and public health decisions, and highlight its differences from the EtD frameworks for other types of decisions. Similarities and differences between the EtD framework for health system and public health decisions and other EtD frameworks are summarised in Table 2. To illustrate the application of each component of the health system and public health EtD framework, we present an example of its use, namely a recommendation regarding women’s groups using participatory learning and action (PLA) cycles to achieve a positive pregnancy experience. We prepared this framework, presented fully in Additional file 2, based on two WHO guidelines, one on antenatal care for a positive pregnancy experience [18] and one on community mobilisation through facilitated PLA cycles with women’s groups for maternal and newborn health [19], in which the EtD for health system and public health decisions was used. We modified the EtD frameworks used in these two guidelines to reconcile differences between them and to clarify the use of the framework in relation to some of its specific components.

Table 2 Similarities and differences between the Evidence to Decision (EtD) framework for health system and public health decisions and other types of decisions

Description of the health system and public health EtD framework

The EtD framework includes three main sections that reflect key steps in going from evidence to a decision, namely formulating the question, making an evidence-informed assessment and drawing conclusions. The EtD framework for health system and public health decisions shares the same basic structure as the EtD frameworks for other types of decisions [10]. Additionally, as for other types of decisions, the framework accommodates both recommendations and decisions with either two options or multiple options (Box 1).

Formulating the question

Panels, sometimes together with a technical team, formulate questions by defining the problem they are addressing (e.g. perinatal and maternal mortality, morbidity and pregnancy experience) and considering options to address it (e.g. women’s groups using PLA cycles). In the framework and in this article, we use the term ‘options’ to describe the interventions or actions available to address a problem. The question also includes the comparison option, the main outcomes, the setting and the perspective from which the decision is being made. Given that their decisions will affect groups of people, or a whole population, health system and public health decision-makers take a population perspective. The specific type of population perspective taken depends on the nature of the decision. For example, a decision about an environmental health policy might be made from a societal perspective (incorporating all important outcomes and all costs and savings, regardless of who benefits or pays). In the participatory women’s groups example, the panel took a population perspective, and considered costs and savings within the healthcare system.

In the example framework in Additional file 2, the team developing this WHO guideline formulated the ‘problem’ positively as a goal, that of a positive pregnancy experience. A positive pregnancy experience was defined as maintaining physical and sociocultural normality, maintaining a healthy pregnancy for mother and baby (including preventing or treating risks, illness and death), having an effective transition to positive labour and birth, and achieving positive motherhood (including maternal self-esteem, competence and autonomy).

The option considered in the framework in Additional file 2 was women’s groups using PLA cycles and the comparison was usual care. Women’s groups using PLA cycles are facilitated by trained facilitators, with the aim of identifying, prioritising and addressing problems women face around pregnancy, childbirth and after birth, and empowering women to seek care and choose healthy pregnancy and newborn care behaviours. Meetings are usually held on a monthly basis and activities are differentially prioritised according to the local context and conditions.

The main outcomes considered were attending four or more antenatal care visits, attending one or more antenatal care visit, delivery in a health facility, perinatal deaths and maternal deaths. The decision setting was a global recommendation, and the perspective was that of ministries of health.

It is often important to consider subgroups or contexts for which it may be appropriate to make separate judgments, and potentially different decisions, particularly when making global recommendations. These should be identified when the question is formulated to ensure that panels consider relevant evidence and their judgments for different subgroups or contexts when making an assessment. For example, although we have not specified any subgroups in the framework in Additional file 2, the WHO panel specifically considered women in rural settings with low access to health services in making its recommendation.

Intellectual and financial conflicts of interest are common and can affect judgments and recommendations or decisions [25,26,27,28,29]. Guideline developers and organisations responsible for healthcare decisions should consider conflicts of interest when a panel is established [29]. In addition, because potential conflicts of interest can vary across questions, panels should consider and report them when formulating each question. They should also specify actions to address these, which can range from simply declaring a conflict of interest to excluding panel members from discussions of specific questions or an entire guideline [27,28,29]. In the participatory women’s groups example, the panel reported that all panellists declared either non-important minor or no conflicts of interest (Additional file 2).

Making an evidence-informed assessment

The EtD framework lists explicit criteria that are important to consider when making a decision. The EtD framework criteria for different types of healthcare decisions are listed in Table 1. For each criterion, the technical team enters a summary of the research evidence and any information for additional consideration by the panel, with links to more detailed information. The panel assesses the research evidence and additional considerations presented, and makes an informed judgment about the options. Research evidence refers to information derived from studies that used systematic and explicit methods. Additional considerations include other evidence, such as routinely collected data, and assumptions and logic used to make a judgment. Panels may make different judgments for one or more subgroups in relation to some or all of the criteria. When relevant, they may also report additional details, such as dissenting views of panel members or the results of voting on judgments for which there was disagreement.

The criteria in the EtD framework for health system and public health decisions include questions about whether the problem is a priority, the magnitude of the desirable and undesirable effects, the certainty of the evidence, consideration of how people who are directly affected value the main outcomes, the balance between desirable and undesirable effects, resource use and cost-effectiveness, impacts on equity, and the acceptability and feasibility of the option. Table 3 shows other criteria that we have incorporated in the framework as detailed judgments, which some organisations might want to consider as separate criteria. For example, a Swedish group that assessed the applicability of an earlier version of the EtD framework for public health decisions concluded that two criteria which we have included as detailed judgments should be added as criteria [23]. The first, individual autonomy, is a detailed judgment for the criterion for acceptability, and the second, sustainability, is a detailed judgment for feasibility (Table 3).

Table 3 Detailed judgments in Evidence to Decision (EtD) frameworks

The problem

Healthcare decisions require setting priorities on how best to use limited resources. In considering whether the problem being addressed in a health system and public health decision is a priority, the number of people affected is important to consider, in addition to the severity, urgency and consequences of the problem, and whether it is a recognised priority (e.g. based on a national health plan or international health targets such as the UN Sustainable Development Goals). The more people that are affected, the more likely it is that doing something to address the problem should be a priority. For example, governments might prioritise problems that represent a larger burden of disease for their population or that place a larger burden on the healthcare system by making an explicit comparison with other health problems of mortality figures and disability adjusted life years.

In the example in Additional file 2, the problem is achieving a positive pregnancy experience. Approximately 303,000 women and adolescent girls died as a result of pregnancy and childbirth-related complications, and approximately 2.6 million babies were stillborn in 2015 [30, 31]. Almost two thirds of the global maternal and neonatal disease burden could be alleviated through optimal adaptation and uptake of existing research findings [32]. In addition, reducing maternal and neonatal mortality are targets of Sustainable Development Goal 3 (ensuring health and well-being for all). Based in part on this evidence, the panel concluded that achieving a positive pregnancy experience is a priority. This judgment applied to all of the panel’s recommendations in the guideline and did not need to be repeated for each one.

Benefits and harms

Panels should consider the evidence about the benefits and harms of the options and how certain that evidence is. They also need to consider how much the people affected directly by the decision value the benefits and harms, whether there is important uncertainty about this, and whether there is important variability in how much people value the benefits and harms [11]. They must then consider all these criteria together to make a judgment about the balance between the desirable and undesirable effects of the option.

The certainty of the evidence for health system and public health interventions is often low or very low [1,2,3,4]. Nonetheless, policy-makers and managers must still make decisions. Panels need to consider indirect as well as direct effects of options; for example, herd immunity may be an important consideration for immunisation programmes in addition to the direct benefits and harms experienced by the people who are vaccinated. How much the people affected by the decision value the outcomes is particularly important when potential harms and benefits are closely balanced. The more uncertainty or variability there is about how much those affected value the main outcomes, the less likely a panel is to make a strong recommendation for an option. There is often a paucity of evidence about how much people value important outcomes. When this is the case, panels should state the basis for their judgments and any assumptions that they have made.

In the example in Additional file 2, the panel judged that the desirable effects of participatory women’s groups are moderate, but uncertain, the undesirable effects are trivial, and the overall certainty of the effects is low. The panel was not presented any evidence of how much women value the main outcomes that were considered. However, women’s groups very likely increase communication and social support, which women value [33]. Given that no adverse effects of women’s groups were identified, there was no important uncertainty or variability in how much women value the main outcomes. Based on these judgments, the panel determined that the balance of the desirable and undesirable effects probably favours the option of women’s groups using PLA cycles.

Resource use and cost-effectiveness

Because resources are limited, policy-makers and managers making health system and public health decisions must consider the resource implications of implementing alternative options, in addition to the extent to which an option is cost-effective. Typically, there are uncertainties regarding the inputs that are required, the effects, the economic consequences of those effects, and the resources consumed or saved due to implementation of an option. There may be limited or no available economic analyses, nor resources available to conduct one [34, 35]. As with clinical recommendations and coverage decisions, the greater the budget impact and the more uncertainty there is about the resources required and cost effectiveness, the less likely a panel is to recommend or decide to implement a health system or public health option [11, 13]. Conversely, the greater the savings, the more appealing an option becomes.

For the example in Additional file 2, amongst other costs, facilitators’ time, training and supervision must be considered. These costs are difficult to estimate and may vary widely across settings [36]. The panel’s judgment was that there are moderate costs associated with implementing women’s groups using PLA cycles, but that the certainty of this evidence was low.

On the other hand, a systematic review of the cost-effectiveness of strategies to improve the utilisation and provision of maternal and newborn healthcare in low- and lower middle-income countries reported that there was high certainty evidence for the cost-effectiveness of women’s groups using PLA cycles [36, 37]. The panel’s judgment was that, despite wide variation in the incremental cost per neonatal death averted in the included trials, the available cost-effectiveness evidence probably favours the option of women’s groups using PLA cycles.

Equity

The impacts on equity of health system and public health options are important because these decisions are always taken from a population perspective. Decision-makers can address potential impacts on equity by considering the possible differential effects of options on disadvantaged populations, and in relation to characteristics that are likely to be associated with disadvantage [38, 39], including economic status, employment or occupation, education, place of residence, sex and ethnicity. Decision-makers also need to consider who will bear the costs (or benefit from the savings) of options, and the impacts that this will have on equity [38].

In the example in Additional file 2, there is evidence that the effect of participatory women’s groups on neonatal mortality rates was greatest among the most socioeconomically marginalised [40, 41]. Participatory women’s groups are one way of facilitating dialogue with, and empowering, disadvantaged populations to engage in efforts to improve health and to strengthen broader community support. Based on this evidence, the panel’s judgment was that women’s groups using PLA cycles probably increase equity.

Acceptability

Decision-makers should consider the acceptability of options – and of decisions of whether or not to implement them – to all the key stakeholders. For health system and public health decisions, key stakeholders may include those affected by the problem and the option, public officials and politicians, healthcare managers, the general public, health workers and their unions, and special interest groups [42]. The acceptability of an option may depend on evidence presented for some of the preceding criteria, such as the distribution and timing of harms, benefits and costs, and how much different stakeholders value the harms and benefits.

An option might be more or less acceptable to some stakeholders depending on the distribution of the impacts and demands of an option across the people affected, including the burden. For example, shifting tasks from one health worker cadre to another (e.g. from midwives to lay health workers) might increase both groups’ job satisfaction, but might also increase the workload for one of these groups, making that option more, or less, acceptable to them [43].

The timing of the impacts and demands of an option might also affect its acceptability, particularly for preventive interventions, in which costs are incurred in the short term and the benefits are in the future. Most stakeholders prefer to delay undesirable effects and costs, rather than incur them in the present, and they value immediate desirable effects more than those in the future [44]. The size of this preference – the discount rate – varies, and some stakeholders might have a substantially higher or lower discount rate than others. In particular, discount rates may affect politicians’ decisions.

Disagreement amongst stakeholders regarding the values assigned to the desirable and undesirable effects of an option might also affect acceptability to some stakeholders. This might be due to their perceptions of the relative importance of the effects for others. Ethical principles, such as autonomy, non-maleficence, beneficence and justice [45, 46], may affect acceptability. For example, stakeholders for whom autonomy is especially important might be opposed to options that limit people’s autonomy, such as legislation that mandates the use of helmets or vaccines, even when they agree about the balance between the desirable and undesirable effects of those interventions.

In the example in Additional file 2, there was high certainty evidence that women readily engage with interventions designed to increase communication with and support from other pregnant women and healthcare providers, and that participatory women’s groups are likely to do this [47]. There was also high certainty evidence that healthcare providers are willing to supply pregnancy-related information and offer psychological and emotional support to women (either via women’s groups or antenatal visits) provided the resources and organisational support are available. Based on this evidence, the judgment of the panel was that women’s groups using PLA cycles are probably acceptable to all stakeholders.

Feasibility

Limitations in the feasibility of a health system or public health option can affect decisions about whether to recommend or implement the option. They can also have implications for how to implement the option, if it is recommended or implemented. It is unhelpful to recommend options that are impractical. Therefore, it is incumbent on health system and public health decision-makers to consider important barriers to implementing options and strategies to address any barriers they identify [48, 49].

In the example in Additional file 2, there was high certainty evidence that healthcare providers may need additional training if they are to be involved in facilitating women’s groups and this may be a barrier in some resource-poor settings. Women’s groups are also likely to require a reorganisation of services, and using existing public sector health workers and local health systems may be more feasible and more likely to succeed than project-based interventions [50]. Despite these barriers, the panel’s judgment was that the implementation of participatory women’s groups is probably feasible.

Drawing conclusions

After making a judgment in relation to each criterion, panels must make a decision based on all of those judgments. Often, it will be straightforward for a panel to consider the implications of their judgments, and to make a decision; sometimes, it is not. Panels making health system or public health recommendations should make explicit assessments about the strength of their recommendations based on their judgments for all the criteria [5,6,7].

By strength of recommendations we mean the extent to which the panel can be confident, after considering all the relevant criteria, that all of the desirable consequences of an option outweigh all of its undesirable consequences (Additional file 1). A strong recommendation means the panel is confident that the desirable consequences outweigh the undesirable, or vice versa; a conditional recommendation means the panel is less confident. When panels make a conditional recommendation, they should provide clear guidance regarding the specific conditions that favour implementing or rejecting the option.

For all decisions, panels should provide a justification in which they summarise their judgments for the criteria that were most important for their decision. For difficult decisions or decisions on which panel members disagree, it can be helpful to consider explicitly the implication of each judgment and the weight given to each of the criteria.

In the example in Additional file 2, the panel made a conditional recommendation for women’s groups using PLA cycles in rural settings with low access to health services. Their justification for this decision was not provided in the guideline, but is likely to have been because the balance between the desirable and undesirable effects, the impact on equity, and the cost effectivity probably favour the option. The conditionality of recommending it in rural settings is likely because this is where the studies were performed, and the panel was uncertain about whether the effects would be the same in urban areas [19].

Policy-makers and managers making decisions (rather than recommendations) cannot make strong or conditional decisions. The choices available to them are fully implementing the option, implementing it and conducting an impact evaluation, conducting a pilot study prior to fully implementing it, implementing it only in some settings or populations, postponing a decision, or not implementing the option [51].

Subgroup considerations

Health system or public health decision-makers should reach conclusions about relevant subgroups or contexts, and include any important considerations in their conclusions. A previous WHO recommendation on community mobilisation through facilitated PLA cycles with women’s groups for maternal and newborn health [19] was used to inform the recommendation in Additional file 2. The panel for that WHO recommendation concluded that it is possible that PLA cycles with women’s groups could have different effects in urban versus rural areas, but elected not to make the recommendation for rural areas only because urban areas can also experience poor access and marginalisation. However, it also recommended that more research should be carried out in urban areas.

Implementation

Health system and public health options are often complex, and decision-makers may require detailed guidance on implementation. Key implementation considerations can typically be drawn from the evidence, additional considerations and judgments that were made regarding equity, acceptability and feasibility. Key organisational, political, social and resource considerations are listed in the framework in Additional file 2 for women’s groups using PLA cycles.

Monitoring and evaluation

Because there is often important uncertainty about the effects of health system and public health interventions, monitoring and evaluation considerations are a key element of most conclusions.

Decision-makers should consider what monitoring and evaluation will be necessary. They should consider what indicators are important to monitor, and whether impact evaluation is needed [51, 52]. When there is a lack of evidence or there are important uncertainties about the impacts of health system and public health interventions, it is particularly important to monitor and evaluate their implementation.

The panel in the participatory women’s groups example stipulated that, to ensure high quality implementation adapted to the local context, ongoing monitoring and evaluation is necessary. They also listed research priorities (Additional file 2).

Dissemination of EtD frameworks

The target audiences of health system and public health decisions may be policy-makers and managers deciding whether to implement policies, programmes or recommendations in their jurisdictions, or the people who are affected by those decisions.

People making policy decisions should be accountable to the people affected by their decisions. The EtD framework facilitates a transparent approach for policy-making and for documenting the basis for decisions so that this is accessible. Use of an interactive EtD tool makes it possible to generate presentations for different audiences, including a recommendation to decision presentation for policy-makers and managers considering whether to implement a recommendation in their jurisdiction. An example of a recommendation to decision presentation generated from the EtD framework in Additional file 2 can be found here. This presentation provides a structure for policy-makers and managers to discuss and decide about the adoption and adaptation of a recommendation. They can consider the same criteria as the panel who made the recommendation, while incorporating evidence specific for their setting and making judgments that are appropriate for their context. This is particularly important for global recommendations.

Discussion

The health system and public health EtD framework supports systematic, structured and transparent use of evidence for recommendations and decisions. Policy-makers are likely to understand the summarised evidence presented in the framework better than systematic reviews [53], and the framework strengthens the credibility of decisions by documenting the evidence-based decision-making process. This includes showing how judgments were made when there was a lack of evidence.

The use of EtD frameworks requires ensuring that panel members are familiar with, and have a shared understanding of, the contents of the framework and its role in making decisions. This is a potential limitation of EtD frameworks. However, once this is achieved, panel members have reported that the frameworks help to structure discussion, often saving time, and ensure systematic and explicit consideration of all of the relevant factors [14].

Many governments and organisations responsible for health system and public health decisions do not have sufficient resources and expertise to prepare an optimal EtD framework. This is not a limitation of the EtD framework, but a practical limitation to whether the EtD framework is used or not. It will often be necessary to take shortcuts. In these situations, the EtD framework can still help decision-makers ensure that they use a systematic and transparent approach to making decisions, while enabling documentation of the evidence that  was used to inform their judgments, and how judgments were made when evidence was lacking. Indeed, as noted at the beginning of this article, key purposes of the EtD framework are to support panels by helping them to structure and document discussion, and to identify reasons for disagreements. The framework can help to improve decision-making regardless of the resources available to populate a framework with evidence or the availability of reliable evidence to inform decisions.

Using the EtD framework may require additional resources to synthesise evidence in addition to that of effectiveness, for example, qualitative evidence of acceptability and feasibility. This is sometimes critical, but it is not always necessary. For example, systematic reviews of qualitative evidence have been essential for some health system recommendations [20], but may not be needed if, for example, there is high certainty evidence of effects and large net benefits or harms, with little concern about the acceptability or feasibility of an intervention.

A strength of the EtD framework is its flexibility. In addition to flexibility with regards to how much effort goes into populating a framework with evidence, it is flexible in terms of the relative importance attached to the included criteria, since this depends on the nature of the decision being made. Moreover, the EtD framework can be adapted by organisations, for example, by modifying the key criteria that are included [23]. The interactive EtD tool [54] and GRADE’s official software GRADEpro GDT [55] include 16 templates for different types of recommendations and decisions for either two or multiple options. The templates include clinical recommendations from different perspectives, coverage decisions, recommendations about tests, and the four templates for health systems and public health recommendations and decisions shown in Box 1. It is possible for organisations to translate and adapt these to meet their specific mandate and needs.

WHO and other organisations that make health system and public health recommendations can facilitate the use of the EtD framework by policy-makers and managers by providing them with recommendation to decision presentations (such as the one in Additional file 3), which mirror the EtD frameworks used by panels making recommendations. Recommendations that use EtD frameworks can, in this way, substantially reduce the amount of work required by governments or organisations at national and sub-national levels and facilitate adaptation of recommendations to specific settings.

Conclusions

The health system and public health EtD framework provides a structured and transparent approach to support policy-making informed by the best available research evidence, while making the basis for decisions accessible to those whom the decisions will affect. This framework can also be used to facilitate dissemination of recommendations, and enable decision-makers to adopt, and adapt, recommendations or decisions.

Abbreviations

EtD:

Evidence to Decision

PLA:

Participatory learning and action

References

  1. Ciapponi A, Lewin S, Herrera CA, Opiyo N, Pantoja T, Paulsen E, et al. Delivery arrangements for health systems in low-income countries: an overview of systematic reviews. Cochrane Database Syst Rev. 2017;9:CD011083.

    PubMed  Google Scholar 

  2. Wiysonge CS, Paulsen E, Lewin S, Ciapponi A, Herrera CA, Opiyo N, et al. Financial arrangements for health systems in low-income countries: an overview of systematic reviews. Cochrane Database Syst Rev. 2017;9:CD011084.

    PubMed  Google Scholar 

  3. Herrera CA, Lewin S, Paulsen E, Ciapponi A, Opiyo N, Pantoja T, et al. Governance arrangements for health systems in low-income countries: an overview of systematic reviews. Cochrane Database Syst Rev. 2017;9:CD011085.

    PubMed  Google Scholar 

  4. Pantoja T, Opiyo N, Lewin S, Paulsen E, Ciapponi A, Wiysonge CS, et al. Implementation strategies for health systems in low-income countries: an overview of systematic reviews. Cochrane Database Syst Rev. 2017;9:CD011086.

    PubMed  Google Scholar 

  5. Guyatt GH, Oxman AD, Kunz R, Falck-Ytter Y, Vist GE, Liberati A, et al. Going from evidence to recommendations. BMJ. 2008;336:1049–51.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Andrews J, Guyatt GH, Oxman AD, Alderson P, Dahm P, Falck-Ytter Y, et al. GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol. 2013;66:719–25.

    Article  PubMed  Google Scholar 

  7. Andrews J, Schünemann HJ, Oxman AD, Kunz R, Brozek J, Vist G, et al. GRADE guidelines: 15. Going from evidence to recommendations: determinants of a recommendations direction and strength. J Clin Epidemiol. 2013;66:726–35.

    Article  PubMed  Google Scholar 

  8. Akl EA, Kennedy C, Konda K, Caceres CF, Horvath T, Ayala G, et al. Using GRADE methodology for the development of public health guidelines for the prevention and treatment of HIV and other STIs among men who have sex with men and transgender people. BMC Public Health. 2012;12:386.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Treweek S, Oxman AD, Alderson P, Bossuyt P, Brożek J, Davoli M, et al. Developing and Evaluating Communication Strategies to Support Informed Decisions and Practice Based on Evidence (DECIDE): protocol and preliminary results. Implementation Sci. 2013;8:6.

    Article  Google Scholar 

  10. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016;353:i2016.

    Article  PubMed  Google Scholar 

  11. Alonso-Coello P, Oxman AD, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ. 2016;353:i2089.

    Article  PubMed  Google Scholar 

  12. Schünemann HJ, Mustafa R, Brozek J, Santesso N, Alonso-Coello P, Guyatt G, et al. GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health. J Clin Epidemiol. 2016;76:89–98.

    Article  PubMed  Google Scholar 

  13. Parmelli E, Amato L, Oxman AD, Alonso-Coello P, Brunetti M, Moberg J, et al. GRADE Evidence to Decision (EtD) framework for coverage decisions. Int J Technol Assess Health Care. 2017;2017(33):2.

    Google Scholar 

  14. Rosenbaum SE, Moberg J, Glenton C, Schünemann HJ, Lewin S, Akl E, et al. Developing Evidence to Decision frameworks and an interactive Evidence to Decision tool for making and using decisions and recommendations in health care. Global Challenges. 2018. https://doi.org/10.1002/gch2.201700081.

  15. Lavis JN, Permanand G, Oxman AD, Lewin S, Fretheim A. SUPPORT tools for evidence-informed health policymaking (STP). 13. Preparing and using policy briefs to support evidence-informed policymaking. Health Res Policy Syst. 2009;7(Suppl 1):S13.

    Article  PubMed  PubMed Central  Google Scholar 

  16. The SURE Collaboration. SURE Guides for Preparing and Using Evidence Briefs for Policy. Version 3.0 [updated May 2014]. The SURE Collaboration. 2014. http://www.who.int/evidence/sure/en/. Accessed 15 May 2018.

  17. Vogel JP, Oxman AD, Glenton C, Rosenbaum S, Lewin S, Gülmezoglu AM, et al. Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: An international survey. Health Res Policy Syst. 2013;11:19.

    Article  PubMed  PubMed Central  Google Scholar 

  18. World Health Organization. WHO recommendations on antenatal care for a positive pregnancy experience. Geneva: World Health Organization; 2016. http://www.who.int/reproductivehealth/publications/maternal_perinatal_health/anc-positive-pregnancy-experience/en/. Accessed 15 May 2018.

  19. World Health Organization. WHO recommendation on community mobilization through facilitated participatory learning and action cycles with women’s groups for maternal and newborn health. Geneva: World Health Organization; 2014. http://apps.who.int/iris/bitstream/10665/127939/1/9789241507271_eng.pdf?ua=1. Accessed 15 May 2018.

  20. Glenton C, Lewin S, Gülmezoglu AM. Expanding the evidence base for global recommendations on health systems: strengths and challenges of the OptimizeMNH guidance process. Implement Sci. 2016;11:98.

    Article  PubMed  PubMed Central  Google Scholar 

  21. World Health Organization. Optimizing health worker roles to improve access to key maternal and newborn health interventions through task shifting. Geneva: World Health Organization; 2012. http://www.optimizemnh.org. Accessed 15 May 2018.

  22. World Health Organization. Health worker roles in providing safe abortion care and post-abortion contraception. Geneva: World Health Organization; 2015. http://www.who.int/reproductivehealth/publications/unsafe_abortion/abortion-task-shifting/en/. Accessed 15 May 2018.

  23. Guldbrandsson K, Stenström N, Winzer R. The DECIDE evidence to recommendation framework adapted to the public health field in Sweden. Health Promot Int. 2016;31:749–54.

    Article  PubMed  Google Scholar 

  24. Rosenbaum S. Improving the user experience of evidence: a design approach to evidence-informed health care. Oslo: The Oslo School of Architecture and Design; 2011. https://brage.bibsys.no/xmlui/handle/11250/93062. Accessed 15 May 2018.

  25. Choudhry NK, Stelfox HT, Detsky AS. Relationships between authors of clinical practice guidelines and the pharmaceutical industry. JAMA. 2002;287:612–7.

    Article  PubMed  Google Scholar 

  26. Schunemann HJ, Osborne M, Moss J, Manthous C, Wagner G, Sicilian L, et al. An official American Thoracic Society Policy statement: managing conflict of interest in professional societies. Am J Respir Crit Care Med. 2009;180:564–80.

    Article  PubMed  Google Scholar 

  27. Boyd EA, Bero LA. Improving the use of research evidence in guideline development: 4. Managing conflicts of interests. Health Res Policy Syst. 2006;4:16.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Guyatt G, Akl EA, Hirsh J, Kearon C, Crowther M, Gutterman D, et al. The vexing problem of guidelines and conflict of interest: a potential solution. Ann Intern Med. 2010;152:738–41.

    Article  PubMed  Google Scholar 

  29. Schunemann HJ, Al-Ansary LA, Forland F, et al. Guidelines International Network: principles for disclosure of interests and management of conflicts in guidelines. Ann Intern Med. 2015;163:548–53.

    Article  PubMed  Google Scholar 

  30. World Health Organization. Maternal Mortality. Fact Sheet No. 348. Geneva: World Health Organization; 2014. http://www.who.int/mediacentre/factsheets/fs348/en/index.html. Accessed 15 May 2018.

  31. Blencowe H, Cousens S, Bianchi Jassir F, Chou D, Mathers C, et al. National, regional, and worldwide estimates of stillbirth rates in 2015, with trends from 2000: a systematic analysis. Lancet Glob Health. 2016;4:e98–e108.

    Article  PubMed  Google Scholar 

  32. Fisk NM, McKee M, Atun R. Relative and absolute addressability of global disease burden in maternal and perinatal health by investment in R&D. Tropical Med Int Health. 2011;16:662–8.

    Article  Google Scholar 

  33. Downe S, Finlayson K, Tunçalp Ӧ, Metin Gülmezoglu A. What matters to women: a systematic scoping review to identify the processes and outcomes of antenatal care provision that are important to healthy pregnant women. BJOG. 2016;123:529–39.

    Article  PubMed  CAS  Google Scholar 

  34. Oxman AD, Fretheim A, Lavis JN, Lewin S. SUPPORT Tools for evidence-informed health Policymaking (STP). 12. Finding and using research evidence about resource use and costs. Health Res Policy Syst. 2009;7(Suppl 1):S12.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Oxman AD, Lavis JN, Fretheim A, Lewin S. SUPPORT Tools for evidence-informed health Policymaking (STP). 17. Dealing with insufficient research evidence. Health Res Policy Syst. 2009;7(Suppl 1):S17.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Mangham-Jefferies L, Pitt C, Cousens S, Mills A, Schellenberg J. Cost-effectiveness of strategies to improve the utilization and provision of maternal and newborn health care in low-income and lower-middle-income countries: a systematic review. BMC Pregnancy Childbirth. 2014;14:243.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Prost A, Colbourn T, Seward N, Azad K, Coomarasamy A, Copas A, et al. Women's groups practising participatory learning and action to improve maternal and newborn health in low-resource settings: a systematic review and meta-analysis. Lancet. 2013;381:1736–46.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Oxman AD, Lavis JN, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP) 10: Taking equity into consideration when assessing the findings of a systematic review. Health Res Policy Syst. 2009;7(Suppl 1):S10.

    Article  PubMed  PubMed Central  Google Scholar 

  39. O'Neill J, Tabish H, Welch V, Petticrew M, Pottie K, Clarke M, et al. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. J Clin Epidemiol. 2014;67:56–64.

    Article  PubMed  Google Scholar 

  40. Tripathy P, Nair N, Barnett S, Mahapatra R, Borghi J, Rath S, et al. Effect of a participatory intervention with women’s groups on birth outcomes and maternal depression in Jharkhand and Orissa, India: a cluster-randomised controlled trial. Lancet. 2010;375:1182–92.

    Article  PubMed  Google Scholar 

  41. Houweling TA, Tripathy P, Nair N, Rath S, Rath S, Gope R, et al. The equity impact of participatory women's groups to reduce neonatal mortality in India: secondary analysis of a cluster-randomised trial. Int J Epidemiol. 2013;42:520–32.

    Article  PubMed  PubMed Central  Google Scholar 

  42. The SURE Collaboration. Engaging and informing stakeholders. In: SURE Guides for Preparing and Using Evidence Briefs for Policy. Version 3.0 [updated May 2014]: The SURE Collaboration; 2014. http://www.who.int/evidence/sure/en/. Accessed 15 May 2018.

  43. Colvin CJ, de Heer J, Winterton L, Mellenkamp M, Glenton C, Noyes J, et al. A systematic review of qualitative evidence on barriers and facilitators to the implementation of task-shifting in midwifery services. Midwifery. 2013;29:1211–21.

    Article  PubMed  Google Scholar 

  44. Schad M, John J. Towards a social discount rate for the economic evaluation of health technologies in Germany: an exploratory analysis. Eur J Health Econ. 2012;13:127–44.

    Article  PubMed  Google Scholar 

  45. Beauchamp TL, Childress JF. Principles of Biomedical Ethics. Oxford: Oxford University Press; 2009.

    Google Scholar 

  46. Saarni SI, Hofmann B, Lampe K, Lühmann D, Mäkelä M, Velasco-Garrido M, et al. Ethical analysis to improve decision-making on health technologies. Bull World Health Organ. 2008;86:617–23.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Downe S, Finlayson K, Tunçalp Ö, Gülmezoglu AM. Factors that influence the uptake of routine antenatal services by pregnant women: a qualitative evidence synthesis. Cochrane Database Syst Rev. 2016;10:CD012392.

    Google Scholar 

  48. Fretheim A, Munabi-Babigumira S, Oxman AD, Lavis JN, Lewin S. SUPPORT tools for evidence-informed policymaking in health 6: Using research evidence to address how an option will be implemented. Health Res Policy Syst. 2009;7(Suppl 1):6.

    Article  Google Scholar 

  49. The SURE Collaboration. Identifying and addressing barriers to implementing the policy options. In: SURE Guides for Preparing and Using Evidence Briefs for Policy. Version 3.0 [updated May 2014]: The SURE Collaboration; 2014. http://www.who.int/evidence/sure/guides/en/. Accessed 14 May 2018.

  50. Memon ZA, Khan GN, Soofi SB, Baig IY, Bhutta ZA. Impact of a community-based perinatal and newborn preventive care package on perinatal and neonatal mortality in a remote mountainous district in Northern Pakistan. BMC Pregnancy Childbirth. 2015;15:106.

    Article  PubMed  PubMed Central  Google Scholar 

  51. The SURE Collaboration. Clarifying uncertainties and needs for monitoring and evaluation. In: SURE Guides for Preparing and Using Evidence Briefs for Policy. Version 3.0 [updated May 2014]: The SURE Collaboration; 2014. http://www.who.int/evidence/sure/guides/en/. Accessed 15 May 2018.

  52. Oxman AD, Bjørndal A, Becerra-Posada F, Gibson M, Block MA, Haines A, et al. A framework for mandatory impact evaluation to ensure well informed public policy decisions. Lancet. 2010;375:427–31.

    Article  PubMed  Google Scholar 

  53. Petkovic J, Welch V, Jacob MH, Yoganathan M, Ayala AP, Cunningham H, et al. The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review. Implement Sci. 2016;11:162.

    Article  PubMed  PubMed Central  Google Scholar 

  54. iEtD – Interactive Evidence to Decision Frameworks. https://ietd.epistemonikos.org. Accessed 9 May 2018.

  55. GRADEpro GDT. https://gradepro.org/. Accessed 9 May 2018.

Download references

Acknowledgements

Members of the GRADE Working Group who contributed to writing this paper include Jenny Moberg, Andrew Oxman, Pablo Alonso Coello, Holger Schünemann, Gordon Guyatt, Sarah Rosenbaum, Angela Morelli, Elie Akl, Claire Glenton, Metin Gulmezoglu, Signe Flottorp, Simon Lewin, Reem A. Mustafa, Gabriel Rada, Jasvinder Singh, Erik von Elm, Josh Vogel and Joseph Watine.

We would like to acknowledge Tess Lawrie and Ӧzge Tuncalp for their comments, particularly on the EtD framework; Philippe Duclos, Daria O’Reilly, Melanie Marti, Jean-Eric Tarride and Shaun Treweek for their comments and suggestions; and Andrea Cervera Alepuz for her help with submission of the manuscript.

Funding

Work on this article was partially funded by the European Commission FP7 Programme (grant agreement 258583) as part of the DECIDE project. Sole responsibility lies with the authors and the European Commission is not responsible for any use that may be made of the information contained therein.

SL receives additional funding from the South African Medical Research Council.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

Author information

Authors and Affiliations

Authors

Consortia

Contributions

JM and ADO wrote the first draft of this manuscript and JM, ADO, SR, HS, GG, SF, CG, SL, AM, GR, PAC, EA, MG, RAM, JS, EvE, JV and JW contributed to the writing. All authors read and commented on the results and conclusions in the manuscript, and gave written agreement of their authorship. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Jenny Moberg.

Ethics declarations

Competing interests

All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare: support from the European Commission FP7 Programme (grant agreement 258,583), no financial relationships with any organisations that might have an interest in the submitted work in the previous 3 years, nor other relationships or activities that could appear to have influenced the submitted work. Authors are members of the GRADE Working Group and the DECIDE project. PAC is supported by a Miguel Servet investigator contract from the Instituto de Salud Carlos III (CP09/00137). JM affirms that the manuscript is an honest, accurate and transparent account of the study being reported, that no important aspects of the study have been omitted, and that any discrepancies from the study as planned have been explained.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:

Evidence to Decision Frameworks: Terminology. (DOCX 32 kb)

Additional file 2:

Evidence to Decision Framework for women’s groups. (DOCX 332 kb)

Additional file 3:

GRADE Recommendation to decision (RtD) presentation of an evidence to decision (EtD) framework for a health system and public health decision. (DOCX 29 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moberg, J., Oxman, A.D., Rosenbaum, S. et al. The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Sys 16, 45 (2018). https://doi.org/10.1186/s12961-018-0320-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12961-018-0320-2

Keywords