Intended for healthcare professionals

Analysis Rating quality of evidence and strength of recommendations

Incorporating considerations of resources use into grading recommendations

BMJ 2008; 336 doi: (Published 22 May 2008) Cite this as: BMJ 2008;336:1170

This article has a correction. Please see:

  1. Gordon H Guyatt, professor1,
  2. Andrew D Oxman, researcher2,
  3. Regina Kunz, associate professor3,
  4. Roman Jaeschke, clinical professor4,
  5. Mark Helfand, professor of medicine5,
  6. Alessandro Liberati, professor6,
  7. Gunn E Vist, researcher2,
  8. Holger J Schünemann, associate professor7
  9. for the GRADE working group
  1. 1Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, ON, Canada L8N 3Z5
  2. 2Norwegian Knowledge Centre for the Health Services, PO Box 7004, 0130 Oslo, Norway
  3. 3Basel Institute of Clinical Epidemiology, University Hospital Basel, 4031 Basel, Switzerland
  4. 4Department of Medicine, McMaster University, Hamilton, ON, Canada L8N 3Z5
  5. 5Portland VA Medical Center and OHSU Department of Medicine, Portland, Oregon 97201
  6. 6Università di Modena e Reggio Emilia and Agenzia Sanitaria Regionale, Regione Emilia Romagna, 40127 Bologna, Italy
  7. 7Department of Epidemiology, Italian National Cancer Institute Regina Elena, Rome, Italy
  1. Correspondence to: G H Guyatt guyatt{at}

    Guideline panellists have differing opinions on whether resource use should influence decisions on individual patients. As medical care costs rise, resource use considerations become more compelling, but panellists may find dealing with such considerations challenging

    In this last part of a series describing the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach to making recommendations we will look at how guideline panellists and clinicians can incorporate matters related to the use of resources into recommendations and practice. Clinical recommendations inevitably involve judgments about the allocation of resources, judgments commonly referred to as costs. We will deal with some of the challenges of considering costs, explain reasons for focusing on resource use rather than costs, and discuss how to incorporate considerations of resource use into recommendations.

    Cost as an outcome presents special challenges

    In one sense, cost is just another potentially important outcome—like mortality, morbidity, and quality of life—associated with alternative ways of managing patients. In addition to these clinical outcomes, an intervention may increase costs or decrease costs. However, costs differ from other outcomes in several ways (box).1

    In what way do costs differ from other health outcomes?

    • Patients receive health benefits and bear the burden of adverse health outcomes, but healthcare costs are typically shared by society as a whole (as represented by the government), employers, and patients

    • Attitudes differ as to whether costs should influence a doctor’s decision about treating individual patients

    • Healthcare costs may vary widely among and even within jurisdictions and quickly change over time

    • What societies can purchase if they forego use of healthcare resources (opportunity cost) varies widely between countries. A year’s supply of an expensive drug may pay a nurse’s salary in the US and 30 nurses’ salaries in China

    • When healthcare expenditures demand foregoing expenditures elsewhere, attitudes differ as to whether the health system, public expenditures, or society as a whole should bear the burden

    • Matters relating to resource use are highly political and may result in conflict of interest for a guideline panel (for example, panellists may have an association with industry or government)

    Firstly, for most outcomes other than costs, it is clear who gains the advantages and has to live with the disadvantages—the patient and, secondarily, his or her family. This is not true of all outcomes. Consider vaccinations, in which the entire community benefits from the herd effect, or widespread use of antibiotics that may have downstream adverse consequences of drug resistance. It is costs, however, where the matter of who pays and who gains is most prominent. Healthcare costs are often borne by society as a whole, or in a complex arrangement that involves the patient, an employer, and an insurer. Even within a society, who bears the cost may differ depending on the patient’s age (for example, under or over 65) or situation (for example, whether the patient is receiving social assistance).

    Summary points

    • Costs differ from other healthcare outcomes—costs are shared by patients, employers, and society, and opinions differ as to who should bear the burden; some people think costs should not influence doctors’ decisions; costs differ across and within jurisdictions

    • A balance sheet should inform judgments about whether the net benefits are worth the incremental costs

    • Evidence profiles should always present resource use, not just monetary values

    • A guideline panel may legitimately choose to omit costs as a consideration

    • Formal economic modelling may—or may not—be helpful

    Secondly, depending on who bears the cost, opinions on the extent that costs should influence the decision differ. If costs are borne by a third party payer (the government or another third party payer) some people would argue that the doctor’s responsibility to the patient means that costs should not influence the decision. On the other hand, a clinicians’ responsibility when caring for a patient is discharged in a broader context—resources that are used for an intervention cannot be used for something else and can prevent the health system from best meeting the needs of those it serves.

    Thirdly, costs tend to vary widely across jurisdictions. Costs of drugs are largely unrelated to the costs of producing them but more to marketing decisions and national policies. Most medicines under patent, for example, cost around twice as much in the United States as in Canada. Furthermore, costs may vary widely even within jurisdictions. Hospitals or health maintenance organisations may, for instance, negotiate special arrangements with drug companies for prices substantially lower than are available to patients or other providers. In addition, costs can vary widely over time (for example, when a drug comes off patent).

    Even when resource use remains the same, the resource implications may vary widely across jurisdictions. A year’s supply of a very expensive drug may pay a nurse’s salary in the US, six nurses’ salaries in Poland, and 30 nurses’ salaries in China. Thus, what can be bought with the resources saved if a certain drug is not purchased (the “opportunity cost”)—and the health benefits achieved with those funds—will differ greatly.2

    Lastly, people may have different ideas about which “envelope” opportunity costs fall into. A hospital pharmacy with a fixed budget considering purchase of an expensive new drug will have a clear idea of what that purchase will mean in terms of other drugs the pharmacy cannot afford. In broader settings the opportunity costs will be less clear. For instance, funding a new drug or programme will constrain resources for other public health expenditures, but will refraining from that purchase really mean the equivalent resources are available for our healthcare system? People often assume the envelope from which health care costs must come is public health spending, but might refraining from tax reductions or reducing military expenditure be legitimate alternatives for making an effective but expensive drug widely available?

    All of the points mentioned up to now make the consideration of resource allocation a far more political issue than consideration of other outcomes. Whether the panel does or does not explicitly consider matters related to resource allocation, those politics may bear on a guideline panel’s function through conflict of interest. Conflicts may arise through the association of panel members with commercial entities, the government, or other third party payers.

    A balance sheet should be used to assess benefits versus costs

    Despite the differences, approaches to resource use are similar to other outcomes because guideline panels need consider only important resource implications; decision makers require an estimate of the difference between treatment and control; and panels must make explicit judgments about the quality of the evidence regarding incremental resource use. Special considerations of using resource use as an outcome include which perspective to take, which resource implications to include, how to make judgments about the quality of the evidence, how to present these implications, and the potential usefulness—or lack of usefulness—of a formal economic model in arriving at a wise recommendation.

    A balance sheet is a simple but powerful way to present the advantages and disadvantages of the management options under consideration.3 We illustrated this in a previous article in this series where we used an evidence profile, a type of balance sheet. We did not consider resource implications in that example. Tables 1 and 2 show an evidence profile informed by a large international clinical trial4 and an associated economic analysis5 to assess the advantages and disadvantages of magnesium sulphate in women with pre-eclampsia.

    Table 1

    Summary of findings on whether clinicians should use magnesium sulphate to prevent eclampsia: clinical outcomes

    View this table:
    Table 2

    Summary of findings on whether clinicians should use magnesium sulphate to prevent eclampsia: resource use viewed from the perspective of the health system

    View this table:

    Evidence profiles should present resource use, not just monetary values

    We suggest that guideline developers should document best estimates of resource use, not best estimates of costs. Costs are a function of resources expended and the cost per unit of resource. Given the wide variability in unit costs, reporting only total costs across broad categories of resource expenditure does not provide users with the information needed to judge whether estimates of unit costs apply to their setting.

    Users of guidelines will be best informed if the guideline developers specify resources consumed by alternative management strategies. Firstly, they can judge whether the resource use reflects practice patterns in their setting. Secondly, they can focus on the items of most relevance to them (such as the drug costs for a pharmacy or the hospital costs for a hospital administrator). Thirdly, they can, if monetary values are subsequently assigned to the resources used, ascertain whether the unit costs apply in their setting. If they don’t, they can substitute unit costs that do.

    The magnesium sulphate example (tables 1 and 2) shows the importance of documenting resource use and specifying the setting. The variation in costs associated with magnesium sulphate, its administration, and the associated hospital costs across countries with high, medium, and low gross national income is great. Table 2 documents these differences, but many economic analyses will not. Unless resource use is specified, users in settings other than that on which the analysts focus cannot estimate the associated incremental costs of the intervention.

    The specific context is crucial for considering resource use

    The extreme variability in costs over time and jurisdictions has several implications. Firstly, we have argued in previous articles in this series that clinical practice guidelines should be as specific as possible about the patient population, the nature of the intervention, the comparator, and the healthcare setting; this specificity is even more crucial when bringing cost into the equation. The choice of comparator can be a particular problem in economic analyses. If the choice of comparator is inappropriate (such as no treatment rather than a less effective treatment) conclusions may be misleading.6

    Secondly, a guideline panel may legitimately not consider resource use and make recommendations solely on the basis of other advantages and disadvantages of the alternatives being considered. Resource allocation must then be considered by the ultimate decision maker—the patient and healthcare professional, an organisation such as a hospital pharmacy or a health maintenance organisation, a third party payer, or a government. Guideline panels should be explicit about this decision.

    Thirdly, if panellists do consider resource use they should—before bringing cost into the equation—decide on the quality of evidence regarding other outcomes and weigh up the advantages and disadvantages. Decisions regarding the importance of resource use will flow from this first step. For example, resource implications may be irrelevant if evidence of net health benefits is lacking. If advantages of an intervention far outweigh disadvantages, resource use is less likely to be important. Resource use usually becomes important when advantages and disadvantages are closely balanced.

    Therefore, the GRADE Working Group considered the possibility of panels issuing two recommendations, one not taking resource use into account and a second doing so. This would have the advantage of explicitness, on which GRADE places great value. We were concerned, however, that those with commercial interests in dissemination of an intervention (such as a drug company) would use the recommendation ignoring resource implications as a weapon in their battle for funds—particularly public funds. To prevent this undesirable outcome, GRADE decided that panels considering resource use should offer just one recommendation.

    A broad perspective is desirable

    Conceivably, a recommendation could be intended for a very narrow audience, such as a single hospital pharmacy, an individual hospital, or a health maintenance organisation. Alternatively, it could be intended for a health region, a country, or an international audience.

    Regardless of how narrow or broad the intended audience, guideline groups that choose to incorporate resource implications must be explicit about the perspective they are taking. For example, a guideline panel that took the individual patient’s perspective would consider only resource implications that directly affect individual patients (such as out of pocket costs) and would exclude resource implications for others (such as costs borne by the government). In European healthcare systems in which governments mostly bear the cost of health care, expenses borne directly by patients are usually minimal. Alternatively, a guideline may choose to take a societal perspective, and include all important resource implications, regardless of who bears the costs.

    In most circumstances, few guideline audiences would be happy with a perspective that is narrower than that of the entire health system. For instance, in a publicly funded health system the patient perspective would ignore most of the costs generated. A pharmacy perspective would ignore downstream cost savings resulting from adverse events (for example, stroke or myocardial infarction) prevented by a drug, and a hospital perspective would ignore outpatient costs, either incurred or prevented.7 In the private sector, where disenrollment and loss of insurance can shift the burden of costs from one system to another, estimates of resource use should include the downstream costs of all treated patients, not just those who remain in a particular health plan.

    An even broader viewpoint—that of society—is the most comprehensive perspective because it includes all costs, regardless of who bears them. This perspective is often preferable, particularly if the health intervention has a broad effect (for example, an intervention for heart failure that improves patients’ functioning and reduces the time and cost of family caregivers). Whether cost effectiveness analyses should include the implications of health effects—such as changes in earnings—is more controversial. Economic guidelines advise that such implications are presented separately, not as part of the formal cost effectiveness analysis.

    Clinicians seeing patients who are not covered by either public or private insurance may need to help them make decisions that take into account their out of pocket costs. This is particularly true when clinical advantages and disadvantages are closely balanced and out of pocket costs are substantial. In these circumstances, if a guideline panel has used the GRADE approach, clinicians can review evidence summaries such as those in tables 1 and 2. It is then the clinicians’ responsibility—either through communicating the information directly to the patient or by finding out about the patients’ situation, values, and preference—to ensure that the patient’s decision about drug consumption is consistent with his or her values and preferences.

    Judging quality of evidence for resource use

    As with evidence of rare but serious adverse effects, evidence of resource use may come from sources other than evidence of health benefits. This may be because trials of interventions do not fully report resource use; because the trial situation may not fully reflect the circumstances—and thus the resource use—that we would expect in clinical practice; because the relevant resource use may extend beyond the duration of the trial; or because resource use may vary greatly across settings.

    For resource use reported in the context of trials, criteria for quality assessment are identical to those for other outcomes, as described in the second article in this series.8 As for other outcomes of a trial, the quality of evidence may differ across different resources. For example, drug use may be relatively easy to estimate, whereas use of health professionals’ time may be more difficult, so the estimate of drug use may therefore be of higher quality. For magnesium sulphate in pre-eclampsia, we are more confident of resource use associated with the drug itself and administration of the drug than we are of use of hospital resources (table 2).

    Formal economic modelling may—or may not—be helpful

    Formal economic modelling results in cost per unit benefit achieved: cost per natural unit, such as cost per stroke prevented (cost effectiveness analysis); cost per quality adjusted life year gained (cost-utility analysis); or both cost and benefits valued in dollars (cost-benefit analysis). These summaries can help inform judgments. Unfortunately, published cost effectiveness analyses, particularly of drugs, have a high probability of being flawed or biased,9 and they are specific to the particular setting.

    Guideline groups may therefore consider developing their own formal economic model. Before guideline groups can consider this option, however, they must have the necessary expertise and resources. The larger the difference in resources consumed by the alternative management strategies, the greater the uncertainty about whether the net benefits of an intervention are worth the incremental costs, and the higher the quality of evidence regarding resource consumption, the more likely it is that a full economic model would help inform a decision. Creating an economic model may also be advisable if implementing an intervention requires large capital investments, such as building new facilities or buying expensive equipment.

    Modelling is necessary for taking into account complexities and uncertainties in calculating cost per unit benefit, but it does reduce transparency. In addition, any model is only as good as the data on which it is based. When estimates of benefits, harms, or resources used come from low quality evidence, results of any modelling exercise will be highly speculative.

    Criteria are available to assess the credence to give to results from statistical models of cost effectiveness or cost-utility.10 11 12 13 However, these models generally include a large number of assumptions and evidence of varying quality for the different estimates that are included in the model. For these reasons, we do not recommend including cost effectiveness or cost-utility models in evidence profiles. These models may, however, inform a guideline panel’s judgments, or those of governments or third part payers considering whether to include an intervention among their programmes’ benefits.

    Tables 1 and 2 provide crucial information for making estimates of the incremental cost per episode of pre-eclampsia prevented for severe and non-severe pre-eclampsia in high, middle, and low income countries (table 3). Even when—as in this case—cost effective estimates are credible, they provide no clear answers about appropriate action. Most people, however, would consider the cost per episode of eclampsia prevented to be worth the money in the case of severe pre-eclampsia. For non-severe pre-eclampsia—particularly in low income countries—the decision is more difficult. Ultimately, decision makers must weigh the relative value of preventing eclampsia against the benefits that the health system, or society, would forego in allocating resources to magnesium sulphate administration.

    Table 3

    Incremental cost for each episode of eclampsia prevented by magnesium sulphate

    View this table:

    Concluding remarks

    Clinical decision making is complex. Guidelines have the potential to help clinicians and patients with complex choices, to improve the quality of care, and to help ensure the best use of limited resources. To ensure that guidelines inform rather than misinform, they should build on the best available evidence, and guideline panels should use systematic and transparent processes to make judgments about the quality of the evidence,9 moving from the evidence to a recommendation,14 and incorporating considerations of how resources are used.

    Clinicians and their patients will be best served by guidelines that use an approach, such as the one we have described in this series, to explicitly grade the quality of evidence and the strength of recommendations. Front line clinicians or those constructing local guidelines need not replicate the work done by well resourced guideline developers. However, to make the best use of guidelines, they need an understanding of the evidence and judgments underlying a guideline. They should have access to concise summaries of recommendations, including ratings of the quality of the underlying evidence and the strength of the recommendation, and they should understand the meaning of these grades and their implications for clinical decision making.


    • This is the last in a series of five articles that explain the GRADE system for rating the quality of evidence and strength of recommendations.

    • The members of the GRADE Working Group are: Phil Alderson, Pablo Alonso-Coello, Jeff Andrews, David Atkins, Hilda Bastian, Hans de Beer, Jan Brozek, Francoise Cluzeau, Jonathan Craig, Ben Djulbegovic, Yngve Falck-Ytter, Beatrice Fervers, Signe Flottorp, Paul Glasziou, Gordon H Guyatt, Margaret Haugh, Robin Harbour, Mark Helfand, Sue Hill, Roman Jaeschke, Katharine Jones, Ilkka Kunnamo, Regina Kunz, Alessandro Liberati, Merce Marzo, James Mason, Jacek Mrukowics, Susan Norris, Andrew D Oxman, Vivian Robinson, Holger J Schünemann, Tessa Tan Torres, David Tovey, Peter Tugwell, Mariska Tuut, Helena Varonen, Gunn E Vist, Craig Wittington, John Williams, and James Woodcock.

    • Contributors: All listed authors and members of the GRADE working group helped develop the ideas in the manuscript and read and approved the manuscript. GHG wrote the first draft and collated comments from authors and reviewers for successive drafts. All other authors listed contributed ideas about structure and content, provided examples, reviewed successive drafts of the manuscript, and provided feedback. GHG is guarantor.

    • Funding: None.

    • Competing interests: All authors are involved in the dissemination of GRADE, and GRADE’s success has a positive influence on their academic career. Listed authors have received travel reimbursement and honorariums for presentations that included a review of GRADE’s approach to rating quality of evidence and grading recommendations. GHG acts as a consultant to UpToDate; his work includes helping UpToDate in their use of GRADE. HJS is documents editor and methodologist for the American Thoracic Society; one of his roles in these positions is helping implement the use of GRADE. He is supported by “The human factor, mobility and Marie Curie actions scientist reintegration European commission grant: IGR 42192—GRADE.” AL is helping the use of GRADE by different institutions within the Italian NHS and in this role he has implemented GRADE to produce clinical recommendations in oncology through Grant No 249 (2005-7), Bando Ricerca Finalizzata, Ministero della Salute, Roma, Italy.

    • Provenance and peer review: Not commissioned; externally peer reviewed.


    View Abstract