Research Methods & Reporting

Assessing equity in systematic reviews: realising the recommendations of the Commission on Social Determinants of Health

BMJ 2010; 341 doi: http://dx.doi.org/10.1136/bmj.c4739 (Published 13 September 2010) Cite this as: BMJ 2010;341:c4739
  1. Peter Tugwell, professor1,
  2. Mark Petticrew, chair in public health evaluation2,
  3. Elizabeth Kristjansson, associate professor3,
  4. Vivian Welch, research coordinator4,
  5. Erin Ueffing, coordinator5,
  6. Elizabeth Waters, Jack Brockhoff chair of child public health6,
  7. Josiane Bonnefoy, assistant professor7,
  8. Antony Morgan, associate director8,
  9. Emma Doohan, project manager8,
  10. Michael P Kelly, director8
  1. 1Department of Medicine and Institute of Population Health, University of Ottawa, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
  2. 2Department of Social and Environmental Health Research, Faculty of Public Health and Policy, London School of Hygiene and Tropical Medicine, London, UK
  3. 3School of Psychology, Institute of Population Health, University of Ottawa
  4. 4Ottawa Hospital Research Institute, Institute of Population Health, University of Ottawa
  5. 5Campbell and Cochrane Equity Methods Group, Institute of Population Health, University of Ottawa
  6. 6The McCaughey Centre, Melbourne School of Population Health, University of Melbourne, Melbourne, Australia
  7. 7School of Public Health, Faculty of Medicine, University of Chile, Santiago, Chile
  8. 8Centre for Public Health Excellence, National Institute for Health and Clinical Excellence, London, UK
  1. Correspondence to: P Tugwell tugwellb{at}uottawa.ca
  • Accepted 13 June 2010

A group from the Cochrane Collaboration, Campbell Collaboration, and the World Health Organization Measurement and Evidence Knowledge Network has developed guidance on assessing health equity effects in systematic reviews of healthcare interventions. This guidance is also relevant to primary research

Summary points

  • The Commission on Social Determinants for Health has recommended assessment of health equity effects of public policy decisions

  • This article provides guidance on assessing equity for users and authors of systematic reviews of interventions

  • Particular challenges occur in seven components of such reviews: (1) developing a logic model, (2) defining disadvantage and for whom interventions are intended, (3) deciding on appropriate study design(s), (4) identifying outcomes of interest, (5) process evaluation and understanding context, (6) analysing and presenting data, and (7) judging applicability of results

  • Greater focus on health equity in systematic reviews may improve their relevance for both clinical practice and public policy making

Background to health inequalities and inequities

The terms health inequalities and health inequities are used in different ways in different societies and by different authors.1 We use the term inequity in preference to inequality, following the definitions articulated by Whitehead.2 She describes health inequality as: “measurable differences in health experience and health outcomes between different population groups—according to socioeconomic status, geographical area, age, disability, gender or ethnic group.” She defines health inequity as differences in opportunity for different population groups which result in, for example, unequal life chances, access to health services, nutritious food, adequate housing, etc. These differences may be measurable; they are also judged to be unfair and unjust.2 The World Health Organization’s Commission on the Social Determinants of Health (CSDH) has further defined health equity as “the absence of unfair and avoidable or remediable differences in health among social groups.”3

The need for adequate reporting of health equity effects

The CSDH final report recognised that tackling health inequities requires a firm evidence base.4 The Measurement and Evidence Knowledge Network of the CSDH recommended systematic reviews (which use explicit methods to summarise evidence from multiple primary studies) as one source of evidence for public policy decisions about equity.5 In response to this recommendation, the Cochrane Collaboration and Campbell Collaboration formed a working group to investigate the necessary tailoring of methods of systematic reviews. The Cochrane Collaboration aims to help policy makers, practitioners, and the public make informed decisions about health interventions by preparing, updating, and promoting the accessibility of systematic reviews. The Campbell Collaboration has a focus on evidence related to the wider determinants of health, covering systematic reviews of the effectiveness of social and behavioural interventions in education, crime and justice, and social welfare.6

Process of developing recommendations

This article is the product of four years of methodology research and meetings of the Campbell and Cochrane Equity Methods Group and the Measurement and Evidence Knowledge Network, which included international leaders in systematic reviews and health equity. We held four working sessions (see box 1) with mixed methods experts, social scientists, economists, experts in systematic reviews, experts in public health and health equity, experts from low and middle income countries, and policy advisers who use systematic reviews. For the evidence base, we drew heavily on the extensive work done by the members of the Measurement and Evidence Knowledge Network, who participated in our working sessions. We developed draft guidance, with panel members reviewing the evidence and then working through each component at the meetings to allow input from all participants. Written comments were solicited from those unable to attend. After these meetings and publication of the Measurement and Evidence Knowledge Network’s final report,5 the coordinating group (the authors of the present paper) held five conference calls to revise the challenges and recommendations and prepare the present paper.

Box 1: Timeline of development of recommendations for reporting of health equity effects

  • 2005—Launch of Commission on Social Determinants of Health (CSDH)

  • 2005—Measurement and Evidence Knowledge Network (MEKN) established

  • April 2006—First MEKN meeting (Santiago)

  • December 2006—First Campbell and Cochrane Equity Methods Group meeting (Oslo)

  • February 2007—Second Campbell and Cochrane Equity Methods Group meeting (Ottawa)

  • March 2007—Second MEKN meeting (London)

  • May 2007—Third Campbell and Cochrane Equity Methods Group meeting (London)

  • October 2007—MEKN makes “evidence-based recommendations” to CSDH on reviews of equity

  • August 2008—Final CSDH report

  • October 2008—Fourth Campbell and Cochrane Equity Methods Group meeting (Freiburg)

  • 2009—MEKN and Cochrane and Campbell Equity Working Group discussions

Challenges and recommendations

We identified challenges in seven components of systematic reviews of intervention studies that address equity: (1) developing a logic model, (2) defining disadvantage and for whom interventions are intended, (3) deciding on appropriate study design(s), (4) identifying outcomes of interest, (5) process evaluation and understanding context, (6) analysing and presenting data, and (7) judging applicability of results (box 2).

Box 2: Recommendations for applying the equity lens to systematic reviews

1. Develop a logic model
  • Equity oriented systematic reviews should include a logic model to elucidate hypotheses for how the intervention (whether a policy or a programme) was expected to work, and how factors associated with disadvantage (social stratification) might interact with the hypothesised mechanisms of action. Reviews should incorporate input from relevant stakeholders in defining the research question(s) and developing the logic model.

    • Example—Logic models were developed by the Canadian Collaboration for Immigrant and Refugee Health to guide systematic reviews on interventions for immigrants and refugees.7

2. Define disadvantage and for whom interventions are intended
  • Equity oriented systematic reviews should define population selection criteria based on the question being asked. The reviewer must consider whether a group is truly disadvantaged in the study setting. In the case of targeted interventions, the population sample should be restricted to disadvantaged populations or settings in which most people are disadvantaged. In the case of universal interventions, the reviewer must be able to present data that are stratified by one or more axes of differentiation. When data on disadvantage are not available, proxy measures may be considered.

    • Example—Baseline nutritional status was identified as a proxy for socioeconomic disadvantage in a Campbell-Cochrane systematic review of a school feeding programme.8

3. Decide on the appropriate study design(s)
  • Equity oriented systematic reviews should define selection criteria for study designs according to their “fitness for purpose” rather than following an evidence hierarchy.9 The rationale for the fitness for purpose should be clearly stated and explained.

    • Example—A systematic review of the effects of tobacco pricing on smoking behaviour did not find controlled trials but did find informative observational studies. Nine of 42 studies examined aspects of equity (such as lower versus higher income smokers, ethnicity): these suggested that pricing might have a greater effect in people with lower incomes.10 This observational evidence base is informative about the differential effects of a major tobacco control intervention, whereas reviewing the evidence from randomised controlled trials alone would produce an “empty review”—a review with little to say about the policies’ effects.

4. Identify the appropriate outcomes
  • Equity oriented systematic review outcomes should be chosen based on importance and relevance of outcomes across “PROGRESS+” categories (place of residence, race/ethnicity, occupation, gender, religion, education, socioeconomic status, and social capital—plus age, sexual orientation, and disability).

    • Example—Including “Return to work” as an outcome after tuberculosis treatment might not be meaningful to a person who has little chance of employment because of social disadvantage.

5. Evaluate processes and understand context
  • In equity oriented systematic reviews, a process evaluation should be undertaken, using qualitative methods to assess why, how, when, and under what circumstances an intervention is most likely to be effective. This requires extracting sufficient information from primary studies, and possibly obtaining additional grey literature on the intervention. Furthermore, systematic reviews could include additional historical and contemporary material to enrich an analysis of contextual factors that may enhance or limit the effectiveness of the intervention.

    • Example—In school feeding programmes, was the energy value of the food supplements sufficient to change outcomes?8

6. Analyse and present the data
  • Equity oriented systematic reviews should analyse data on gaps, gradients, and targeted interventions based on the fitness for purpose of the summary measure and availability of data (see Evans et al11 for a thorough discussion of gap and gradient analysis). Where possible, both relative and absolute measures should be presented.12

    • Example—The harvest plot can be used to analyse the presence of gradients in effect size from complex and diverse studies.13

7. Discuss applicability of findings
  • Equity oriented systematic reviews should discuss the applicability, transferability, and external validity of findings according to accepted criteria as well as consider context (such as using theory and judgment). Thorough attention to understanding context and process evaluation will aid judgments about applicability.

    • Example—A Cochrane review assessing the equity implications of training lay health workers concluded that, even though 32 of 48 studies were conducted in high income countries, their findings might well be applicable to lower income countries more generally because the findings were consistent across all settings.14

1. Developing a logic model

All too often, associations of dubious relevance are found, and intermediate or surrogate outcomes are reported that have unclear relationships to the critical equity health and wellbeing endpoints. We recommend that a logic model be developed; this is a schematic that shows the hypothesised relation between interventions and their intended outcomes.15 For example, it can describe how policy interventions may work in different populations. Logic models should address societal and contextual factors that may influence the successful implementation of an intervention.16

2. Defining disadvantage and for whom interventions are intended

To tackle the social determinants of health, the social structure in which the interventions occur must be understood. The Measurement and Evidence Knowledge Network report stresses key axes of social stratification (class, status, education, occupation, income and assets, gender, race, ethnicity, caste, tribes, religion, national origins, age, and residence), while the Campbell and Cochrane Equity Methods Group and the Cochrane Public Health Review Group use the acronym PROGRESS (place of residence, race/ethnicity, occupation, gender, religion, education, socioeconomic status, and social capital).17 18 19 Additional PROGRESS axes such as age, sexual orientation, and disability have been proposed.20

These stratifiers interact, overlap, and cluster together in their implications. Their implications on inequities are dependent on context, so authors of equity oriented systematic reviews must strive to understand and explore the mediating effect of context. When details on relevant axes of social differentiation are not reported, proxy measures can be considered.

Reviewers must also assess whether the studied intervention is universal or targeted. Universal approaches aim at the whole population, but may have differential effects at different levels of the socioeconomic gradient. Targeted interventions are aimed at specific groups, usually the most disadvantaged. Both are relevant, but the reviewer must carefully assess each study’s definition of disadvantage and whether the “disadvantaged” individuals are truly disadvantaged in that context. For universal interventions, the reviewer needs to analyse outcomes, where possible with data from the primary studies, by categories of disadvantage (such as high or low income) to determine whether differential effects exist. This is not relevant for targeted interventions, but these interventions also require a thoughtful approach. One concern is that stigmatisation may result from being targeted. To reduce stigmatisation (and for other ethical and practical reasons), some interventions may target groups instead of individuals (such as whole classes of pupils for school feeding programmes).

3. Deciding on the appropriate study design(s)

The Cochrane Handbook reflects the founding philosophy of the Cochrane Collaboration that the evidence be searched systematically: it “focuses particularly on systematic reviews of randomised trials because they are more likely to provide unbiased information than other study designs.”21 However, the recent chapters on non-randomised studies and public health and health promotion retain the emphasis on the pivotal systematic searching but extend this focus to ensure the evidence optimally reflects the context in which programmes are implemented.22 Likewise, the Measurement and Evidence Knowledge Network report states: “Taking an evidence-based approach does not mean relying on or privileging only one kind of method, such as the randomised controlled trial. It does not mean that there is only one hierarchy of evidence, and it does not mean an epistemological rejection of subjective positions or methods.”5

When moving beyond establishing the efficacy of drugs and vaccines, narrow inclusion criteria that focus only on randomised controlled trials risk overlooking relevant studies. This will exclude, for example, most upstream interventions that address health inequities. Most population-level interventions have not yet been subject to controlled studies, but other designs do provide an evidence base to inform practice and policy, as the tobacco pricing example in box 2 shows.10 Systematic reviewers must therefore consider the “fitness for purpose” of criteria for evidence inclusion, considering which study designs will provide meaningful evidence to answer the review question.

4. Identifying the appropriate outcomes

The Cochrane Handbook recommends that systematic reviews include all outcomes likely to be meaningful to the public, practitioners, and policy makers.21 In the case of equity-focused reviews, outcomes important to disadvantaged people should be included when available. In specifying important outcomes, attention should be given to measures of wellbeing and not simply to measures of morbidity and mortality. Although much work has been done recently to produce better measures of wellbeing,23 24 their application in intervention studies in the main remains atheoretical. Given the complex nature of wellbeing, the Measurement and Evidence Knowledge Network report calls for reviewers to take a more robust approach to the conceptualisation of it and its associated constructs (such as quality of life, social cohesion, and community integration) so that important links between wellbeing and physical outcomes can be made explicit and more easily understood.25

Intermediate outcomes also may be appropriate, but only insofar as they predict important health outcomes,26 and this link should be shown in the logic model (No 1 above); data on adverse effects should always be included where available.

5. Process evaluation and understanding context

Another challenge is ensuring appropriate assessments of process and implementation and of wider contextual issues that could explain the study findings.

Process evaluation assesses the intervention’s quality and any interaction with disadvantage. It elucidates how the intervention or programme was delivered, the mechanisms of effect, for whom it worked, in which respects, and under what circumstances.21 Diverse methods are available to conduct a process evaluation, including structured checklists and qualitative analysis of the context and quality of the intervention as delivered. The fidelity of an intervention describes the extent to which the programme was delivered as planned, and whether the plans were sufficient to achieve the desired outcomes. The population to whom the intervention is actually delivered is also important, since the failure of programmes to reach the poorest in society is well documented.27

Understanding context—The context within which a study is undertaken is vital to effectiveness; changes in the wider economic context (such as recession) can override the impact of smaller scale interventions (such as community kitchens). Furthermore, intervention effectiveness may differ beyond the contexts in which the intervention was originally developed. Thus, reviewers should always carefully extract, synthesise, and discuss evidence on the context’s impact on effectiveness and implications for transferring the intervention to other settings. However, in bringing different sources of evidence together, they must ensure that the different study designs are clearly linked with the appropriate research question articulated in the study’s protocol. In this instance, process evaluations will not be competing with randomised controlled trials (from a hierarchical point of view) but can be assessed against the same broad principles of evidence based public health, including transparency of approach and relevance or applicability to the context of the population being studied.

6. Data analysis and presentation

Equity oriented reviews need to present evidence on outcomes in disadvantaged populations. The Measurement and Evidence Knowledge Network report reviews three ways in which health inequities are conventionally described—health disadvantage, health gaps, and health gradients.5 Health disadvantage focuses on differences between distinct segments of the population or between societies. The health gaps approach focuses on the differences between the most disadvantaged and other groups (often the least disadvantaged). The health gradient approach addresses health differences that exist across the whole population spectrum—that is, it focuses on systematically patterned gradients in health inequalities. The Measurement and Evidence Knowledge Network focused on the gradient approach because it considered a society-wide approach to health inequities.

The choice of summary measure for gradient and gap analysis and presentation in systematic reviews needs to fit the research question and be comprehensible and useable for the target audience. Subgroup comparisons should be specified a priori.21 Analysis needs to consider that disadvantaged people may be under-represented in the primary studies.27

7. Applicability of findings

The key component of applicability is the transferability of findings: will effectiveness in controlled conditions transfer to real world settings,28 how much has cultural or political context shaped the original studies’ designs and their subsequent interpretation,5 and will interventions that are effective in one setting work in different contexts? Thus, applicability relates to the context within which the primary data were collected and the setting to which they will be extrapolated.29 Conditional cash transfers are one example: they worked well in Latin America, where there is adequate infrastructure, but have failed in sub-Saharan Africa.30

Conclusions

In this article we provide guidance to assist authors conducting and writing equity oriented systematic reviews, editors and reviewers considering such articles for publication, and readers critically appraising published articles.

The strengths of our approach are inclusiveness (by enabling geographical, methodological, and disciplinary diversity) and assessment of the feasibility of the guidance using published systematic reviews as case studies. It also places the focus strongly on the quality of primary studies, and on extending the range of information to be extracted from them. This requires a paradigm shift in the generation and synthesis of evidence.

The limitations of our approach include the focus on Cochrane and Campbell reviews, which comprise only 20% of published systematic reviews.31 The selection of participants in the meetings was based on purposive and opportunistic invitations to individuals with expertise or specific interest in equity related methods. This was not designed to be representative, but rather to identify key issues in developing guidance. Moreover, the list of issues is not exhaustive, nor is it the final word: we intend it to stimulate methodological development, discussion, and practice in this field.

Equity focused reviews require deeper investigation of primary studies. This needs great care in extracting data on outcomes and study quality, and a deeper consideration of the implementation theory, process, and context. Although these data are not always available to the systematic reviewer, it is essential to seek them, and to ask that authors of primary studies report them.

Notes

Cite this as: BMJ 2010;341:c4739

Footnotes

  • We acknowledge the contributions of participants in the equity meetings in Oslo, Ottawa, Dublin, Vancouver, and London (listed at www.equity.cochrane.org). We also thank Marion Doull, Ronald Labonte, Johan Mackenbach, Jessie McGowan, Andy Oxman, and Helen Roberts for their valuable comments on drafts of our manuscript.

  • Contributors: PT initiated the paper. All authors contributed to the development of the manuscript. All authors made significant contributions to a series of drafts of the manuscript, and all saw and approved the final version. PT had final responsibility for the decision to submit for publication; he is the guarantor for the article. There were no professional medical writers involved in this paper.

  • Funding: PT and VW have support from the Canadian Agency for Drugs and Technology for Health (CADTH) for the submitted work; the Campbell and Cochrane Equity Methods Group is funded by the Canadian Institutes of Health Research (CIHR); PT is funded by a Canadian Research Chair Tier 1 Award. VW was supported by a CIHR doctoral award. None of the funders had input or influence on the content or conclusions of this paper; the final decision to publish rested with the authors and the guarantor, PT.

  • Competing interests: PT and MP are joint convenors of the Campbell and Cochrane Equity Methods Group; EU is the salaried coordinator. VW is a research coordinator with the Methods Group. JB, AM, and MPK were members of the Measurement and Evidence Knowledge Network of the Commission for Social Determinants of Health, and ED was a project manager on the same knowledge network. EW is the coordinating editor of the Cochrane Public Health Review Group. All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that (1) PT and VW have support from the Canadian Agency for Drugs and Technology for Health (CADTH) for the submitted work; the Campbell and Cochrane Equity Methods Group is funded by the Canadian Institutes of Health Research (CIHR); PT is funded by a Canadian research chair tier 1 award; VW was supported by a CIHR doctoral award; (2) the authors have no relationships with companies that might have an interest in the submitted work in the previous 3 years; (3) their spouses, partners, or children have no financial relationships that may be relevant to the submitted work; and (4) the authors have no non-financial interests beyond those stated above that may be relevant to the submitted work.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References