Education And Debate

Using evidence to inform health policy: case study

BMJ 2001; 322 doi: http://dx.doi.org/10.1136/bmj.322.7280.222 (Published 27 January 2001) Cite this as: BMJ 2001;322:222
  1. Sally Macintyre (Sally{at}msoc.mrc.gla.ac.uk), directora,
  2. Iain Chalmers, directorb,
  3. Richard Horton, editorc,
  4. Richard Smith, editord
  1. a MRC Social and Public Health Sciences Unit, Glasgow G12 8RZ
  2. b UK Cochrane Centre, NHS Research and Development Programme, Oxford OX2 7LG
  3. c Lancet, London WC1B 3SL
  4. d BMJ, London WC1H 9JR
  1. Correspondence to: S Macintyre
  • Accepted 18 September 2000

Editorial by George Davey Smith

Doctors are exhorted to practice evidence based medicine, and the British government says that it wants public policy to be based on evidence.1 We share the government's enthusiasm, but can it become a reality rather than an aspiration? How much do we know about what works, and are researchers, government ministers, and civil servants truly committed to developing and using the best evidence?

We had an opportunity to learn about the realities of evidence based policy making when we were invited in late 1997 to become an “evaluation group” to assist the independent inquiry into inequalities in health.2 Our reflections concern the material submitted and the evaluation process, not the final report.

Summary points

The UK government is trying to create a culture of “evidence based policy” that will apply to public health and social policy as well as to health care

Submissions to a government inquiry about possible interventions to reduce health inequalities lacked evidence of effectiveness and information on costs and possible harms

There is a need to improve the evidence base used for making public policy

The independent inquiry into inequalities in health

In 1997 the minister for public health for England and Wales, Tessa Jowell, commissioned Donald Acheson, the former chief medical officer for England, to “moderate a Department of Health review of the latest available information on inequalities in health … and in the light of evidence, to conduct—within the broad framework of the Government's overall financial strategy—an independent review to identify priority areas in future policy development, which scientific and expert evidence indicates are likely to offer opportunities for Government to develop beneficial, cost effective, and affordable interventions to reduce health inequalities.”2 The report was to be submitted by the end of June 1998. Three key limitations were placed on the inquiry: recommendations had to be based on evidence, they had to be made within the framework of no increase in public spending, and they had to be made within a year.

Previous inquiries

A previous inquiry under the chairmanship of Douglas Black had been set up in 1977 under the then Labour government.3 Its recommendations were rejected by the Conservative secretary of state for health in the government that followed on the grounds that they would involve too much public expenditure and that there was no evidence of the effectiveness of what had been proposed. Thereafter “inequalities in health” was a taboo topic. But towards the end of that Conservative administration it became possible to tackle similar issues in the name of “variations in health,” although within rather strict limits.

In 1994 a group was set up to review “what the Department of Health and NHS could do about variations in health.”4 That group (of which SM was a member) commissioned a systematic review on “research on the effectiveness of health service interventions to reduce variations in health.”5 Because of the earlier, hostile government reaction to the Black report, the remit of this group and of the review it commissioned was tightly controlled and explicitly excluded recommendations on fiscal policy, benefits, employment, transport, and similar topics. However, two broader reviews of strategies to reduce inequalities in health were published in 1995. 6 7

At the same time a scientific advisory group was appointed to assist Acheson. At the first meeting of the group one of the authors (IC) suggested that an efficient first step might be to ask the three teams that had published reviews in 1995 (box) to produce a paper highlighting agreements, disagreements, and uncertainties about the research. This suggestion was not taken up.

The inquiry commissioned “input papers” from experts on 17 topics including inequalities at particular stages of life, and the role of transport, income, or housing. Each of these experts made a presentation to the group and another expert nominated by the inquiry commented on the topic.8 Submissions were also invited from interested individuals and organisations, and many were received.2

Methods

The chairman asked the four of us in December 1997 to form a group “to examine the quality of the evidence underpinning the scientific advisory group's emerging recommendations and to identify any gaps.” We had to report by the end of February 1998, which gave us a very short time. All the input papers, commentaries, other submissions, and draft minutes of the advisory group were made available to us.

Guiding premises

We wanted to concentrate on empirical evidence about the effects of interventions intended to reduce inequalities in health. There are, however, few well controlled studies designed to assess the effects of interventions on health inequalities. We therefore decided to broaden our focus to assess the extent to which the policy recommendations were either based on “less targeted” intervention studies, which nevertheless suggested that disadvantaged people in particular might be expected to benefit, or were targeted at people who had been defined by socioeconomic circumstances, sex, place of residence, ethnicity, or some other dimension along which health inequalities have been documented to occur.

We then looked for evidence not only of effectiveness but also of potential harm. In the past some well intentioned and apparently logical policy recommendations have done more harm than good (box).9 Some interventions may increase health inequalities despite improving aggregate health because higher social classes may benefit most from the intervention. For example, a dental health campaign aimed at mothers of 5 year old children in Scotland had the most impact among more educated mothers,10 and a media campaign intended to reduce socioeconomic differences in women's intake of folic acid around the time of conception was associated with an increase in differences between social classes in the use of folic acid.11

Some adverse effects of interventions in health and social care

Policy: Social work services provided to boys who were delinquents

Effect: Increase in reoffending, drug and alcohol dependence, and mental illness12

Policy: Parents advised to place infants in the prone position to sleep

Effect: Increased risk of cot death13

Policy: Bed rest recommended for a range of medical conditions

Effect: No outcomes improve when bed rest prescribed and some are worsened by it14

Policy: Separate mothers and newborn infants to reduce infection

Effect: Increase in infection and difficulties in mother-infant interaction15

Since many of the interventions recommended for reducing health inequalities are likely to have modest effects that could easily be confused with differences resulting from biases that were not controlled for, we wanted to assess whether the evidence used to support recommendations had been derived from systematic reviews of controlled (not necessarily randomised) intervention studies. For some interventions persuasive, experimentally derived evidence is available (for example, periconceptional folate supplementation16). For others, there is no reason why it could not be available; for example, at least 10 randomised trials have been done on the effectiveness of providing income support.17

For other interventions—particularly for big changes, such as using fiscal measures to modify tobacco or alcohol consumption—randomised experimental evidence is never likely to be available, but careful before and after comparisons of “naturally occurring” experiments may be indicative. The methodological principles governing the assembly and review of relevant data should be observed regardless of the most reliable study design likely to be available. There is a common view that that these principles apply only to reviews that use meta-analysis to evaluate randomised trials of clinical procedures. We believe that they should apply to efforts to minimise biases in reviews of all types of research evidence, and it is clear that such biases exist in reviews about reducing health inequalities.

Finally, we developed a matrix of criteria against which we believe policy recommendations should be judged (box). Research on the effectiveness of policies will never be more than one of the factors that must be considered by policymakers. The box shows some of the other factors likely to be relevant. In appraising policy options it seemed sensible to us to select as priorities those that are supported by evidence and argument, would also bring about benefits not related solely to health, would fit with government policy, would be unlikely to do harm, and would be easy and relatively inexpensive to implement.

Criteria used to evaluate policy recommendations

  • Supported by systematic, empirical evidence

  • Supported by cogent argument

  • Scale of likely health benefit

  • Likelihood that the policy would bring benefits other than health benefits

  • Fit with existing or proposed government policy

  • Possibility that the policy might do harm

  • Ease of implementation

  • Cost of implementation

Method of working

The 17 input papers and their accompanying commentaries and related submissions were divided among the four authors. We checked that any recommendations made in a paper were supported by the publications cited as evidence; we also checked the recommendations against other submissions to the inquiry; the reviews in 1994 and 199557; the Database of Abstracts of Reviews and Effectiveness; the Cochrane Register of Systematic Reviews; the Cochrane Controlled Trials Register; and our own knowledge and contacts.

Preliminary evaluations of each of these sets of papers were discussed by the whole group, which met four times in three months, and then redrafted by the lead evaluators in light of these discussions. Our approach was not as systematic as we would have liked (and indeed one of our recommendations would be that in inquiries of this sort systematic reviews should be commissioned or consulted at an early stage), but we had little time.

Findings

Having devised our matrix for evaluating policy proposals, we found that we were unable to use the criteria devised for evaluating the policies because the proposals lacked sufficient information.

Evidence of effectiveness?

It was disappointing to see that there was little empirical evidence about the effectiveness of strategies for reducing health inequalities. The material submitted to the inquiry contained a wealth of data documenting inequalities, and it described a growing amount of research that explored mechanisms through which these inequalities might be mediated, but there was little about effective interventions (partly reflecting the recent state of research in this field nationally and internationally). Many of the submissions to the inquiry (including the input paper on geographical inequalities in health by SM)18 consisted of wish lists of potentially useful interventions without evidence of their effectiveness in practice.

Systematic and transparent methods?

None of the input papers had a methods section explaining criteria for including or excluding studies, and we encountered several instances of partial and selective citation of evidence. We recommended to the scientific advisory group that it should recognise that it would be open to criticism if it could not describe its materials and methods more clearly than had been done in most of the input papers.

Harms, costs, and priorities?

There was little discussion in the materials submitted to the inquiry about possible harms accruing from any recommended policies; about their costs, including opportunity costs; or about their relative priority.

Which type of intervention?

The evidence for the effectiveness of suggested interventions was usually clearer for more specific, “downstream” proposals that focused on individuals (for example, smoking cessation strategies such as nicotine replacement therapy) than for more macro level, “upstream” proposals focusing on legislation or cost (for example, using fiscal policies to affect smoking prevalence). Systematic reviews and well designed evaluations have been more common in the evaluation of clinical interventions than in evaluations of social, economic, or educational policy. The fact that there is more evidence available about interventions aimed at individuals does not mean that interventions aimed at whole communities are not effective but rather reflects the paucity of good quality studies of these more “upstream” interventions.

Monitoring implementation?

The papers sent to us for evaluation rarely presented empirical evidence to support their policy recommendations. This should not be taken to mean that we were confident that no such evidence exists or that we thought that the inquiry should not propose policies formulated on the basis of logic and common sense. However, we felt that it was important to make explicit the lack of strong empirical evidence for certain recommendations and the possibility that they might not have the desired effects. Furthermore, we felt that the government should ensure that steps are taken to monitor the impact of any policies implemented. Lack of evidence of the sort we sought is not an argument for inaction. The attraction of proposals such as introducing free fruit in schools or installing smoke alarms is clear, but some interventions might not be as cost effective as others or might cause harm or increase health inequalities. It is therefore important to evaluate the consequences of the introduction of any programmes. We strongly endorse the inquiry's recommendation to undertake assessments of the impact on health inequalities of policies and programmes.

Reflections

In 1997 the incoming Labour government was keen to reduce inequalities in health, particularly as this topic had been neglected under the previous government. It wanted to obtain advice from the public health community about how to reduce inequalities, but it set clear limits about what advice it would find acceptable: the government wanted the advice quickly, the advice had to be backed by evidence about “what works,” and the recommendations had to fit with the government's policy of not increasing public expenditure. The public health and other communities responded enthusiastically, and much interesting material was produced by and for the inquiry.

But despite the presence in the United Kingdom of two relevant “industries” (research on inequalities in health and the systematic review and evaluation of health related research), there was little evidence about the likely or actual effectiveness of many of the recommendations suggested to the inquiry. There was also a striking lack of use of existing, government funded datasets and techniques for searching for relevant information and avoiding bias.

We do not wish to argue that governments cannot develop or implement policies unless they have unequivocal empirical evidence to support them. Nevertheless, we were struck by the lack of empirical evidence available for a government to base policies or decide on priorities, despite the large amount of research undertaken and published on the subject in the United Kingdom. We were also struck by the readiness of researchers to recommend policies the effectiveness of which they knew little about in contrast to their caution in interpreting the results of epidemiological or clinical evidence. We doubt that the inquiry into inequalities in health is different in these respects from other governmental inquiries.

The interaction between political imperatives and research evidence will continue to present challenges, but that is not a reason for researchers to shrink from the important part that they can and should play in the process. If researchers and those who publish their work are to rise to this challenge effectively then the cumulative nature of scientific evidence needs to become manifest in the way they operate. We were fortunate that the results of three attempts to review the evidence on the effects of interventions to reduce health inequalities had been published three years earlier, but our task, and the task of the inquiry more generally, would have been easier if these reviews had been kept up to date.

One of our recommendations was that systems should be established to ensure that the inquiry's recommendations would be reviewed regularly as new information emerged from updated systematic reviews. The Cochrane Collaboration has established a structure for doing this. In addition to the many Cochrane reviews on elements of health care that are used in treatment, other reviews relate to the effects of social and educational policies such as, for example, preschool day care19 and media interventions intended to reduce smoking in young people.20

The principles that have led to the rapid evolution of the Cochrane Collaboration should be applied more widely to provide additional infrastructures for preparing, maintaining, and publishing electronically systematic reviews of research in areas beyond health care. There are now signs that people working in education, social welfare, criminal justice and other fields relevant to public policy are ready to collaborate internationally; this collaboration has been named for Donald Campbell, the American psychologist and epistemologist. 21 22 We are pleased that a Centre for Management and Policy Studies has been set up to provide advice on how the “effectiveness ethos” can be promoted across government departments.23 Additionally, the Economic and Social Research Council has established a coordinating centre for evidence based policy and practice in the United Kingdom.

The investment of public resources in “primary” research has been substantial and remains so, but the returns on this investment remain far less than they might be, and the results of the investment are scattered and not synthesised. People faced with tasks and timescales similar to those that confronted us and other contributors to the independent inquiry would be assisted greatly if up to date systematic reviews were more readily available. Politicians and civil servants need to be aware that in many fields there are no unequivocal answers to the question “what works?”

Acknowledgments

The work of the evaluation group was supported by the Department of Health. We thank Ray Earwicker, who was a helpful and good humoured secretary to the group. We also thank George Davey Smith, editorial staff at the BMJ, and Donald Acheson and his advisers for their comments on an earlier draft of this article.

RS is editor of the BMJ; this paper was peer reviewed without his involvement. Another BMJ editor initially read the paper and selected an external reviewer, and decisions on the paper were made at an editorial committee where RS was not present.

Contributors: The four authors formed the evaluation group commissioned by Donald Acheson. The idea of writing this paper emerged collectively from the group. SM, as chair, wrote the first draft; coordinated the drafting of subsequent versions, to which all authors contributed; and is guarantor.

Footnotes

  • Competing interests IC has for the past quarter century attempted to encourage policymakers and practitioners to take account of the results of reliable research in their prescriptions and proscriptions for other people. In pursuit of this objective IC has used journal articles, chapters in books, books, formal and informal oral presentations, occasional interviews with the media, and conversations and correspondence. IC has sought and received salary support for himself and colleagues to do this. IC has claimed travel and subsistence expenses, and has been offered speaker's and author's fees, almost all of which have been paid to the organisations for which he has worked. IC has no stocks or shares, and well over 99% of his income comes from his salary as an employee of the NHS.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.