The part that evidence should play in informing public health policy and in reducing health inequalities has been a topic of recent debate among researchers and policymakers.^1–4 Some commentators however have argued that while researchers would like their research to influence policy, in practice this often does not happen because they take little account of the needs of policymakers and of the reality of the policy process. Moreover, researchers have been castigated for their political naivety, for lacking understanding of how policy is made, and for having unrealistic expectations about what research can achieve.^3,5 A particular problem, it is suggested, may be caused by their over-reliance on simplistic, linear models of decision making, though such models rarely apply even where the policy questions are ostensibly fairly clear, such as in health technology assessment.^3,5–7

All this suggests that strengthening the bridge between research and policy is an urgent priority if evidence based policy is to become the norm, rather than the exception. Although improving the accessibility of primary and secondary research evidence will help achieve this aim,² we also require a clearer understanding of how current public health evidence is viewed by users; if the evidence currently available is not valued by decision makers, then evidence based policy will be no more than an aspiration. Similarly, we need a clearer understanding of how the production of relevant evidence on health inequalities and their reduction may be fostered, if the well reported gaps in this evidence base are to be filled.

We sought to advance this understanding by seeking views and advice from senior members of the UK policy and research community, at two separate workshops. This first paper presents a summary and analysis of the discussions held at the workshop attended by policymakers. The workshop itself presented a forum for this group to reflect on how evidence influences public health policy making, and on how its relevance and utility could be improved, with specific reference to the research evidence on the production and reduction of health inequalities. A companion paper presents the researchers’ perspectives on what types of evidence have had an impact on policy.⁸ This study was carried out as part of the “Evidence Network”, a project funded by the UK’s Economic and Social Research Council to improve the social science evidence base for public policy.⁹

METHODS

A residential workshop was held at a secluded conference centre in Scotland in April 2002, at which seven participants with a senior role in policy development across a range of sectors (mainly public health, but also including education, social welfare, and health services) attended for two days to discuss the research into policy process, and how researchers can improve it (box 1).

All the UK participants were senior civil servants, and were purposively selected as being recipients and users of research evidence concerned with public health and specifically health inequalities, with a role in interpreting this evidence for ministers and other senior civil policy advisers. Two participants were from overseas (one from the United States and one from mainland Europe), selected to give an international perspective, and were again advisors to national governments. The meeting was chaired at various points during the day by each of the authors in turn, who played a neutral part—facilitating the discussion (for example, by questioning the speakers where appropriate to draw out or help develop the main points of the argument) but without participating directly in the debate. As comparatively few people work in this field, most of the participants and authors knew one another. Chatham House rules applied, that is, anonymity was assured and participants were free to speak knowing that any comments they made would not be attributed to them either by name or in any way that could be traced back to them. The participants did not permit their discussions to be tape recorded as they felt strongly their dialogue would be freer and more honest if it was not recorded. The dialogue was therefore transcribed independently by hand by three researchers, who then compared notes to produce a single agreed transcript. Two authors coded the transcript, with input from the other authors, and from this identified the main themes. These were discussed with the other authors to achieve agreement. Four focused sessions were conducted at which the authors took turns at chairing and facilitating, but did not interpose their own views. The results that are presented in this paper represent a summary of the discussions, and a description of the main themes that emerged, based on a detailed coding and analysis of the transcript.

RESULTS

After coding, the issues raised were grouped into three broad themes, as follows.

1 How evidence informs policy: influences on the use of evidence in policy process

The participants began by identifying some of the key structural, organisational, and political influences on the use of evidence on health inequalities. Three key influences were identified. Firstly, local compared with national needs for evidence were different, and so evidence at a local, micro level was often required in addition to evidence that provided a national picture. Secondly, they felt that the mechanisms and structures within which policymakers work also needed to be understood by researchers—for example, how organisations actually worked, and particularly the timescales they worked to. Finally, political influences on the use of evidence were alluded to, for example where public opinion (as compared with scientific research) could influence the adoption of particular policies or targets that need to be achieved.

Understanding day to day constraints

In general, this group felt that the structures and constraints within which policymakers work needed to be better understood by researchers. The need for action also meant that policymakers were used to making decisions without evidence, because action was often needed whether “strong” evidence was there or not. In these circumstances, research “quality” was often less crucial to policymakers who needed to make short term decisions than it was to researchers, although sound methodology was felt to be important on a longer time scale. Quality may also be outweighed by more pragmatic considerations—such as cost (box 2). Policymakers also noted that research findings were frequently used for purposes other than those intended by the researcher; for example to reinforce policies that had already been decided (referred to by one participant as “fig leaf” evidence).

The value of “a good story”

Specific techniques for getting research into policy were discussed. The timing of presentation of evidence was felt to be particularly important; while there is often a “window of opportunity” within which evidence could have an impact, it is rarely open for long. To maximise their impact on the policy process, researchers therefore needed to find out what a government department was planning to do, align the evidence accordingly, and communicate it clearly. The format in which evidence was presented was clearly important too; several participants pointed out the value of a “good story,” and in the words of a UK policy advisor:

“[What is important is] How convincingly the evidence is presented, and how interesting you make it. The face validity of a ‘good story’ is an example of how presentation style can influence politics…what ministers call ‘evidence’ is what they get from their constituents at their Saturday surgery”.

In response, others noted that, although true, this is antithetical to the idea of evidence based policymaking. However, as one UK participant pointed out, it is not a case of “either/or”: “Stories themselves can be used in a credible way along with the evidence”.

While there was some agreement that the presentation of evidence is important, one of the most convincing types of evidence was felt to be evidence on the costs of action, or inaction: one case study (cited by an international participant) referred to tobacco control policies. Here the best argument for getting governments to listen was not necessarily the direct health impacts of smoking, but the financial impacts of reducing consumption, because the reduction of smoking in population could save billions in health service expenditure on related illnesses: “What makes evidence talk?… Answer: money!” (box 2).

The “indirect insinuation” of evidence into policy

It would be a mistake however to assume that research is only heeded when purposive dissemination takes place at the right time, and in the right format. The process can of course be more ad hoc, and the insinuation of evidence into policy can also be done indirectly. One participant cited as an example the role that a research paper on excess winter deaths in Scotland played in influencing the Scottish Executive’s Healthy Homes Initiative. One of the aims of this policy was to provide central heating to all properties in the social rented sector, and to private sector properties where the head of household or spouse was aged over 60. The study reported that a 1°C decrease in mean temperature was followed by a 1% increase in deaths one week later, and the authors suggested that this resulted from the population being unable to protect themselves adequately from the effects of cold.¹⁰ In his own words, the policy advisor “stumbled across” the article and used it to “plant a seed” in the minds of his colleagues. Later, when the Warm Homes Initiative was launched, the announcement listed “health” as one of the prime motivating factors behind the policy. For this policy advisor, there were two important lessons: firstly, that explicitly citing “health improvement” as a possible outcome can be successful in moving policies up the political priority list; and secondly that an indirect approach can be a more potent way of getting evidence to affect policy than simply trying to dictate action to policymakers.

2 Policymakers’ perceptions of current evidence

The expert group went on to discuss what policy advisors need in terms of evidence in the field of health inequalities—and what they currently receive. They drew attention to the prevalence of “policy free evidence”—research that does not answer clear, or policy relevant questions (box 3), and discussed how researchers’ lack of knowledge (“researcher naivety”) of the policy environment militates against knowledge transfer between science and policy.

Plausible causal pathways: the need for research

They identified a particular problem with the type of evidence they receive: much of the available evidence on health inequalities was from far down the causal chain, and was often concerned with health behaviours and clinical issues, rather than broader social determinants of health. Moreover, this research was often explanatory, rather than evaluative, and often had little to say about inequalities, as it commonly reported on average rather than distributional effects. Information on the cost effectiveness of interventions to inform the selection of appropriate strategies was also largely absent. Aetiological and evaluative research therefore often lacked both policy relevance and an equity dimension. In addition, much research was felt to have weak underlying theoretical underpinnings, for example, plausible causal pathways were rarely considered in detail. Consequently, the results of researchers’ analyses could be impossible to interpret in terms of what is known about the ways in which health and illness are generated.

3 Looking forward: How can the policy relevance of research on health inequalities be improved?

The expert group was then asked how researchers could go about assembling the evidence that policy makers need, and where the main gaps in the evidence base lay. Participants specifically identified a need for evaluations of the effectiveness and cost-effectiveness of policy and other interventions to reduce health inequalities. Predictive research was also seen as important (for example modelling the effects of globalisation, and simple predictive models to help identify “best buys”). Our expert group also identified a need for methodological research to further develop methods for assessing the impact on health of clusters of interventions (“policy clustering”), perhaps taking the form of “systems analysis” for public health—that is, an analysis of the entire “policy system”, including analysis of interactions between policies and their outputs, and their contribution to the production of health.¹¹ Several also saw a definite need for more work to help with identifying relevant key indicators of progress toward health inequalities targets.

The “mixed economy” of evidence

The concluding discussion focused on whether it was possible and useful to achieve a consensus on the key criteria for assessing effectiveness in public health (box 4). There was much doubt among the group about the value of a “hierarchy of evidence” in public health as used in evidence based health care. They noted again the problem with “high concept” notions of evidence preferred by academics, and pointed out that in policy circles a “mixed economy” of evidence actually prevailed, in which different types of experimental and non-experimental evidence are brought to bear on policy questions. It was felt that researchers therefore need to help policymakers with managing this mixed economy; for example, to help deal with many small pieces of evidence, of variable quality, (and with many gaps), but all pointing in the same direction. It was suggested that public health researchers could perhaps learn from economists with respect to interpreting and drawing firm conclusions from mixed evidence of variable quality.

DISCUSSION

Research into the role of evidence in public policy has a long history.^3,12,13 However, the interest among policymakers in making public health policy more explicitly evidence based has led to recent debate about the extent to which this can be achieved in practice, and about what “evidence based policy” means to policy makers and researchers. This is an area that has received much less attention, and this focused workshop gave public health policymakers the opportunity to articulate their views on, and experiences of, evidence based policy, with specific consideration of the evidence around the reduction of health inequalities. These experiences add weight to the argument that policy has simply outstripped both basic social science and monitoring frameworks.¹⁴

They also map quite closely onto a model of the policy process that emphasises enlightenment rather than rational, linear processes.¹⁵ This is perhaps not surprising as there is now considerable evidence that rational, linear models of decision making are probably the exception rather than the norm. Over the past 25 years Weiss has highlighted how the impact of research evidence is often diffuse, and more recent commentators can still point to the lack of empirical evidence that interventions based on simple linear models are effective.^15,16 On the positive side, there is at least evidence that multifaceted and organisationally focused approaches may be effective¹⁷; perhaps offering support for the participants’ view that researchers need to understand better the structural constraints under which policy is produced if their research is to find its target.

However, the expert group identified additional issues that extended and deepened our understanding of the policy process. For example, while methodological soundness and a degree of uncertainty about research findings are at the core of most research, these issues can simply cloud the message for policymakers. Specific gaps were also uncovered, in particular the need for information on the distributional effects of interventions, and on cost effectiveness. This view that there is a significant imbalance in the evidence base—with little evidence from studies of effectiveness, as compared with descriptive studies—has been supported more recently in the UK by the Wanless Report, which aimed to make recommendations on the means of improving public health and tackling health inequalities. Wanless reported that “…there is generally little evidence about the cost-effectiveness of public health and preventative policies or their practical implementation,”¹⁸ reflecting the findings of an earlier extensive review of the evidence base.¹⁹

The impact of this “inverse evidence” law is described elsewhere by Nutbeam; it means that there is considerable evidence about risk factor modification, but relatively little about some of the wider social economic and environmental determinants of health, so that with respect to health inequalities we too often have the right answers to the wrong questions.²⁰ Crucially, the more general problem of “lack of evidence” may extend beyond issues of effectiveness: participants also pointed to a need for stronger theoretical underpinnings for existing quantitative research and for methodological work on policy evaluation, to take account of plausible causal pathways to ill health.

It should be noted however that even if such evidence were available, it would not necessarily be acceptable to policymakers. It has been suggested that this may be particularly the case with evidence on health inequalities. For example, the issue of health inequalities was well aligned with the political aims of the newly elected Labour government in 1997, though not with those of the previous administration²⁰ (although there was also an extensive programme of research on health inequalities in the UK before Labour took power).²¹

What messages can policymakers give to researchers?

As well as providing a view of the evidence base on inequalities from a policy perspective, the themes identified by this group provide messages for researchers on how to increase the impact of their research on policymakers. They highlighted the importance of simplicity, with a particular need to keep research messages simple and unclouded by jargon; they also emphasised that research should be “timely”, referring to the need to align evidence to users’ timescales. “Relevance” was also key, by which they meant that researchers needed to ensure that the evidence they produce really is relevant to ongoing policy debates, and clarity was also seen as important, which to some meant that researchers should temper their apparent obsession with bias, ambiguity, and uncertainty in the evidence base. As outlined above, information on the costs and distributional effects of interventions was also seen as crucial, but absent.^2,19

While clarity, relevance, timeliness, and so on are important, they are not of course the only influences on how, or whether evidence is used. For research findings to have any impact at all, systems need to be in place to allow relevant scientific information to be identified, synthesised, and disseminated appropriately. In many (perhaps most) countries these systems may not be well developed; the absence of basic resources (including a lack of training, funds, information centres, and web based and other sources of information) certainly hamper the dissemination and communication of research findings. This may be the case in many developing countries, for example.²² Furthermore, many of the criticisms made by these policy advisors may not even be accepted by researchers; in particular they may not agree that highlighting the biases and ambiguities in research evidence is necessarily a problem. These issues were pursued further with researchers themselves in subsequent workshops, the results of which are reported in paper 2.⁸

There are many similarities between the issues raised by participants about the reality of basing policy on evidence, and those which have been raised by Davies et al and Weiss.^13,15 Some of the messages from this workshop are also similar to those that emerged from workshops held by Canadian researchers examining the impact of health services research.²³ However, while there is now good evidence about how the impact of research may be increased with respect to clinical interventions, there is still very little information as to how this may be achieved in the case of evidence on health inequalities, where the issues are in many respects more complex. Moreover the type of evidence mainly being sought by participants in this study—that is, evidence on the means of addressing the social determinants of health inequalities—is sparse, and is often less accessible than clinical evidence. In short, less ambiguity and greater timeliness in knowledge transfer may well be necessary, but this may often be difficult to achieve given the timescales needed to collect evidence on interventions to reduce inequalities, and the practical and methodological difficulties of carrying out the necessary economic and other evaluations.

In reflecting on the limitations of this study, we need to consider whether bias was introduced in the selection of participants and the conduct of the focused discussion. In particular, all the selected policy advisers were known to one or more of the researchers, and in some cases, authors had previously received funding from the government departments to which the participants were attached. In addition, the researchers facilitating the workshop were senior and well known in this field and there was a potential danger that they would themselves dominate the discussion and even impose their views on the participants. The discussions summarised in this paper were therefore collected in an unusual environment, in which senior researchers and senior policymakers were engaged in an extensive in depth discussion, and it could be suggested that the issues raised may have been different had the discussions been chaired by “neutral,” or more unknown researchers. From the outset, we tried to guard against the discussion being unduly influenced by the researchers, by adhering to a strict set of rules of conduct for the facilitation of each session, explained to participants at the beginning. Researchers were not allowed to express any views or enter into the discussion, beyond asking agreed questions or raising points for clarification. The transcripts provide evidence that these rules were adhered to: the policy advisers did virtually all the talking and there were no instances of researchers butting in and giving their own opinions. In the event, the policy advisers’ main concern was about confidentiality: they stated firmly that if the proceedings were tape recorded, their responses would be more guarded and less free ranging than if manual recording and Chatham House rules were adopted. On the issue of having “neutral” or more junior researchers, we had to weigh up whether we would achieve the recruitment of senior policy advisers, and their retention for the full two days of the workshop, if the invitation came from unknown investigators. From past experience, we judged that we needed to be identified and involved with the workshop to secure the participation of these senior people. Moreover the fact that the discussions were chaired by researchers who were experienced and knowledgeable in the field of health inequalities ensured that the discussion stayed focused and enabled issues to be debated in some depth.

It was notable that there was considerable agreement among participants, and this was unexpected as the sample had been purposively selected with the intention of obtaining a range of viewpoints. For example, there was a striking consensus among participants that much current research on health inequalities was of little use to policy making, and there was general consensus about the need for specific types of research evidence to bridge the gaps. We also saw no clear evidence of participants disagreeing strongly with each other, which may be unusual. However, we feel that this may simply reflect a wider movement toward consensus about the types of public health research we need, a consensus fostered by the common day to day experiences of these policy advisers when faced with the current national policy agenda on inequalities.

One other potential limitation of this study is that it reports policymakers’ perceptions, as compared with their actions. This raises the possibility that users of public health evidence may express a need for (say) evidence of outcomes, as in this study, but in reality they may not use this evidence. There therefore remains the real possibility that when more relevant evidence on health inequalities becomes available, it may still remain unused—perhaps for political reasons, as discussed above. Researchers involved in producing this evidence should probably prepare for this eventuality; the direct use of research evidence in policymaking still seems to be the exception, rather than the rule.¹³ For this reason, further work on the use of research in public health policy is valuable, as it can identify opportunities to improve knowledge transfer.²⁴ It has also been suggested that documentary analysis may be a particularly fruitful avenue for further research.^24–26

Despite the limitations outlined above, we believe that the findings from this workshop derive legitimacy from being rooted in the reality of a high level group of policymakers who are committed to evidence based public health policy and actively engaged in its development. Public health researchers’ perspectives, (which are often different and offer a detailed criticism of some of the policymakers’ views), were elicited at a subsequent workshop, and are discussed in an accompanying paper.⁸