Intended for healthcare professionals

Education And Debate

Prospective health impact assessment: pitfalls, problems, and possible ways forward

BMJ 2001; 323 doi: https://doi.org/10.1136/bmj.323.7322.1177 (Published 17 November 2001) Cite this as: BMJ 2001;323:1177
  1. Jayne Parry, director (J.M.Parry.1{at}bham.ac.uk)a,
  2. Andrew Stevens, professorb
  1. a Health Impact Assessment Research Unit, Department of Public Health and Epidemiology, University of Birmingham, Birmingham B15 2TT
  2. b Department of Public Health and Epidemiology, University of Birmingham
  1. Correspondence to: J M Parry
  • Accepted 25 June 2001

“The general objective of such assessments is to improve knowledge about the potential impact of a policy or programme, inform decision-makers and affected people, and facilitate adjustment of the proposed policy in order to mitigate the negative and maximize the positive impacts.”

European Centre for Health Policy1

The long tradition of never considering the impact on health of public investment has ended. The white paper Saving Lives: Our Healthier Nation exhorted local decision makers to “think about the effects which their policies have on health, and in particular, how they can reduce health inequality,”2 a recommendation that echoes statements made in the Acheson report, by the European Union, and by the World Health Organization.3-5

In the United Kingdom, government support for assessment of the health impact of policies has continued with the recent call for proposals for health impact assessment projects under the inequalities in health research programme, the establishment of a cross departmental health impact assessment group in central government, and the organisation and publication of methodological seminars and reports.6-8 The European Centre for Health Policy of the World Health Organization has produced a series of publications that includes the “Gothenberg paper,” regarded by many as the key document in stating the aims, objectives, and methods for health impact assessment.1

Increasing recognition of the effects of the socioeconomic and physical environment on health may, on the face of it, make it difficult to question the philosophy of health impact assessment—indeed the hype currently surrounding health impact assessment would imply that it is the indispensable condition of policy investment. But what is health impact assessment, and can it in its present format reliably inform better decision making?

Summary points

Proponents of health impact assessment claim that it can inform policy and decision making to maximise benefits and minimise negative impacts on health

Current health impact assessment is insufficiently rigorous to make robust assumptions on the magnitude or even the direction of the health impacts of policy interventions

Review of the literature and consultation with stakeholders are the key tools of health impact assessment, but both have associated problems

Validation of the predictions of health impact assessment raises issues such as the accurate measurement of health and the use of control populations

Local decision makers should adopt a process of mini health impact assessment, involving the use of available evidence, little quantification, and limited consultation

Full (maxi) health impact assessment should be undertaken only in a rigorous and effective way, involving robust methods and evaluation after implementation

What is health impact assessment?

Health impact assessment has been variously, albeit similarly, defined (see box). In short, it is about estimating the health consequences of new projects and policies, whether they be mobile telephone masts, the building of new motorways or sports stadiums, or tinkering with street lighting or parish halls.

Definitions of health impact assessment

“The estimation of the effects of a specified action on the health of a defined population.”(Scott-Samuel A. Health impact assessment: theory into practice. J Epidemiol Community Health 1998;52:704-5.)

“A combination of procedures or methods which enable a judgement to be made on the effect(s)—positive or negative—of policies, programmes or other developments on the health of a population or on parts of the population where inequalities in health are concerned.” (Kemm J. Developing health impact assessment in Wales. Cardiff: Health Promotion Division, National Assembly for Wales, 1999.)

“A combination of procedures, methods and tools by which a policy, program or project may be judged as to its potential effects on the health of a population, and the distribution of those effects within the population.” (European Centre for Health Policy, World Health Organization Regional Office for Europe. Health impact assessment: main concepts and suggested approach. Brussels: WHO, 1999.)

“A methodology which aims to identify, predict and evaluate the likely changes in health risk, both positive and negative (single or collective) of a policy, programme, plan or development action on a defined population.” (British Medical Association. Health and environmental impact assessment: an integrated approach. London: Earthscan Publications, 1999.)

The advocates of health impact assessment make it predominantly a predictive rather than an empirical research tool, and its claims are substantial—to be able to inform policy and decision making to maximise benefits and minimise negative impacts on health. The definitions accorded to health impact assessment and its proposed utility in terms of modifying policy imply an objective, sophisticated, and apolitical process. The perception is that the estimation of health impacts has been achieved through the application of robust methods and is of sufficient validity to enhance the decision making process. However, we would argue that at present health impact assessment is excessively subjective, subject to political drivers, and insufficiently rigorous to make any robust assumptions on the magnitude or even the direction of the health impacts of policy interventions.

The conduct of health impact assessments

Many models of health impact assessment exist, but they all encompass a series of similar steps—screening, scoping, impact assessment, policy modification, and, in some cases, evaluation. 1 9

Screening

The initial step—“screening”—determines whether or not a project should be subject to health impact assessment. Many organisations believe that a health impact assessment is needed for every proposed project, a belief substantiated in the United Kingdom by both the reading of Saving Lives and the suggestion that different levels of health impact assessment (rapid appraisal, comprehensive, and overview) exist to cover community interventions of all shapes and sizes.1-10 But policy interventions are innumerable. In the Netherlands, where a national health impact assessment pilot scans all proposals going before parliament, some 400 projects are being submitted for evaluation each day.11 Selectivity is needed.

The criteria for selection are unclear. Attention may be best directed at big and expensive projects that will have obvious and immediate effects on health—for example, a new road transport policy. But it may be more important to subject to health impact assessment those projects where the health impacts are less obvious or may not become apparent for several years—precisely because of the uncertainty surrounding their impact. The conundrum here is that the identification of projects where the impacts might be latent or less apparent needs a health impact assessment in itself, and ignoring smaller projects may risk missing important effects of many similar but individually small proposals. The need for “strategic screening” of single sector and cross sectoral activity (analogous to strategic environment assessment within the field of environmental impact assessment) must be given further consideration.7

Scoping and assessment

The potential health impacts of the proposed intervention are identified in the second stage of the health impact assessment (“scoping”), and their direction and magnitude are assessed in the third stage. Evidence is accrued by reviewing the literature and obtaining the views of key stakeholders—professional, political, or lay—who have expert knowledge of the project or who are involved in and potentially affected by its implementation. The results are then tabulated in a series of checklists or grids to indicate possible or probable health effects. Both sources for this part of the exercise are potentially inaccurate and biased, and robust methods are needed (see below) if valid predictions of health impacts are to be made.

Policy modification and evaluation

“The final but essential step is to act on the results of the health impact assessment.”1 That is, the results are then used to modify the proposed intervention to maximise positive and minimise negative health impacts. Clearly, the assiduousness with which this step should be undertaken will depend on what informs it. Action without good science will do little more than cover up the “control” (action-free) scenario.


Embedded Image

Health impact assessment for a new Wembley Stadium might involve consultation with key stakeholders such as the Metropolitan Police, England manager Sven Goran Eriksson, and minister for sport Richard Caborn

The need for a further stage, evaluation, is mentioned in some health impact assessment models; where it is mentioned, the focus is divided between the undertaking of evaluations of process and evaluations of outcome. One consistent observation, however, is the marked difference in the attention given to the prediction of health impacts (considerable) and that given to the appraisal of whether those predictions that resulted in modification of policy were actually accurate (fleeting).

Consultation and literature reviews: the key tools

Reviewing the evidence

Prediction of the health impacts of any intervention depends on a synthesis of all available evidence to produce an estimate of the likely effect and the application of this estimate to the affected population. McIntyre and Petticrew note that “the identification and incorporation of relevant evidence, its appraisal for methodological soundness and relevance, and its incorporation with qualitative evidence is likely to be difficult, but crucial to the validity of health impact assessments.”12 An explicit description of the search strategies and quality assessment criteria used to identify the evidence cited in published health impact assessments is often absent; this suggests that most have not been informed by systematic review of the evidence.w1-w7 Such flawed reviews are likely to result in biased and inaccurate effect estimates.13 w8-w11

The time and effort needed to undertake systematic reviews and synthesise evidence relevant to policy interventions are likely to be considerable. Within the domain of environmental impact assessment, some work has already been done to guide researchers on the optimal use of epidemiological evidence.14 However, much of the work that is relevant to health impact assessment has been done within disciplines not necessarily familiar to public health analysts (for example, criminology or education). Furthermore, many reports are published only in the “grey literature” and are thus difficult to access. Many evaluations of community based initiatives will not be in the form of randomised controlled trials but will include quasi-experimental designs, cohort studies, case series, and qualitative research. The methods for synthesising robust evidence from such sources are complex and still developing.13 w12 w13 However, the newly formed Campbell Collaboration and outputs from the Evidence Based Policy Centre funded by the Economic and Social Research Council will undoubtedly contribute substantially to both the evidence base to inform health impact assessment and the development of methods relevant to this area. 15 16

Consultation with stakeholders

The second source of evidence used to inform the estimation of potential health impacts is consultation with stakeholders. It is difficult to argue against the process of consultation with experts and the community as a mechanism to make public policy decision making more transparent. But if the consultation is to usefully and reliably inform the estimation of health impacts, the process of consultation itself needs to be rigorous and its impacts as an intervention in its own right need to be recognised.

Consultation with the community and other experts has four main problems. Firstly, there is the simple matter of conducting the consultation in a manner that is balanced and reliable. Social scientists have provided us with a wealth of experience in the techniques of sampling, interviewing (at both individual and group level), and analysing qualitative data.w14 w15 However, the methods used in such consultations are not clearly reported in the published case studies. How were “hard to reach” groups accessed? Was random, opportunistic, or purposive sampling used? What was the participation rate? Who did not participate? Why? Were the interviews or focus groups recorded and their findings transcribed? Which thematic analytical approach was adopted?

The second problem in consultation is the inherent danger of relying on the opinions of stakeholders to predict the effect of a policy. Often, intuition has not been supported by the findings of subsequent appropriately conducted studies.17 Some of these decisions—for example, putting babies to sleep on their front because it mimicked the resuscitation position—had notable adverse consequences.

Thirdly, when health impact assessment includes community involvement it becomes an intervention in its own right. The problems associated with perception of risk and the communication of magnitude of risk have been extensively discussed.18 The mere acknowledgment that a health impact assessment is needed may change the community's perception of the risk of the intervention in an unpredictable manner. Thus the very process of undertaking an health impact assessment may have an impact (positive or negative) on community health.

Finally, the involvement of the community could be seen as a way in which unpalatable political decisions are offloaded from the decision makers (the “we went out to consultation” argument). The ethical implications of involving the community in health impact assessment need further consideration.

Validating the predictions of a health impact assessment

Given the flaws in the process of health impact assessment outlined above, how confident can we be that the predictions of health impact used to change implementation of policy are accurate? We need some empirical tests of the predictive process. Indeed, there is a strong case for collecting more evidence from concurrent health impact assessments not used to change policy at the time. A number of prospective health impact assessment case studies in the United Kingdom have now been published, but none has reported on the follow up and assessment of the accuracy of their predicted health impacts. Recent guidelines on the conduct of health impact assessment produced by the English Department of Health state that “the monitoring of health trends and outcomes will probably be conducted by officers in an organisation or partnership and incorporated into existing systems for data collection and monitoring.”9 But is such an expectation realistic?


Embedded Image

The community may identify factors such as discarded syringes and graffiti as important determinants of health, but these are difficult to quantify

Assessment of the impact of a policy intervention on health needs substantive and reliable data based on a clear and consistent definition of health. Measures of mortality, morbidity, and use of health services, although routinely available, are insensitive to short term and small effects. An alternative approach is to focus on changes in a selection of determinants of health, although this has problems too. Many indicators are collected by local authorities, voluntary organisations, and health authorities, but the spatial unit may be too large (for example, at local authority level only) or the frequency of collection too infrequent (for example, census data every 10 years) to permit analysis of relatively small changes.19 Even data that are reliably collected and sensitive to change may have questionable relevance to health and thus not be a valid indicator. Unfortunately, factors that the community may identify as important determinants of health—such as dog faeces, discarded syringes, and graffiti—are difficult to quantify.

Even if we can identify relatively simple tools to measure health, other issues remain. The simple “before and after” approach to the measurement of outcome indicators is, in isolation, insufficient given the many and varied confounding factors. Consideration needs to be given to how the relation between cause and effect can be elucidated through the use of control populations constructed from different sources (see box).

Selection of control populations

The New Deal for Communities initiative is multimillion pound policy action plan designed to reduce social exclusion and inequalities. Comparators for evaluation include:

  • The “new deal” sites themselves—to consider how the introduction of essentially similar novel interventions affects different communities, so permitting an appraisal of how contextual differences have an impact on policy implementation

  • Other community populations within each local authority to act as “control” populations, with an assessment at baseline and during follow up of the outcome variables undertaken. These populations would be of two types:

    1. communities that applied for but were unsuccessful in attaining new deal status

    2. communities with similar health and sociodemographic profiles to new deal communities at baseline

  • Artefactual homogeneous populations constructed from the aggregation of enumeration districts with comparable geo-economic and socioeconomic characteristics to each of the new deal populations

Other problems also need to be tackled. For example, techniques that attempt to quantify health by examination of ecological measures or random sampling for surveys of quality of life or participation in focus groups assume a static population. However, migration rates vary substantially throughout the United Kingdom, with some communities experiencing turnovers of housing tenure in excess of 20-30% a year.20 Migration is non-random and is associated with deprivation.21 Is it therefore an outcome variable in itself or a confounding factor that needs to be explicitly acknowledged and adjusted for? Routinely collected (albeit imperfect) data sources (for example, electoral register, NHS central register, general practitioner lists, council tax databases) might throw light on rates of migration in small areas. 22 23 Proponents of health impact assessment should consider how the results of such work can be incorporated into the evaluation of policy interventions through, for example, sensitivity analyses and modelling of population based outcomes.

An additional uncertainty is the time profile of health impacts after a policy is implemented. If we are to validate predictions about the nature and magnitude of an intervention's impact on the health of a community, then at what point during follow up do we attempt to re-measure health? The different impacts associated with an intervention will vary with time and will be different for different communities and different subgroups of a community.

These issues, although not addressed in the existing literature on health impact assessment, are not new. There are few references to work in the United States, where for the past 50 years social scientists have attempted to evaluate many arenas of social policy.24w16 These researchers have already begun to wrestle with the problems of assessing the impact of “comprehensive community initiatives” and have developed rigorous frameworks for incorporating qualitative data gained from community consultation exercises into predictive and evaluative cause-effect models.25 w17 These techniques—and existing quantitative epidemiological techniques such as cluster randomisation, time series measurement, and bayesian and decision analysis modelling—need to be incorporated into health impact assessment.26

The way forward

Substantial concerns remain about the available methods for health impact assessment, and inadequate and inappropriate assessments may be produced in the desire to be seen to be doing health impact assessment. At best, this may merely waste time, effort, and money; at worst, it may result in delayed and flawed decision making and the adoption of policies, programmes, and projects that exacerbate health inequalities.

A two pronged attack

We believe that a two pronged attack is needed. Firstly, as the demand for health impact assessment remains and excessive expectations are part of the problem, proponents of health impact assessment should lower their expectations of the process. Given the inherent limitations of the methods available, health impact assessment is unlikely to offer a mechanism by which accurate predictions of the health impacts of proposed policy interventions might be made. Health impact assessments involving non-systematic reviews of the literature and suboptimal consultation exercises should be avoided, and a rapid “mini health impact assessment” process should be adopted. 1 9 10 A mini health impact assessment will use existing information and measure few variables, and experts will meet for a maximum of one day—all in all a much smaller exercise than the standard health impact assessment (see box).

Levels of health impact assessment

Mini health impact assessment (suggested default policy)

  • “Desk top” exercise

  • Reliant on information already available “off the shelf”

  • Minimum quantification of impacts

  • Limited consultation—single meeting with selected stakeholders

  • Standard health impact assessment (current usual practice)

  • Limited literature search, usually non-systematic

  • Mostly reliant on routine data

  • Quantification of impacts

  • Participation of stakeholders, but sampling methods not rigorous

  • Maxi health impact assessment (policy evaluation)

  • Extensive literature search

  • Secondary analysis of existing data

  • Collection of new data

  • Extensive quantification and sensitivity analysis

  • Full participation of stakeholders—validated qualitative techniques with participants identified through robust sampling methods

  • Evaluation of both the process of health impact assessment and the impacts of policy intervention, including use of control populations where possible

Although the empirical evidence to support such an assertion is lacking, we believe (both from our own experience and from the uncertainty surrounding the prospective assessment of policy interventions) that the relation between the number and nature of impacts identified and the time resource input into an health impact assessment is far from linear. Local organisations might achieve more “bangs for their buck” (and a more realistic expectation of those bangs) by adopting the mini health impact assessment approach (figure). Such bangs may not include a robust assessment of future health impacts but might embrace a mechanism to get agencies (other than health) involved in intersectoral collaboration, a tool to undertake consultation, and a process to develop a framework for evaluation.

Figure1

Possible relation between inputs and outputs in health impact assessment

Secondly, full (“maxi”) health impact assessment should be undertaken only in a rigorous and effective way and must be followed up after implementation of the project. If health impact assessment is to attain its objective and provide added value to the decision making process, a fundamental shift is needed from the current position to one where estimates of effect are identified through a systematic review of the evidence and their accuracy is validated through post-intervention evaluation. The standard health impact assessment should be abandoned. Lessons from classic epidemiology, the social sciences, and the evaluation of American urban, educational, and other social policy need to be taken on board, not only to provide information on the direction and magnitude of likely impacts but also to guide workers as to the methods to be adopted.

Maxi health impact assessments undoubtedly represent a substantial challenge both intellectually and in terms of the time and resources needed, but their strength lies in the validation of anticipated health impacts. Although the immediate practical benefit is in the modification of the project only after it has been completed, lessons can be learnt to inform future decision making. Such an approach is entirely concordant with the ongoing development of evidence based policy. 15 16

Conclusion

Health impact assessment is an intuitively appealing and simple concept. However, there is a gap between the objectives of health impact assessment (making predictions about future health impacts in order to change policy actions) and the methods currently adopted by practitioners.

Exponents of prospective health impact assessment as it currently stands should be explicit about the limited evidence and uncertainty with which they are working. We believe that such limitations compromise predictions of impact to such an extent that the standard model of health impact assessment should be abandoned. Instead, we recommend that local decision makers adopt a process of mini health impact assessment, whereby a reduction in the time and effort dedicated to individual interventions will result in little or no loss of information gained. A robust evaluation of the impacts of community based interventions will need substantial investment in maxi health impact assessment.

Footnotes

  • Competing interests None declared.

  • Embedded Image Additional references are on the BMJ's website

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
View Abstract