Effects of the Heartbeat Wales programme over five years on behavioural risks for cardiovascular disease: quasi-experimental comparison of results from Wales and a matched reference areaBMJ 1998; 316 doi: https://doi.org/10.1136/bmj.316.7134.818 (Published 14 March 1998) Cite this as: BMJ 1998;316:818
- Chris Tudor-Smith, director of research and developmenta (, )
- Don Nutbeam, professor of public healthb,
- Laurence Moore, lecturer in health services researchc,
- John Catford, professor of health strategiesd
- a Health Promotion Wales, Ffynnon-las, Llanishen, Cardiff CF4 5DZ
- b Department of Public Health and Community Medicine, University of Sydney, NSW 2006, Australia
- c University of Bristol, Department of Social Medicine, Canynge Hall, Bristol BS8 2PR,
- d Deakin University, Burwood, Victoria 3125, Australia
- Correspondence to: Mr Tudor-Smith
- Accepted 18 November 1997
Objective: To assess the net 5 year effects of intervention of a community based demonstration project, the Heartbeat Wales programme, on modifiable behavioural risks for prevention of cardiovascular disease.
Design and setting: Quasi-experimental design comparing results from two independent cross sectional population surveys conducted in 1985 and 1990 in Wales and a matched reference area in north east England.
Subjects: Random, stratified samples of people aged 18-64 years (18 538 in 1985 and 13 045 in 1990) in Wales and in north east England (1483 and 4534, respectively).
Intervention: A coordinated range of activities for heart health promotion in Wales entailing public education campaigns along with supportive policy and infrastructure change. In the reference area no additional community heart health promotion was planned, though considerable activity did take place, “contaminating” the reference area.
Main outcome measures: Fifteen self reported behavioural indicators relating to dietary choice, smoking, frequency of exercise, and weight.
Results: Positive changes (for health) in behavioural outcomes were observed among the population in Wales, including a reduction in reported smoking prevalence and improvements in dietary choice. There was no net intervention effect for the programme over and above observed change in the reference area.
Conclusions: No definite conclusions can be drawn concerning the efficacy of the programme in terms of behavioural outcomes. With hindsight, the difficulties of evaluating such a complex multifaceted intervention were underestimated. Further debate on the most appropriate methods for assessing the effectiveness of community based health promotion programmes is called for.
Heartbeat Wales was set up in 1985 as a community based programme to demonstrate risk reduction for cardiovascular disease
Important changes were observed in modifiable risks for cardiovascular disease in Wales between 1985 and 1990
There was an unexpectedly rapid uptake of heart health promotion activity in the reference area
No net intervention effects were found for the programme over and above changes in the reference area
Improvements in methods of evaluation for community based health promotion programmes are required
Cardiovascular disease remains one of the major causes of morbidity and premature mortality in the United Kingdom.1 During the 1980s a consensus evolved on the need to reduce this toll of ill health and death through population-wide preventive measures (see, for example, papers by the World Health Organisation2 and Rose et al3). The Welsh Office and the existing national agency for health education, the Health Education Council, agreed to establish a community based demonstration programme in Wales directed towards reducing modifiable behavioural risks for cardiovascular disease.
The programme was publicly launched in 1985 as Heartbeat Wales with three strategic aims: leadership—to coordinate, support, initiate, and monitor action at local and regional levels which would encourage improvements in modifiable behavioural risks for prevention of cardiovascular disease; demonstration—to stimulate, disseminate, and assist the development of strategies and programmes to promote health and prevent cardiovascular disease throughout the United Kingdom; and experimentation—to research, develop, and evaluate a range of new projects and initiatives for heart health promotion and provide feedback on their feasibility and impact.4
Heartbeat Wales drew on the experiences of other community based risk reduction programmes for cardiovascular disease, particularly those in Finland and the United States.5-8 The programme used a range of established health promotion methods directed towards both changing health behaviours in individuals and achieving environmental, organisational, and policy changes that support healthy choices.9 Among the resources developed and interventions undertaken by Heartbeat Wales were television series with BBC Wales and HTV such as Don't Break your Heart, Fit for Life, and the BBC Diet Programme; “Quit and Win,” a smoking cessation project; food labelling and nutrition education with a major grocery retailer; “Heartbeat Awards,” a restaurant and canteen scheme to increase the availability of healthy food choices and smoke free areas; and Make Health Your Business, a worksite health promotion programme with CBI Wales.
Further details of the Heartbeat Wales intervention have been published elsewhere.9-13 To assess behavioural outcomes of the intervention a quasi-experimental evaluation design was adopted on the basis of comparison of change in modifiable behavioural risks for cardiovascular disease in Wales with that in a reference area in the United Kingdom closest in sociodemographic and health profile to Wales at the 1981 census. The reference area selected was north east England (Tyne and Wear, Cleveland, Durham, and North Yorkshire). The Health Education Council indicated that there would be no major additional resources in that area for heart health promotion between 1985 and 1990.
Two population surveys were conducted in 1985 and 1990 in Wales and the reference area. To assist with the interpretation of the findings from these surveys, a range of other studies described elsewhere14 was also planned for Wales but not the reference area. These studies have suggested that Heartbeat Wales achieved its basic aim of establishing a region-wide approach to the prevention of cardiovascular disease and that many of the key elements of the programme have been taken up and used elsewhere both in the United Kingdom and overseas. 4 10-13 15-18 It has also been shown that there were significant reductions in prevalence of smoking and improvements in food choices between 1985 and 1990 in Wales.19 This current paper compares these and other changes in modifiable behavioural risks for cardiovascular disease in Wales with those that took place in the reference area over the same time to assess net intervention effects of the Heartbeat Wales programme.
Subjects and methods
Data were collected in random sample, cross sectional surveys during the summer and autumn of 1985 and 1990. In each survey households were selected with a multistage cluster sampling design, within 10 strata defined by the nine Welsh district health authorities and the reference area. Sample size in the 1985 survey was determined (α=0.05, β=0.2) to detect a 5% change in prevalence of smoking within each strata by using a two tailed significance test. In 1990 sample size in the reference area was increased to improve the power of analyses that compared Wales with the reference area. Brief interviews were undertaken at each household, and one self completion questionnaire was then left for each resident aged 18-64. Respondents to all three surveys were asked a set of identical questions covering key health related behaviours such as smoking, diet, and physical activity as well as health knowledge and beliefs. In Wales the response rate for the household interview was 88% in 1985 and 79% in 1990 and the self completion response was 67% and 61%, respectively. In the reference area the respective figures were 84% and 77% for the household interview and 64% and 61% for the self completion questionnaire. Altogether, 31 583 questionnaires (18 538 in 1985 and 13 045 in 1990) were returned over the two surveys in Wales, with 6017 (1483 and 4534, respectively) returned in the reference area. Data were weighted before analysis by sex, age group, social class, and population distribution within each strata to minimise bias due to differential response rates between groups. Further details of survey methodology and weighting are available elsewhere.19
Fifteen indicators were selected as key outcomes for analysis. They represented those health related behaviours that were most consistently targeted during the intervention period and for which measurements were available. These indicators are listed and defined in table 1. Of the 15 indicators, eight represent dietary choices, five are concerned with smoking, and one each with participation in regular exercise and being overweight. Two sets of analyses were undertaken: firstly, at the level of the individual respondent; secondly, at the community level.
Individual level analysis
Standard errors of survey estimates and 95% confidence intervals were estimated on weighted data from the 37 600 completed questionnaires returned in the two surveys by using the SUDAAN (survey data analysis for multistage sample designs) statistical software package.20 This package uses the Taylor series linearisation method to compute appropriate standard errors for estimates obtained from complex survey designs and takes account of the effects of stratification and clustering on the precision of survey estimates. For each indicator, percentage point changes between 1985 and 1990 in Wales and the reference area (with 95% confidence intervals) were calculated.
Community level analysis
Analysis to compare change in Wales with change in the reference area was undertaken at community level by using the nine district health authorities in Wales and the four counties in the reference area as the units for analysis.
The intervention effect to be estimated was defined as the ratio of percentage change between Wales and the reference area (PC90W/PC85W)/(PC90R/PC85R), where PCY= proportion reporting characteristic C in year (Y=1985 or 1990) in Wales (W) or the reference area (R).
For each of the 13 analysis units (nine district health authorities in Wales and four counties in the reference area) the prevalence of each of the 14 binary categorical variables was calculated for the baseline and follow up (1990) surveys. Similarly, mean number of cigarettes per day for daily smokers was calculated for each unit in the two surveys. The logarithm of the intervention effect ratio was then estimated by fitting the following model:
where PCY =proportion reporting characteristic C in year (Y=1985 or 1990) in each unit (or mean number of cigarettes per day for daily smokers); Int=exposure to intervention: 1 in Wales, 0 in the reference area; and Z=ln(RC90)−ln(R85), where R is the proportion of respondents in each unit living in a household where the head of household is in a non-manual occupation. The variable Z was included as a covariate to adjust the analysis for variations between the surveys in the composition of social group of the samples within each unit. The parameter d was included to control for the possibility that the degree of change may be dependent on the baseline value.
Weighted least squares linear regression models were fitted for each of the 15 variables weighted by the mean sample size in each unit over the two surveys. Analyses were undertaken for all respondents, and additional analyses were also run for seven subgroups: men, women, three age groups (18-34, 35-49, 50-64 years), and people living in households where the head of household was in a manual or non-manual occupation. Two tailed t tests of the null hypothesis that the parameter b (the logarithm of the intervention effect) was equal to 0 were undertaken and the adjusted intervention effect ratio (exp(b)) and its 95% confidence interval were calculated.
Table 2 shows the prevalence of the 15 key outcome indicators in both Wales and the reference area in 1985 and 1990. These data indicate that in Wales there were positive (for health) changes in all 15 indicators except the proportion overweight, with all changes being significant except for the proportion of smokers who had tried to stop and mean daily cigarette consumption. Similarly, in the reference area there were positive changes in 13 of the 15 indicators, although only nine of these were significant.
Further analysis showed that the baseline prevalence of two of the indicators was significantly lower in Wales than in the reference area—namely, the preference for low fat milk and the consumption of fried food. By 1990 four indicators were significantly lower in Wales than in the reference area; these were preference for low fat milk, consumption of fried food, daily smoking, and participation in exercise. The preference for butter was significantly higher in Wales than in the reference area in both 1985 and 1990, while the proportion overweight was significantly higher in Wales in 1990 only.
Table 3 presents the findings from the community level analyses and indicates that there were two outcome indicators (consumption of fried food and daily smoking) for which there was a consistent intervention effect across all seven subgroups in favour of Wales. There was also one consistent difference (consumption of green vegetables) in favour of the reference area. These effects were each significant in no more than one subgroup, however, and when the community level analysis was undertaken with all the respondents no significant differences were found.
The results indicate important changes in modifiable risks to health among the population in Wales and in the reference area in the north east of England for the period 1985-90. These changes should lead to subsequent reductions in premature death from cardiovascular disease in Wales and the north east of England over the coming decade. As welcome as improvements in smoking levels, dietary habits, and exercise patterns may be, the results do not show clear and consistent net intervention effects of the Heartbeat Wales programme after 5 years in comparison with activities in the reference area.
Sample size and contamination
Interpretation of the results reported here, however, requires a clear understanding of the strengths and weaknesses of the study design and of the context of the intervention. The critical assumption made in the study design was that the contrast between the intervention in Wales and existing activity in the reference area would be large enough and sustained over a 5 year period to show a clear net intervention effect. This was not the case for two reasons. Firstly, the sample size at the baseline measurement in the reference area was too small to give sufficient statistical power for the detection of a likely net intervention effect. Secondly, a previous paper by the authors has documented the diffusion of Heartbeat Wales projects and programmes to the reference area far faster and to a far greater extent than had initially been expected, along with the introduction of additional resources for heart health promotion through the development of the Look After Your Heart project, which was launched across the whole of England in 1987, and the Heartbeat Yorkshire programme, which was conducted in the reference area from 1988.4 This paper clearly shows the “contamination” of the reference area with initiatives that can be traced back to Heartbeat Wales in whole or part, including the uptake of policy changes in the health services, adoption of several unique projects, and receipt of a number of mass media interventions, especially those developed with BBC Wales. In addition, the paper provides evidence of increases in funding for heart health promotion in the reference area commensurate with increases in Wales during the Heartbeat Wales intervention period.
In retrospect, it was naive to believe that a high profile programme such as Heartbeat Wales could remain in quarantine for such a long period. A direct result of its success as a national demonstration programme was the attenuation of differential exposure to heart health promotion between Wales and the reference area and thus a dilution of any measurable intervention effect. In addition to these identifiable confounding factors are the favourable secular trends in smoking and dietary choices in the United Kingdom as a whole, which further confuse the interpretation of results. 21 22
Because of these design problems, no definite conclusions can be drawn concerning the efficacy of the Heartbeat Wales programme in terms of behavioural outcomes. Indeed, two directly conflicting conclusions could be drawn, both of which would be compatible with but not proved by the results presented here: on the one hand, the improvements in risk behaviours for cardiovascular disease in Wales suggest that the Heartbeat Wales programme has been effective, with positive changes in the reference area also associated with increased community heart health promotion; on the other hand, the lack of any net intervention effect compared with the reference area suggests that the Heartbeat Wales programme has had no impact additional to secular trends.
These results from the United Kingdom can be set alongside results from other comparable programmes in the United States operating during the same period. Although encouraging results have been reported from the Stanford five city project, these were obtained from a cohort of subjects in the intervention communities. As in Wales, no differences were found in the comparison between independent samples in the intervention population and the reference communities.6 Similarly, analysis of results from independent samples in the Minnesota and Pawtucket heart health programmes showed that intervention effects were “modest in size and duration and generally within chance levels”7 and “very limited.”8 The strengths of positive secular trends relating to behavioural risks for cardiovascular disease were cited as a reason why net effects were difficult to identify despite positive results in the intervention communities.23 Unlike Heartbeat Wales, there is no indication that these studies included active monitoring of the diffusion of the programme in the reference area(s) or the extent of contamination through other unpredicted events.
The major conclusion to be drawn from this study is that the basic quasi-experimental design was inappropriate and insufficiently sensitive to answer the complex research question being asked. By their very nature, successful long term community based programmes can result in complex and wide ranging effects, many of which may be unexpected and not confined to any one predetermined intervention community, making the measurement of any impact and attribution of causality highly problematic.23-28 Solving these problems will remain a continuing dilemma for advocates of prevention and should be a cause for reflection among academics and researchers concerning appropriate methods for assessing the results from such programmes. New evaluation techniques need to be developed that combine the strengths of quantitative and qualitative research methods and make better use of more proximal outcomes.29
We thank Geoffrey Berry, Jim Dwyer, Les Irwig, Russell Luepker, Lucie Richard, Judy Simpson, Chris Roberts, and colleagues at Health Promotion Wales for their help and constructive comments.
CT-S managed the 1990 data collection and coordinated the writing of the paper. DN had overall responsibility for the implementation of the research strategy and contributed to the intervention design and implementation. LM was responsible for and undertook the statistical analyses. JC led the development and implementation of the programme. All authors contributed to the writing of the paper. CT-S and DN are the guarantors for the study.
Funding: Welsh Office.
Conflict of interest: None.