Effects of Sure Start local programmes on children and families: early findings from a quasi-experimental, cross sectional studyBMJ 2006; 332 doi: https://doi.org/10.1136/bmj.38853.451748.2F (Published 22 June 2006) Cite this as: BMJ 2006;332:1476
All rapid responses
Belsky and colleagues state that children from relatively more
socially deprived families were adversely affected by living in areas
where there were Sure Start Local Programs. As the authors acknowledge the
cross sectional design and lack of randomisation mean that these
conclusions need to be treated with caution. It is clear from the data
that there are significant differences between Sure Start and comparison
populations, most notably there are marked differences in the level of
income and ethnicity of the populations. Although the authors claim to
adjust for area characteristics it is not clear which of these are used in
the analysis. It seems that it is possible that the differing outcomes
experienced by the most deprived families are due to other differences
between the areas not related to the provision of Sure Start.
One such difference that does not seem to have been taken into
account by the authors is that there is greater income inequality in the
Sure Start group than in the comparison group. Several authors most
notably, Wilkinson(1) have pointed out that income inequality, has an
impact on social capital and health outcomes, independent of mean income.
Much of this work is based on larger geographical areas mainly in the
United States although one ecological study found an association between
mortality and income inequality in English local authority areas (2). One
explanation therefore for these results could be that the children of the
most deprived families fair better in areas were there is more equality in
income. The potential for confounding in this study could have been
minimised if intervention and comparison areas had been matched for some
of these characteristics.
Specialist Trainee in Public Health,
Central Lancashire PCT,
1. Wilkinson, R. The impact of Inequality. How to Make sick
societies healthier. London: Routledge; 2005
2. Stanistreet, D., Scott-Samuel, A., Bellis, M. A. Income in
inequality and mortality in England. Journal of Public Health. 1999;21:205
Competing interests: No competing interests
Belsky and colleagues reported that Sure Start programmes only had a
limited effect on children and their families1. It is a pity that the
Government refused to consider using a randomised trial to evaluate this
important initiative. Initial non-randomised evaluations of the ‘Head
Start’ programme in the USA found little or no benefit of that
initiative2. Because of this lack of effectiveness evidence senior US
policy makers, quite rightly, called for it to be stopped. However,
evaluations using randomised controlled trials were commissioned instead
and these found that Head Start did have a significant beneficial impact
on outcomes. In the acknowledgments section of the Sure Start evaluation
the authors claim that the Department for Education and Skills (DfES), did
not consider a randomised design because the programme was delivered to
entire communities. There is quite a bit of evidence from Table 3 that
the two groups were not similar at baseline and the lack of a major impact
may, as in the early Head Start evaluations, be due to confounding.
Although an individually randomised trial may have been inappropriate, a
cluster design could have been used. Had the DfES had the foresight to
commission a RCT of this initiative then we would have been clearer as to
whether it worked or not. Would the Government allow major pharmaceutical
companies to market new products on non-randomised data? If not, why
should they ‘market’ their own products on the community without using
1 Belsky J, Melhuish E, Barnes J, Leyland AH, Romaniuk H. Effects of
sure start local programmes on children and families: early findings from
a quasi-experimental, cross sectional study. BMJ 2006.
2 Wortman PM. An exemplary evaluation of a program the worked: The
High/Scope Perry Preschool Project. Evaluation Practice 1995;16:257-65.
Dr Carole Torgerson
Senior Research Fellow
Dept of Educational Studies,
University of York
Competing interests: No competing interests
Dear Ms Godlee
As a Member of Parliament whose constituency contains significant
pockets of social deprivation (like many seaside and coastal towns) and
which now has three Sure Start projects (two of them estate-based)
operating as part of the Government’s strategy to combat this, I was
naturally concerned to see media reports that the report you published in
your June 16 issue of the British Medical Journal by Jay Belsky and his
colleagues was suggesting that Sure Start projects were making things
worse, not better, for some of those who used them.
I have therefore now read carefully the piece that you carried, which
seems to be very far from substantiating the way in which it has been
treated in the non-specialist media. Unfortunately it also seems to me
that the conclusions Jay Belsky and his colleagues draw from what they
themselves describe as ‘early findings from a quasi-experimental cross
sectional study’ have also been too sweeping and have contributed, however
unwittingly, to the distorted headlines and intros that appear in, for
example, the Daily Mail of June 16: ‘Flagship project to help deprived
children and families is making crime and truancy worse, not better, an
alarming investigation revealed.’
I accept as a layman that others may be better placed than I to
critique some of the details and protocols that this study used. But
apart from my constituency interest – which has given me close contact
with the Sure Start centres and meetings with staff, support workers and
parents using them – I have followed these issues closely as a member of
the Commons Education Select Committee and national vice-president of
Early Education. I also – in a previous existence – spent nearly five
years working as a public relations adviser on medical and related
subjects, followed by twelve years as a magazine editor, which gave me a
crash-course on the complexities and sensitivities of interpreting
research presentations to a non-specialist audience and the pitfalls that
lie in wait for the unwary.
In the case of the assertions in the Belsky report that ‘children
from relatively more socially deprived families…were adversely affected by
living in SSLP’ – the assertion leapt on by the Daily Mail and others –
there appears to be little substantial material elsewhere in the report to
back it up. We are told that children of teenage mothers, like those who
lived in workless or lone parent households, ‘scored lower on tested
verbal ability’ – hardly surprising given that such mothers often have
themselves had schooling and family difficulties. But the crucial
question – how does this compare with their skills before they entered the
Sure Start programme – does not appear to have been asked. Sure Start
programmes have come in successive waves – some of those interviewed may
have been in them for two to three years, some for much less. Again, no
allowance appears to have been made for this in the evidence presented.
It is surely a fundamental principle that in order to make valid or
worthwhile judgements about how well the Sure Start programmes have
performed – either overall or for particular groups – you have to look at
the participants before and after entering the programmes to measure the
progress – or non-progress – between A and B. The Belsky article does not
appear to do this. We are told that ‘mothers of children aged 36 months
(but not 9 months) living in SSLP areas rated their communities a little
less favourably than mothers in comparison areas.’ Again, this is hardly
surprising, given that pre- and continuing publicity for such Sure Start
projects – as here in Blackpool – would rightly have highlighted they were
being set up to help families in areas with elements of deprivation and
social exclusion. It is not an evidential basis on which to judge their
I would add that the ability of the non-specialist – and possibly
many of the specialist – readers to penetrate and assess the Belsky paper
is not helped by linguistic monstrosities such as on page 2: ‘the findings
presented are based on multiple imputed data sets in which missing values
of all independent and dependent variables were estimated based on
standard multiple imputation procedures.’ I would hope that contributors
to the BMJ, as a long-established journal of record with an impressive
pedigree in communicating to a broader public as well as to the
specialist, could do better than that!
Jay Belsky and his colleagues have reported positively elsewhere in
their article on other aspects of the Sure Start programme and, to be
fair, issue their own health warning about the paper: ‘because this
evaluation was quasi-experimental, cross sectional, and evaluated the
impact of a programme that had been in place for only a few years, the
detected effects of SSLPs and the conclusions much be treated with
caution.’ Wise words, but the presentation of the paper’s conclusions did
not follow them – and the broadsheet coverage which exploited them
certainly did not.
This is not just a specialist issue about academics’ findings which
can cheerfully be discussed and argued over at learned symposia and
conferences. The impact on the morale and self-esteem of the people using
and involved with Sure Starts countrywide – including my constituents –
will be affected by the coverage of papers such as this. I do believe
that academics and researchers have a public responsibility to consider
how they present such material – and to be acutely aware of how it may be
misinterpreted or abused. It is hardly as if in taking on the Sure Start
project the authors of your paper would have been unaware of the high-
profile and public sensitivities of what they wrote. I hope they will
reflect on how their study has been presented and make some effort to
address the distortions and the issues I have raised here.
Competing interests: No competing interests
Both Sure Start and this evaluation are disappointing on this showing. The evaluation concludes with the assertion that the study indicates that
improving parenting is one of the mechanisms by which SSLPs promote child wellbeing.
I accept that this sentiment is part of the theoretical basis for Sure Start but I'm not at all sure that this evaluation has demonstrated that it is happening. Except, of course, in those "relatively less socially deprived" groups who may be depriving others of "access to services" that they may experience as "stressful and instrusive". I would like to have seen more discussion about whether it is the 'stress and intrusion' that is responsible for the 'harm' or the putative "less access to services": there is a case for saying that this paper has presented these two explanations without an acknowledgement that they may be mutually exclusive.
A long-term follow-up study of the children of depressed parents reports that:
The risks for anxiety disorders, major depression, and substance dependence were approximately three times as high in the offspring of depressed parents as in the offspring of nondepressed parents. Social impairment was also greater.
There is a growing body of evidence to suggest that environmental exposure to hostility or aggression (whether physical or verbal) is linked to poorer lung function and an increased number of asthma exacerbations in children: and the environment may be the home, school or neighbourhood. These findings may be relevant to this programme and the interpretation of its results.
It would have been useful if this evaluation had attempted to cost the advantages of supporting "relatively less socially deprived families" who may be benefiting by increasing their resilience, with all the improved physical, cognitive, social and emotional benefits that that confers for the family and the reduced costs for society. Interviewed for the New York Times, leading psychiatrist and resilience expert Sir Michael Rutter said:
We now have well-replicated findings showing that genes play a major role in influencing people's responses to adverse environments. But the genes don't do anything much on their own.
If Sure Start has to continue, why not sort out the civil liberties issues and do something exciting and useful like collect genetic material and test for the so-called resilience gene? Collecting data on this scale would be a valuable contribution to the work of researchers looking at the tandem effect of genetics and environment.
In 2003 Avshalom Caspi and his colleagues published a fascinating paper in Science that discussed the relationship between the gene, 5-HTT, and childhood maltreatment in causing depression. Current theories say that 5-HTT is crucial for the regulation of serotonin to the brain; and that the proper regulation of serotonin protects us against depression in response to trauma or stress.
In humans, each 5-HTT gene has two alleles, and each allele occurs in either a short or a long version. Scientists are still figuring out how the short allele affects serotonin delivery, but it seems that people with at least one short 5-HTT allele are more prone to depression. And since depression is associated with unemployment, struggling relationships, poor health and substance abuse, the short allele could contribute to a life going awry. A Question of ResilienceNew York Times
If we insist on experimenting with social engineering, let's gather more data and look at the tandem effects of genetics, environment and life experiences.
Competing interests: No competing interests