Realist review to understand the efficacy of school feeding programmesBMJ 2007; 335 doi: http://dx.doi.org/10.1136/bmj.39359.525174.AD (Published 25 October 2007) Cite this as: BMJ 2007;335:858
- Trisha Greenhalgh, professor of primary health care1,
- Elizabeth Kristjansson, associate professor2,
- Vivian Robinson, doctoral candidate3
- 1Department of Primary Care and Population Sciences, University College London, London N19 5LW
- 2School of Psychology and Institute of Population Health, University of Ottawa, Canada K1N 6N5
- 3Institute of Population Health, University of Ottawa
- Correspondence to: T Greenhalgh
- Accepted 19 September 2007
Our Cochrane review of school feeding programmes in disadvantaged children included trials from five continents and spanned eight decades.1 Although we found that the programmes have significant positive effects on growth and cognitive performance, the trials had many different designs and were implemented in varying social contexts and educational systems; by staff with different backgrounds, skills, and cultural beliefs; and with huge variation in the prevailing social, economic, and political context. Simply knowing that feeding programmes work is not enough for policymakers to decide on the type of intervention that should be implemented. We therefore looked at the trials more closely to determine the aspects that determine success and failure in various situations.
We analysed the 18 studies (reported in 29 articles) included in our Cochrane review2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 using the methods of a realist review. Realist review exposes and articulates the mechanisms by which the primary studies assumed the interventions to work (either explicitly or implicitly); gathers evidence from primary sources about the process of implementing the intervention; and evaluates that evidence so as to judge the integrity with which each theory was actually tested and (where relevant) adjudicate between different theories.31 32
We read, re-read, and discussed the papers and constructed a matrix on an Excel spreadsheet to collate information for each trial on:
Study design, sample size, and outcome data
Nature of the experimental and (where present) control interventions, including intensity and timing
Process detail, especially comments on the fidelity of the intervention, changes made by staff on the ground, and reasons for those changes
Aspects of the study's history and context, especially those highlighted as important by the study's authors
Any theories or mechanisms postulated (or assumed) by the study's authors to explain the success or failure of the programme.
We considered relevant data first on a trial by trial basis in terms of the interaction between context, mechanism, and outcome, and then across the different trials to detect patterns and idiosyncrasies. We discussed preliminary conclusions and synthesised key findings using a narrative and interpretive approach.33 We identified four broad areas relevant to this analysis: the historical context of school feeding programmes (see bmj.com), theories to explain the success of particular programmes (box 1), theories to explain their failure or qualify a partial success (box 2); and measurement issues (see bmj.com).
Box 1 Process factors that seem to enhance efficacy of school feeding programmes
Strong process evidence across many trials
Target group has clear nutritional deficiency (usually, inadequate energy intake) and trial is oriented to correcting this rather than to short term hunger relief
Well organised schools that form part of an efficient distribution chain for the supplement
Intervention developed with local teams rather than designed by distant experts
Supplement is piloted to exclude intolerance and confirm palatability and acceptability
Measures are in place to ensure that the food supplement is consumed (eg close supervision of eating)
In disaffected young people, attention is paid to social aspects of the meal
Limited process evidence from one or few trials
Use of local ingredients and cooking methods
In extreme poverty, intervention is designed so that attending school is more economically viable than keeping children at home
Intervention seeks to induce a change in home diet by educating or inspiring children
Possible factors that might be tested in future studies
Better nutrition and health literacy in this generation reduces intergenerational cycle of poverty
Box 2 Process factors that seem to reduce the efficacy of school feeding programmes
Participants not aware of, signed up to, or trained to take account of the research dimension of the trial
Study design involves role conflict or ethical difficulties for staff (eg requirement to serve nutritious meal to some but not all undernourished children)
Insufficient measures in place to reduce confounding (eg controlling for benevolent attention)
Adverse prevailing policy climate (eg policy conflicts with trial protocol or prompts rebranding of mainstream activity to gain research funding)
Measurement issues (see bmj.com)
Theories of why school feeding programmes work
Long term correction of nutritional deficiencies
Nine trials in our sample were based on a theory that school feeding corrects overt nutritional deficiencies, which in turn improves brain growth and performance.5 6 7 8 11 16 17 24 25 Such trials assumed that food supplements should be rich in energy, protein, and vitamins and continued for a substantial period before their effect can be shown.
Most trials in low and middle income countries that set out to correct nutritional deficiency had positive results, although in two measurement of weight was distorted by oedema associated with kwashiorkor.13 20 Trials in high income countries had mixed results. Long term nutritional supplementation generally affected growth (and sometimes performance) when the children were genuinely undernourished, but not when they were not. For example, two trials of school milk supplements in Britain in the 1920s (a time of economic recession, high unemployment, and food shortage) showed a significant positive effect on children's growth5 7; but a trial in the 1970s showed no significant benefit, according to our statistical analysis, with the same supplement.1 2
Short term hunger relief
Two trials in low and middle income15 20 and three in high income3 4 10 countries were built around the theory that school feeding leads to short term rises in blood (and hence brain) glucose levels, which counteracts the negative effect of hunger on concentration, memory, motivation, and other psychological prerequisites for learning.34 Overall, the effect of interventions built on a hunger relief theory was not constant across different areas of performance (verbal, non-verbal, mathematical) or across studies.
Children feel valued and looked after
Powell criticised studies that failed to control for the effect of benevolent attention and recommended that, at the very least, the control group should receive a low energy drink or piece of fruit along with teacher or researcher attention.23
Bro and colleagues did two studies of “at risk” teenagers (school drop-outs, drug users, teenage parents, or from families with other social problems) who, though not malnourished, rarely ate before school. They showed that a generous breakfast cooked in a practical class before the lesson began improved attention to set tasks.3 4 Qualitative process data suggested that a meal at school can be a social event that engages, motivates, and stimulates the students.
Of the studies that measured attendance objectively, most of those in low and middle income countries showed significantly higher attendance levels in supplemented groups,12 20 24 whereas studies in high income countries had non-significant effects on attendance.6 8 11
Improved school diet inspires improved home diet
In one study, when children were given breakfast at school, their families subsequently bought more milk, meat, fish, and high vitamin C foods, whereas the families of a control group did not change their buying habits.11 This study was done in Canada at a time of rapid social change and rising affluence; two other studies in low and middle income countries (where parents presumably had less choice in what they bought) found no changes in home eating patterns.16 24
Improved literacy reduces intergenerational cycle of poverty
Several authors speculated about a longer term impact of school feeding—namely, that it would lead to higher literacy rates, which would offer the chance to break the cycle of poverty, giving the next generation of children better opportunities for good nutrition and health.14 16 Such an effect is difficult to measure because of the long time frame involved, but it should be borne in mind in future research.
Theories why school feeding programmes do not work
The commonest reason for failure was that the programme was built around a misguided theory (such as correcting a nutritional deficiency that did not exist) but other reasons may also apply.
Food offered is not consumed, or provides too little of the missing nutrient
Studies that piloted different supplements until they identified one that was readily consumed or that let children choose from a menu were, in general, more likely to improve growth.8 20 Very poor children rarely rejected food in any form, and in these studies the supplement generally had a significant effect.8 16 22 30 In contrast, those trials with adequate nutrients but less impact on growth generally documented incomplete consumption, sometimes because the children did not attend the meal.6 11
In one pilot study, 25% of children rejected a cows' milk supplement even when it was chocolate flavoured, strongly suggesting lactose intolerance.8 Use of a specially formulated low lactose milk supplement refined in response to the children's feedback on its palatability had a significant effect on growth.
Most trials in this review provided at least 15% of the recommended daily allowance of energy to the intervention group. Two studies that provided considerably less than 15% of the recommended daily allowance had no significant effect on weight.13 25 However, a study targeting calcium deficiency in teenage girls, which provided less than 15% of the recommended energy levels, did show a positive effect on the primary end point of height gain,17 suggesting that targeted correction of micronutrient deficiency may be effective.
Low bioavailability was occasionally invoked as an explanation for lower than expected effect of a feeding programme. Grillenberger and colleagues, for example, proposed that the milk supplement might have decreased the absorption of iron and zinc.19
Compensatory reduction in food intake at other times
A few studies documented a compensatory adjustment in appetite (supplemented children ate less at the next meal).8 24 Four studies in which the benefit of supplements was less than expected were done in very poor areas in Peru,20 Jamaica,23 Kenya,22 and India.12 The authors of two of these studies concluded that children who had been given a substantial supplement at school were provided with less food at home (substitution).12 20 The authors of the Jamaican study looked for substitution at home in a second study 15 years later24 and found no evidence that it was still occurring.
Supplementation occurs too late
Authors of a study published in 1962 speculated that the failure of their feeding programme in children aged 7-12 was because the children were “too far along the track of malnutrition” and recommended that subsequent studies should target younger children.13 Our statistical analysis, which showed significantly greater gains in weight in younger children,1 supports the notion that the earlier feeding supplementation occurs, the better it is for growth, although this was not the case for cognitive outcomes.
Programme is not implemented as planned
Deviation from the study protocol can be an important problem. One study reported, for example, that: “It was originally intended that Group 1 would be a control group. However it was impossible to obtain cooperation without distributing some supplements to all the boys.”13 Asking school staff to withhold food from hungry children when others are getting fed, or trying to stop children sharing food with their friends, was poor study design as well as ethically questionable. Consultation with the target population at design stage seems to help prevent such problems by producing an intervention that engages staff and incorporates their practical wisdom of what is workable. Lieberman and colleagues, for example, originally intended to randomise their participants by pupil but a steering group drawn from the local community rejected this and they eventually allocated by school, with one experimental and one control school.6
Some studies that did not apply selection criteria to schools, and where there was weak or absent sign-up to the research dimension, reported a high rate of sample attrition or non-fidelity of the intervention.9 10
Non-fidelity of the intervention was also explained by staff and senior management viewing the research as a way of gaining funding for a cash strapped organisation or community, and having little or no interest in the scientific elements. Shemilt et al, for example, evaluated a randomised controlled trial of school breakfast clubs in socioeconomically deprived parts of England in the early 2000s. At the time, there was a strong push for all schools in deprived areas to provide breakfast clubs. This led to the absurdity that “at second follow-up 72.2% of pupils in the intervention arm and 77.0% of pupils in the control arm had a breakfast club operating at their school.”10
Confounding or external variables
Several authors commented on factors—both internal (such as the wide variation in timing of puberty growth spurt2 17) and external (infections and infestations,26 seasonal variation,11 etc) that would have reduced the measured impact of the programme, making a (potentially) real difference non-significant.
A complex, community based intervention inevitably operates at multiple levels, and controlled trials of such interventions must be interpreted in their appropriate historical and policy context. Although the factors listed in boxes 1 and 2 should be seen as preliminary and non-exhaustive, we believe they will be useful to policymakers who need to know not merely whether school feeding programmes work but what sort of programme (if any) to put resources into. Our analysis supports concentrating school feeding on pupils with documented nutritional deficiencies, and for a development phase (working in partnership with the local community to optimise and pilot an intervention) before the programme is tested in an experimental trial—a finding that fits with the UK Medical Research Council's recommendations on the design of complex interventions.35 Consultation with practitioners and the local community may result in important changes in study design, which in turn may require a change in sample size, so should not be seen as mere formality.
Process data from some trials suggest that in situations of absolute poverty even severely malnourished children may not benefit from school feeding programmes because of substitution at home. In these very specific (and increasingly rare) circumstances, further research should take account of this theory. For example, the feeding protocol might be designed to provide a higher energy meal (to compensate for the food that will be withheld at home), or give food as a mid-morning snack (perhaps less likely to be substituted than a meal) or a different intervention might be used (such as rations to take home or income supplementation).
We focused only on the 18 trials included in the Cochrane review, but the analysis would undoubtedly be enriched by inclusion of descriptive studies, theoretical papers, and grey literature. Moreover, we were unable to distinguish between something that was not done and something that was done but not reported because of the stringent word count constraints of medical journals.
Is it time to shift the balance in what we define as quality from an exclusive focus on empirical method (the extent to which authors have adhered to the accepted rules of controlled trials) to one that embraces theory (the extent to which a theoretical mechanism was explicitly defined and tested)? If authors of trials of complex interventions were required to meet minimum quality standards for theory as well as method, far fewer systematic reviews might conclude that “more primary research is needed.”
Systematic reviews rarely give detailed information on the context, mechanisms, and outcomes of interventions and the theories that underpin them
This realist review describes the theory and processes in 18 trials of school feeding programmes
Programmes should be aimed at children with documented nutritional deficiencies
Programmes are more likely to be effective when designed in partnership with the local community and interventions are piloted
In situations of absolute poverty even severely malnourished children may not benefit from school feeding programmes because they may receive less food at home
We thank the authors of the original Cochrane systematic review. We also thank Ray Pawson and Penny Hawe for helpful comments on an earlier draft.
Contributors and sources: TG conceptualised the study, adapted the method from Pawson's original, drew up the original spreadsheet, conducted the initial realist analysis, and drafted the paper. EK and VR independently read all the primary empirical studies, contributed to a substantial revision of the realist analysis, and helped revise the paper. TG is the guarantor.
Competing interests: EK was given travel funds to present findings from this review at an experts seminar on school feeding in Rome and to attend an expert panel on school meals in Santiago.