- Edoardo Masset, research fellow1,
- Lawrence Haddad, director1,
- Alexander Cornelius, research assistant2,
- Jairo Isaza-Castro, research officer3
- 1Institute of Development Studies, University of Sussex, Brighton BN1 9RE, UK
- 223 Bennett Drive, Hove, UK
- 358 Lewes Court, University of Sussex Park Village Road, Brighton BN1 9RU
- Correspondence to: E Masset
- Accepted 2 November 2011
Objective To assess the effectiveness of agricultural interventions in improving the nutritional status of children in developing countries.
Design Systematic review.
Data sources Published and unpublished reports (after 1990) in English identified by searching 10 databases (Agris, Econlit, Eldis, IBSS, IDEAS, IFPRI, Jolis, PubMed, Web of Science, and World Bank), websites, previous systematic reviews, and reference lists and by contacting experts.
Study selection Included studies assessed effects of agricultural interventions aiming at improving the nutritional status of children (bio-fortification, home gardens, small scale fisheries and aquaculture, dairy development, and animal husbandry and poultry development). Only studies that used a valid counterfactual analysis were included. Before/after studies and participants/non-participants comparisons affected by selection bias were excluded.
Data analysis Results were analysed for four intermediate outcomes (programme participation, income, dietary diversity, and micronutrient intake) and one final outcome (prevalence of under-nutrition). Analysis was by summary tables of mean effects and by meta-analysis (for vitamin A absorption).
Results The review included 23 studies, mostly evaluating home garden interventions. The studies reviewed did not report participation rates or the characteristics of participants in programmes. The interventions had a positive effect on the production of the agricultural goods promoted, but not on households’ total income. The interventions were successful in promoting the consumption of food rich in protein and micronutrients, but the effect on the overall diet of poor people remains unclear. No evidence was found of an effect on the absorption of iron, but some evidence exists of a positive effect on absorption of vitamin A. Very little evidence was found of a positive effect on the prevalence of stunting, wasting, and underweight among children aged under 5.
Conclusions The question posed by the review cannot be answered with any level of confidence. The data available show a poor effect of these interventions on nutritional status, but methodological weaknesses of the studies cast serious doubts on the validity of these results. More rigorous and better designed studies are needed, as well as the establishment of agreed quality standards to guide researchers in this important area.
Agricultural interventions have long been thought to have an effect on nutrition. In the past 10-15 years, people have accepted that for agricultural interventions to have a greater chance of affecting nutritional status, they must be implemented with that objective. Examples of interventions with the explicit goal of improving the nutritional status of children include home gardens and the production of bio-fortified crops. Our review assesses the evidence for the effectiveness of this type of intervention. It complements and expands previous reviews of links between agriculture and nutrition. Compared with previous reviews, ours is more systematic and more focused on interventions explicitly targeting nutrition, has stricter inclusion criteria, and covers a range of interventions launched in the past 10 years. The aim of the review is twofold: to inform policymakers on the effectiveness of these interventions and to suggest to researchers which particular designs, methods, and metrics should be adopted for better assessment of the effects.
We do not review all agricultural interventions here, but only those that have the explicit goal of improving the nutritional status of children. Interventions falling into this category include bio-fortification, home gardening, aquaculture, small scale fisheries, poultry development, animal husbandry, and dairy development. Other agricultural interventions, such as irrigation, watershed development, land reforms, agricultural extension, and food processing and storage, may have a significant effect on nutrition but are not included because a reduction in under-nutrition is not their main objective.
Figure 1⇓ outlines how these interventions are expected to affect children’s nutritional status. The first step is participation; not all targeted populations are able to join programmes, and the interventions can miss vulnerable groups. Participating households are invited to adopt new production technologies. This may consist of new production activities (such as livestock and fisheries) or new varieties of existing production activities (such as the production of fortified food). The adoption of a technology increases household income and changes the food basket consumed by households or the nutritional content of existing baskets. Both effects—higher incomes and better diets—lead to improvements in nutritional status. Masset et al contains a more detailed exposition of this causal chain.
Under-nutrion is a complex phenomenon determined by multiple causes. Several factors that are not included in fig 1⇑ can affect children’s nutritional status, such as parents’ caring practices and health status. For example, in one of the studies reviewed, an intervention successfully increased the consumption of food rich in vitamin A among children, but a parasitic infestation in the intervention areas reduced the observed values of serum retinol. This example shows that to assess these interventions properly, we should either take into account all potential confounders at each stage of the causal chain or assess the success of the interventions at each step of the causal chain separately. We decided to follow the second approach of documenting the effect of these interventions along their impact pathways. Hence, in addition to reviewing the effect on nutritional status, we reviewed the effect on the following intermediate outcomes: participation, household income, diversity of diet, and micronutrient intake.
We searched 10 databases covering nutrition, agriculture, rural development, and social sciences and including both published (Econlit, IBSS, PubMed, and Web of Science) and unpublished literature (Agris, Eldis, IDEAS, IFPRI, Jolis, and World Bank). We did searches over the period from July to September 2010. To minimise the risk of missing relevant papers, we searched the reference lists of key studies, applied the “forwards citation” utility in Google scholar to find papers that cited these studies, reviewed the full reference lists of the systematic reviews previously done on the same subject, and contacted several experts in the relevant field who provided further references.
We included a study only if it was produced after 1990, was written in English, was conducted in a middle or low income country as classified by the World Bank, and investigated effect on at least one of the outcome indicators identified by the review. Given the complexity of these interventions, randomised trials are rarely used because they are very expensive and difficult to conduct. We decided to exclude studies that did not use a control group and those based on before-after comparisons. We included cross sectional and longitudinal project-control comparisons and randomised field trials. We also included studies comparing participants and non-participants over a single cross section unless the risk of selection bias was not acknowledged and provided a minimum attempt was made to correct for this bias, such as matching project and control observations on pre-intervention characteristics or testing differences in characteristics between the project and control observations before the interventions.
Data extraction and validity assessment
We screened the studies in two stages. At the first stage, one investigator, by reading titles and abstracts, selected the studies that were written in English, were produced after 1990, and included the outcome indicators identified by the review. At the second stage, two investigators independently reviewed the methods used in the studies selected at the first stage and excluded those that did not meet the methodological standard set by the review. Disagreements between the investigators were resolved by discussion. Full texts of the selected studies were uploaded to the EPPI reviewer database, and outcome data were reported on a predefined Excel spreadsheet.
We evaluated the validity of the studies along two dimensions of internal and external validity. Given the limited number of trials, the impracticality of blinding, and the variety of statistical methods found, we decided not to rely on standard risk of bias forms designed for randomised trials in clinical settings. Instead, we scored studies as of low, medium, and high quality along four dimensions: statistical validity of counterfactual analysis, sample size and power calculations, assessment of intermediate outcomes, and assessment of effect on subgroups.
Methods for synthesis
The small number of included studies, the variety of outcome indicators, and the diversity of metrics used offered little scope for doing meta-analyses. Therefore, we reported results by using summary tables of effects. These summaries report the number of statistically significant effects against non-statistically significant ones, thus providing a very general indication of the direction of effect based on the available evidence. Whenever possible, we report effect sizes and P values, but these were often not available or not comparable across studies.
In the case of effect on serum retinol concentration, we were able to do a meta-analysis, albeit on a small number of studies. We used a fixed effect model, because when the number of observations is small the estimates of a random effect model tend to be imprecise. However, this approach provides a simple descriptive analysis of the studies reviewed, and the summary effect reported has little generalisability.
Search results and characteristics of studies
Electronic and hand searches returned 7239 studies. The first stage selection excluded 6932 studies simply on the basis of language, year of publication, and reporting of relevant outcomes. The second stage selection screened 307 studies on the basis of methodological characteristics and included 23 of them in the final review.9 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 The web appendix lists these studies with their main characteristics and outcomes.
Although the initial search returned an equal number of studies of the interventions considered, studies of home gardens (n=15) figure prominently among the 23 studies finally selected. We found a much smaller number of studies of bio-fortification interventions (2), small scale fisheries and aquaculture (3), dairy development (1), and animal husbandry and poultry development (1). Rigorous evaluations of the effect of dairy development, animal husbandry, and fisheries projects are extremely rare. In the case of bio-fortification programmes, the lack of evidence is largely due to the novelty of the interventions. Few bio-fortification programmes have been in operation for a sufficiently long time to be rigorously evaluated.
Overall, the methodological quality of these studies was not high by the standards set by the review. Nine, mostly unpublished, randomised trials were included in a meta-analysis of the impact of high protein maize. None of the primary studies reviewed was based on a randomised design. Studies were cross sectional or longitudinal project-control comparisons in which the controls, either households or villages, were selected on the basis of similar characteristics that were either very few or not made explicit. Power calculations were rarely done or presented, and samples were often small in terms of both individual participants and clusters. No study did a rigorous subgroup analysis of effect differentiating, for example, households of different wealth, sex of head of household, or location of residence.
Most studies described the population targeted by the interventions only in very general terms, such as poor geographical areas, women, poor households, or remote communities. No study reported participation rates or the programme’s ability to reach the targeted population. No study described the socioeconomic characteristics of participants or the determinants of participation. This is unfortunate, because we cannot tell how well targeted these programmes were, whether vulnerable groups were effectively reached, or what the effect was on specific subgroups.
Some studies reported a positive effect on incomes from a particular source, such as income from home gardens. This measure of effect is imprecise, because substitution effects in production are possible. As income from one source increases, income from another source may decrease if households shift, for example, labour supply from farming to home gardens. As a result, the overall effect on household income remains unclear. Five studies reported a large positive effect of the interventions on total household income. However, only in one case was the difference between project and control groups statistically tested.
Effects on the composition of the diet cannot be summarised across studies because they refer to different food items and are measured in different ways. Of the 23 studies selected, 19 reported a positive effect on the composition of the diet. With very few exceptions, home garden programmes increased the consumption of fruit and vegetables, aquaculture and small fisheries interventions increased the consumption of fish, and dairy development projects increased the consumption of milk. One difficulty in interpreting these results is that an increase in the consumption of the food item targeted by the intervention does not imply an improvement in the overall diet, because substitution effects in consumption occur. For example, Bushamuka et al found that although the consumption of vegetables, rice, and fish increased after the intervention, the consumption of pulses decreased. This suggests that indicators of the diversity of the diet or analysis of the full consumption basket are better indicators than is consumption of the specific food promoted by an intervention.
The included studies investigated the effect of interventions on two micronutrients: iron and vitamin A. Two studies assessed the effect on iron intake of children and found no statistically significant differences in the average haemoglobin concentrations between children in the project and control groups.13 19 Another study assessed the effect of fish consumption on iron intake at the household level and found a modest effect by using food to micronutrients conversion tables. The observed effect would have been even smaller after consideration of the actual body absorption of the iron ingested.
Nine studies reported effects on concentration of serum retinol from blood samples. However, only four of these studies reported means and standard deviations of observations on children in project and control areas. The forest plot in figure 2⇓ summarises the results of these four studies. The difference between the mean serum retinol concentration in the project and control group for each study is reported with a 95% confidence interval. The size of the squares represents the weight of each study in the calculation of the summary effects. Overall, the effect of the interventions is a difference of 2.4 µg/dL in serum retinol between project and control areas (z test of significance 6.35; P<0.001). This summary effect is the weighted mean of the effects found by the individual studies and is represented by the diamond in figure 2⇓. The width of the diamond is the confidence interval of the summary effect and represents its level of precision. This meta-analysis provides some support to the hypothesis that agricultural home gardens interventions improve vitamin A intake among children under the age of 5.
Children’s nutritional status
Anthropometric data were collected by 13 of the 23 studies included in the review, but only eight studies used these data to calculate prevalence of stunting, underweight, and wasting. The only exception was the study by Gunaratna et al, which used rates of growth in height and weight, rather than prevalence, and found a positive and statistically significant effect on nutrition. Only one of the studies reported the results in terms of z scores. Given the small number of studies reviewed, we could not disaggregate the analysis of effect by type of study or geographical area, for example.
Table 1⇓ shows the results of these studies. Only one study found a statistically significant effect on prevalence of stunting, whereas three studies found a positive effect on prevalence of underweight and two found a positive effect on wasting. The relatively greater success of agricultural interventions in reducing the prevalence of underweight and wasting compared with stunting can be explained at least in two ways. The interventions considered may be better suited to reducing short term under-nutrition rather than chronic under-nutrition. An alternative explanation is that the studies assessed the effect shortly after the interventions had taken place and therefore could not capture long term effects.
Overall, these results provide little support for the hypothesis that agricultural interventions help to reduce under-nutrition. However, they should not be interpreted as evidence of the absence of an effect. Lack of significance can be the result of absence of effect or of absence of statistical power, and many of the studies reviewed included small samples of children.
To explore this further, we did post hoc power calculations for these eight studies and assessed the probability of finding a “small” (2%), “medium” (10%), and “large” (30%) difference in prevalence of under-nutrition. We averaged post hoc power calculations of the studies for the three hypothetical effects. On average, the probability of detecting a small effect for these studies was less than 5%. Even a “large” effect (that is, a reduction in malnutrition by 30%) would be detected in only 50% of cases. A more realistic medium effect of 10% would be detected with a probability of only 15%. On the basis of this analysis, the absence of any reported statistically significant effect of agricultural interventions on nutritional status found by this review, as well as by other reviews that preceded this one, cannot be attributed with certainty to lack of efficacy. Rather, the lack of power of the studies reviewed could have prevented the identification of any effect.
This review assessed the effectiveness of agricultural interventions in improving the nutritional status of children with five outcome indicators: programme participation, income, composition of diet, intake of micronutrients, and children’s nutritional status. We found little information on participation rates and characteristics of programme participants. The interventions reviewed had a positive effect on the production of agricultural goods promoted, but we found no evidence of an effect on households’ total income. The interventions were successful in promoting the consumption of specific foods, but little evidence is available on changes in the diet. We found no evidence of an effect on the absorption of iron and some evidence of an effect on absorption of vitamin A. We found very little evidence of an effect on the prevalence of stunting, wasting, and underweight among children under 5 years of age.
Comparison with earlier reviews and limitations
Our review largely confirms the findings of previous reviews that little evidence exists of an effect of agricultural interventions on the nutritional status of children.2 4 5 Unlike previous reviews, however, we attribute this result to the lack of statistical power of the studies reviewed rather than to the lack of effectiveness of the interventions. Lack of statistical power is one of several methodological weaknesses that had already been identified by previous reviews. In part, this is a consequence of the complexity of the environment rather than of the skills or rigour of researchers. Randomised trials of complex agricultural interventions tend to be expensive and difficult to implement. This may imply that researchers should seek evidence of efficacy rather than effectiveness. By disentangling the complex chain of factors that lead from implementation of a programme to nutritional outcomes, we have shown how poor our understanding is of the circumstances under which people participate in these interventions and what intermediate effect they have on incomes and composition of the diet. In addition, confounding factors, such as the health environment and cultural practices, can interfere with implementation of programmes at each stage. Perhaps this calls for rigorous evaluations of effect on intermediate outcomes rather than large scale effectiveness studies.
Implications for policy and for future research
Our review does not provide clear indications to policy makers, beyond a reasonable degree of caution regarding their expectations of the nutritional effect of agricultural interventions. Nor are we able, on the basis of the existing evidence, to suggest how to prioritise among competing agricultural interventions. This would require an analysis of the cost effectiveness of the interventions, which is another largely neglected aspect of the literature considered. We do, however, have several recommendations for the research community.
Firstly, although randomised trials are rarely feasible for assessing complex agricultural interventions, studies should rely on credible counterfactual methods and avoid the use of before/after and participants/non-participants comparisons. Longitudinal and cross sectional studies of matched project and control sites should be used when randomisation is not feasible. Comparisons of participant and non-participant households should be made only when controlling for selection bias is possible. Secondly, studies should investigate the reasons why poor people choose to participate or not to participate in agricultural programmes, as this provides lessons on the quality of targeting of vulnerable groups and tools for tackling selection bias. Thirdly, researchers should use proper metrics, or design new ones, for the assessment of these interventions. In particular, household income and consumption and indices of diversity of the diet should be used to assess the effect on production and dietary composition. Fourthly, anthropometric measurements should be taken whenever possible, but sample sizes should be sufficiently large to detect an effect if present, and power calculations should always be done. More suggestions on how to do this type of study can be found in an expanded version of this review. We hope that our work will contribute to the definition of a set of agreed standards and guidelines for the evaluation of agricultural interventions.
The question this systematic review set out to answer was “how effective are the agricultural interventions that aim to improve the nutritional status of children?” We have concluded that we cannot answer this question with any confidence, which is unfortunate. Agricultural interventions command large resources. If only some of them could be made to reduce under-nutrition, we would accelerate progress towards millennium development goal number 1.
What is already known on this topic
Little is known about the effectiveness of food based agricultural interventions in reducing under-nutrition among children in developing countries
What this study adds
Food based agricultural interventions effectively increase the production and consumption of the food promoted, and some evidence suggests that this leads to higher vitamin A intake
The available evidence shows no effect of these interventions on nutritional status of children, but methodological weaknesses of the studies reviewed cast serious doubts on the validity of these results
Agreed standards and guidelines for rigorous evaluation of the effect of agricultural interventions are needed
Cite this as: BMJ 2012;344:d8222
This review was financially supported by the Department for International Development (DFID). We thank Imran Choudury (DFID), Max Gasteen (DFID), and Todd Benson of the International Food Policy Research Institute (IFPRI) for their guidance in the process of designing the protocol. We thank Jeff Brunton of the Institute of Education (IOE), University of London, for providing and supporting the use of the EPPI-Reviewer software. We thank Jeremy Lind and Christophe Béné of the Institute of Development Studies (IDS) for their help in refining the search of the pastoralist and fisheries literatures. Finally, we thank Anna Taylor (DFID), Tim Wheeler (DFID), Hugh Waddington of the International Initiative for Impact Evaluation (3ie), and particularly Todd Benson (IFPRI) for their comments on an earlier draft of the review. The views and the conclusions expressed in this review are those of the authors and do not necessarily reflect views of DFID or of the aforementioned people.
Contributors: AC searched the databases and did the first stage selection of the documents. J-IC and EM did the second stage selection process of the documents. EM designed the protocol, analysed the data, and wrote the review. LH contributed to the definition of the goals of the study, the design of the protocol, and the formulation of conclusions and policy recommendations. EM is the guarantor.
Sponsors: DFID provided the initial study question to be answered by the systematic review in the following way: “What is the impact of interventions to increase agricultural production on children’s nutritional status?” DFID, through a “policy advisor” and an “internal reviewer,” provided comments on the protocol of the review to align the goals of the study with the policy interests of the institution. Several DFID reviewers provided comments on the first draft of the review.
Funding: The study was fully funded by a grant from DFID. The grant was awarded in May 2010 after a competitive bidding process within a DFID/3ie programme to promote systematic reviews on international development. The authors are completely independent of the funders. The funders contributed to the definitions of the goals of the review but had no role in the research or in deriving the conclusions and policy implications.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: all authors had financial support from the UK Department for International Development (DFID) for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not needed.
Data sharing: The data used in the meta-analysis, the statistical codes for the ex-post power analysis, and all other data used in the review including technical appendices are available from the corresponding author at.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.