Do health improvement programmes fit with MRC guidance on evaluating complex interventions?BMJ 2010; 340 doi: https://doi.org/10.1136/bmj.c185 (Published 01 February 2010) Cite this as: BMJ 2010;340:c185
All rapid responses
ON COMPLEXITY, HETEROGENEITY AND EVALUATION CAPACITY BUILDING
Sanjeev Sridharan1, Mhairi Mackenzie2, Catherine O’Donnell3, Emma
Halliday4, April Sridharan5, Stephen Platt6
We thank Prof. Bond et al. for their comments (1). In our original
paper we raised conceptual, methodological and translational challenges in
implementing and evaluating complex policies. As is sometimes the case in
such dialogues, methodological aspects are privileged while conceptual and
translational issues are largely ignored. We are concerned, however, not
only about the conduct of experiments but also about learning from
evaluations of policies that are heterogeneous in respect of their
implementation and the understandings of key stakeholders.
Differences between broader learning goals and the narrower goal of
demonstrating impact using experimental trials have been extensively
discussed in the evaluation literature (2-11). One clear message is that
learning should not be reduced to that which can be demonstrated by
implementing controlled trials. On this point, we are encouraged by the
authors’ claim that the revised Medical Research Council guidance is “not
intend[ed] … to be prescriptive but to help researchers, funders, and
other decision makers to make appropriate methodological and practical
choices” (Craig et al., 2008, p. 982). However, we continue to believe
that much more needs to be done before the guidance can serve the broader
goal of learning from evaluation.
Our purpose in this note is threefold: to describe the challenge of
translating complex policies; to respond briefly to Bond et al.; and to
raise questions that need to be addressed in order to move forward the
field of the evaluating complex interventions.
TRANSLATING COMPLEX POLICIES: THE PROBLEMS OF HETEROGENEITY
In our original paper we stressed the importance of paying attention
to the heterogeneities of translating a complex policy into practice, and
how these heterogeneities need to be considered at every stage of the
evaluation process. The Keep Well example presented in the original paper
provides one example of heterogeneities in policymakers’ understanding of
the theoretical model of policy implementation and of approaches to ‘scale
up’ the original Tudor-Hart model of anticipatory care (12, 13).
Heterogeneity also existed in practitioners’ understanding of the policy:
different general practices had different implicit theories of inequities;
definitions of ‘hard-to-reach’ methods of reach differed across general
practices. Given such heterogeneities, there was a shift and perhaps loss
in translation between the original policy intent and implementation of
In view of the conceptual and organisational challenges of responding
to such heterogeneities, we remain unconvinced that solutions can be found
purely in the methodological realm. Likewise, treating such
heterogeneities as ‘noise’ that can be ignored is also a mistake. On the
contrary, we need to pay heed to what can be learned from such
heterogeneities. As example, in a recent JAMA article titled,
“Heterogeneity Is Not Always Noise: Lessons From Improvement”, Davidoff
(11) persuasively argues: “As physicians, we are ambivalent about
heterogeneity in medicine. We actively suppress it— ignore it, tune it out
— because doing so is crucial for establishing the efficacy of tests,
drugs, and procedures. But heterogeneity is ubiquitous in complex systems,
including all of biology and human society.... we recognize that
suppressing it exacts a heavy price and struggle to take it into account”
(p. 2580). While Davidoff focuses on the substantive implications of
heterogeneities in a clinical setting, the parallels to our paper are
striking, including his discussion of the implications of contexts. As
he argues: “...context cannot be experimentally ‘controlled out’ of social
programs because it is a major determinant of any given program’s
effectiveness” (p. 2581).
Through a focus on the complexities of translating policy intentions
we aim to stimulate the evaluation and knowledge translation communities
to consider their implications, including the multiple roles that
evaluators might play in addressing heterogeneities. In our view, ignoring
such heterogeneities is a sure recipe for undermining the ability of key
stakeholders to use the evaluation for accountability, learning and
decision making purposes. Ignoring such heterogeneities can also result in
bad science (11).
TECHNICAL STRENGTHS AND IMPLEMENTABILITY
Much of the Bond et al. response is an argument for the technical
strengths of RCTs and their appropriateness in health improvement
settings. As there have been many useful and extended discussions on the
technical strengths and weaknesses of RCTs (3-11) we do not repeat the
debates here. However, we point the interested reader to Ravaillon (11) as
an excellent, brief and balanced statement for the need for humility in
treating such methodological debates as a shut case, and perhaps also to
Rodrik (7) and Deaton (5) on the need to be cautious about contributing to
the polarisation of this discussion.
Our arguments have not been about whether or not experimental trials
can be implemented in health improvement settings. Instead, we sought to
raise important questions about the role of experimental trials, given the
various types of heterogeneities that may arise in the implementation of
policies such as Keep Well. It is important not to gloss over such
complexities or to treat such heterogeneities as analytical problems that
can be addressed through interesting evaluation methods. The key issue is
not whether the randomised design will provide an ‘unbiased’ estimate of
programme impacts. Rather, given the variations inherent in both policy
understanding and implementation, as well as changes in policy
implementation over time, we ask: What does it mean to have an ‘unbiased’
estimate under these conditions? Unlike Bond et al., we remain
unconvinced that this is purely a design or methodological question.
THE NEED FOR GREATER EXPLICATION OF A GUIDANCE DOCUMENT
While the revised Medical Research Council guidance document does
represent a step forward, we continue to call for much greater conceptual
and methodological explication. Specifically, greater clarity is needed
on: a definition of complexity; what is a ‘good enough’ theory of
implementation; programme context; and approaches to integrating
evaluation methods and issues of learning.
• Defining complex intervention and the sources of complexity: While we
agree with Craig et al. (2) that a sharp differentiation between simple
and complex intervention might be difficult, we are struck with the
continuing lack of clarity in the literature of what makes an intervention
complex and how the evaluation of a complex intervention differs from that
of a simple intervention. We believe that a guidance document on complex
interventions needs to go further on understanding the sources of
complexity and the implications for evaluation design. In our experience,
evaluations of complex interventions need to recognise at least three
different sources of complexity. First there is complexity due to the
multiple, potentially interacting components of complex interventions. The
evaluation challenge is to understand and assess their impacts. A second
source of complexity is the dynamic nature of programmes. In common with
other complex interventions, public health interventions often change over
time in response to a number of factors, and these changes have
implications for both programme theory and evaluation design. A third
source of complexity can also be due to the contextualisation of the
intervention. The act of translating an initiative requires adaptation to
local settings in which the programmes are located. Each of these sources
of complexity – multiple interacting components, dynamic complexity and
contextualisation complexity – has implications for both programme theory
and design. Two larger questions for the field emerge from this
discussion: Can the complexity of an intervention be operationalised? And,
if so, What are the implications for the design of evaluations?
• Clarity on ‘good-enough’ theory: We welcome the focus of the MRC
guidance on intervention theory. Yet there continues to be a lack of
clarity both in the document and in the broader literature on what
constitutes a ‘good-enough’ theory that can assist with policy
implementation. What are characteristics of a useful implementable
theory? How does the theory of a complex intervention differ from a theory
of a simple intervention? In our experience with public health
interventions, theory is often missing or is grossly coarse at the
planning stages of an intervention. Who should be responsible for
developing theoretical clarity for both the implementation and evaluation
of complex interventions? The same rigour and focus that the field has
expended on defining good-enough methods and gold standards has not been
spent in defining ‘good-enough’ theory that can help with implementation.
• Lack of conceptual and operational clarity on contexts: Part of the
arguments for a different approach to evaluations of complex interventions
is that different programme mechanisms might operate in different
contexts. Given the centrality of the recognition of context in
evaluating complex interventions, it is surprising to see the very limited
discussions in the evaluation literature on conceptual and operational
definitions of contexts. A focus on context and programme mechanism raises
a number of important questions (14-17): Given the need to contextualize
interventions for specific populations and for a range of contexts, in the
absence of well specified theories, where does knowledge of heterogeneous
mechanisms come from? What should be the evaluator’s role in explicating
such theory? Given the absence of a clear understanding of contextual
conditions, how can programs be adapted to fit into local conditions but
still stay true to the original aspirations of the intervention? Given the
difficulties in defining the active ingredients of complex interventions,
what does fidelity in a complex intervention mean? What are some measures
of adaptation and fidelity in respect of a complex intervention?
• Towards methodological pluralism: A problem that also comes through
loud and clear in the recent literature and debates is the recognition
that ‘learning is not the monopoly of a single method’ (11). In fields
such as developmental economics, there is ever growing attention to
learning as a paradigm rather than a narrow focus on experimental trials.
The lesson increasingly is on the types of evidence that are useful given
the heterogeneous nature of complex systems. For example Rodrik (2008)
makes this important point: “The ‘hard evidence’ from the randomized
evaluation has to be supplemented with lots of soft evidence before it
becomes usable.” A productive way forward is to think of ways in which
different methods/evaluation designs can be integrated to generate usable
information. For example, how can theory-driven approaches be combined
with more traditional experimental and quasi-experimental approaches (18)?
• A focus on learning and spread: One of the more surprising things
about a number of evaluations is the lack of clarity in what type of
learning is being “spread” (19) during or at the end of an evaluation.
Key issues related to learning that need to be confronted by the future
guidance document include generalisability, replication, scalability and
utilisation in the design of the evaluation. For example, is an
intervention that is taken to scale qualitatively different from an
intervention evaluated in a pilot/demonstration project? This is an
especially important question because issues of fidelity of complex
interventions have received very limited attention in the evaluation
literature. There is, however, a related growing literature in the field
of knowledge translation that makes persuasive arguments to incorporate
ideas of “knowledge use” more directly into the knowledge development
process ( 20-22). As example, consider Kitson (2007, pp. 125-126): “....
we have accepted a number of assumptions about the intrinsic nature of
guidelines that reflect more how the knowledge/evidence was generated
rather than a consideration of the social context into which the knowledge
has to be implemented or the local needs of the practitioner”.
MOVING THE FIELD OF EVALUATION FORWARD
Much of the narrow and often heated debates on evaluation methods presume
that improving evaluation practice might simply be a matter of “new and
improved” methodological guidelines. However, improving evaluation
practice might take more than another round of methodological debate. For
example, recent evaluation literature has focussed on the importance of
evaluation capacity building (23-27) as a means of addressing the
evaluation gap (24) (the gap is the difference between the evaluation
skills that are needed and what exists in practice). We see very
encouraging signs in the development of evaluation as a field that moves
well beyond debates about methods. As example, in countries as different
as China, India and Sri Lanka, there is growing awareness of developing
evaluation as a profession. Interesting evaluation capacity building
experiments in these countries have begun to emerge. The push for
evidence-based decision-making is an important driver behind some of these
efforts. Evaluation approaches will necessarily become much broader as
methods are adapted which respect the various cultural and organisational
contexts and capabilities in which decisions will be taken (25-27).
1Keenan Research Centre, St. Michael's Hospital, 30 Bond Street,
University of Toronto, Toronto, Ontario, M5B
2Department of Urban Studies, 25 Bute Gardens, University of Glasgow, G12
3 General Practice & Primary Care, Division of Community-based
Sciences, 1 Horslethill Rd, University of Glasgow
4School of Health and Medicine, Division of Health Research, Lancaster
University, Lancaster, LA1 4YT
5Independent Consultant, Address: 21 Cheritan Ave. Toronto, M4R1S3
6 Public Health Sciences, Division of Community Health Sciences, The
University of Edinburgh
Medical School, Teviot Place, Edinburgh, EH8 9AG
1. Bond L, Craig P, Egan M, Skivington K, Thomson H. MRC guidelines
and the evaluation of health improvement programmes: are health
improvement programmes really too complex to assess their effectiveness?
(Rapid response). BMJ.com 2010; 340.
2. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M.
Developing and evaluating complex interventions: the new Medical Research
Council guidance. BMJ 2008;337:a1655.
3. Cartwright N. Are RCTs the gold standard? Biosocieties 2007;2:11–20.
4. Cartwright N. Hunting causes and using them: approaches in
philosophy and economics. Cambridge: Cambridge University Press, 2007.
5. Deaton A. Instruments of development: Randomization in the
tropics, and the search for the elusive keys to economic development.
Proceedings of the British Academy, 2008 Lectures. Oxford: Oxford
University Press, 2009;162:123-160.
6. Abbott A. The Causal Devolution. Sociological Methods &
7. Rodrik D. The New Development Economics: We Shall Experiment, But
How Shall We Learn? In J Cohen, W Easterly (Eds.), What Works in
Development? Thinking Big and Thinking Small. Washington, DC: Brookings
Institution Press, 2009.
8. Heckman JJ, Urzua S. Comparing IV With Structural Models: What
Simple IV Can and Cannot Identify. NBER Working Paper No. 14706, 2009 Feb.
9. Banerjee AV, Duflo E. The Experimental Approach to Development
Economics. Annual Review of Economics 2009;1:151-178/
10. Ravallion M. Should the randomistas rule? Economists Voice 2009
11. Davidoff F. Heterogeneity is not always noise: lessons from
improvement. JAMA 2009; 302(23):2580-6.
12. Hart JT, Thomas C, Gibbons B, Edwards C, Hart M, Jones J, et al.
Twenty five years of audited screening in a socially deprived community.
British Medical Journal 1991;302:1509-13.
13. Hart JT. Clinical and economic consequences of patients as
producers. Journal of Public Health Medicine 1995;17:383-6.
14. Pawson R, Tilley N. An introduction to scientific realist
evaluations. In E Chelimsky, WR Shadish (Eds.) Evaluation for the 21st
Century: A handbook (pp. 405-418). Thousand Oaks, CA: Sage, 1997.
15. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new
method of systematic review designed for complex policy interventions. J
Health Serv Res Policy 2005;10:21-34.
16. Pawson R, Sridharan S. Theory-driven evaluation of public health
programmes. A Killoran, M Kelly (Eds.) Evidence-based public health:
Effectiveness and efficiency (pp. 43-61). Oxford: Oxford University Press,
17. Greenhalgh T, Kristjansson E, Robinson V. Realist review to
understand the efficacy of school feeding programmes. BMJ 2007;335:858–61.
18. Cook, TD. The false choice between theory-based evaluation and
experimentation. New Directions in Evaluation 2000;87:27-34.
19. Massoud, MR, Nielsen, GA, Nolan, K, Schall MW Sevin, C. A
Framework for Spread: From Local Improvements to System-Wide Change. IHI
Innovation Series white paper. Cambridge, MA: Institute for Healthcare
20. Kitson A. Knowledge translation and guidelines: a transfer,
translation or transformation process? Int J Evid Based Healthcare
21. Nutley SM, Walter I, Davies HTO. Using Evidence. How research can
inform public services. Bristol: The Policy Press, 2007.
22. Carlile P. Transferring translating and transforming: an
integrative framework for managing knowledge across boundaries.
Organizational Science 2004;15:555–68.
23. Carden F. Knowledge to Policy: Making the Most of Development
Research. New Delhi: Sage; Ottawa: IDRC, 2009.
24. Carden F. The real evaluation gap. Alliance: For Philanthropy and
Social Investment Worldwide 2007;12(4):53-4.
25. Carden F. Introduction to the Forum on Evaluation Field Building
in South Asia. American Journal of Evaluation 2010;31:219-221.
26. Hay K. Evaluation Field Building in South Asia: Reflections,
Anecdotes, and Questions. American Journal of Evaluation 2010;31:222-231.
27. Sridharan S, De Silva S. On Ambition, Learning, and Co-Evolution:
Building Evaluation as a Field. American Journal of Evaluation 2010;31:246
Competing interests: No competing interests
Peter West highlights that there is an inherent assumption in many
health improvement initiatives that the intervention works. I would
suggest there is another prevalent assumption: that any intervention will
be better than “doing nothing” or putting the money into front line
individual care. In a negative trial of a policy to provide breastfeeding
groups across Scotland (1), “too many initiatives going on” was perceived
by health service staff as detrimental to the quality of service that they
could provide and was one factor influencing trial outcomes (2). Unlike
drugs, complex health improvement interventions consume considerable
amounts of health service staff time which is precious. Without the focus
and rigour of RCT designs, combined with thorough process evaluation,
staff resources are likely to be used ineffectively.
1. Hoddinott P, Britten J, Prescott G, Tappin D, Ludbrook A, Godden
D. Effectiveness of a policy to provide breastfeeding groups (BIG) for
pregnant and breastfeeding mothers in primary care. BMJ 2009;338:a3026.
2. Hoddinott P, Britten J, Pill R. Why do interventions work in some
places and not others: A breastfeeding support group trial. Social Science
and Medicine. 2010;70(5):769-778.
Pat Hoddinott is a health service researcher and a GP who observes how well-meaning initiatives can detract from high quality basic primary care
Competing interests: No competing interests
The authors have confused a range of different issues facing an
evaluation of a community wellness programme. Whatever the problems of
randomisation and standardisation, all kinds of detailed monitoring would
provide some valuable research data, even with imperfections. To take
just one example, different projects, the authors tell us, used different
methods to contact their target population groups. Could they document
how they did it and the response rate? Is that too much to ask? At least
we would know something about getting responses to initiatives of this
In my experience, there are several key problems in this area. One
is that, while I have never met anyone in the NHS who argues that every
drug is 100 per cent effective, those involved in community initiatives
seem generally convinced that their interventions work, so unless made to
evaluate thoroughly by funding bodies, evaluation is not high on their
agenda. This often stems, quite legitimately, from a concern to help
groups with poor health and poor life chances. But if resources are
limited, we need to find the best ways of helping such groups, not assume
that we already know this. A second problem is that existing services,
already in place and not part of any quasi-experiment or test of
effectiveness, frequently collect almost no good quality data. As a
result, if we evaluated new ideas effectively we would still be stuck with
potentially ineffective services. A related issue is that the culture of
data collection and evaluation is poorly developed, making data collection
on new initiatives harder.
Services to improve population wellness should be evaluated as
thoroughly as possible, so that even if some limitations apply to the
approach and methods, we do the best job we can to learn what works and
what does not. Multi-site initiative provide at least a natural
experiment and if we only took the time and trouble to measure what
happens, we would be better equipped in future to decide on the priority
of these services relative to others.
Peter West is a health economist and health service researcher who has attempted to evaluate a range of community interventions, usually without much success!
Competing interests: No competing interests
MRC guidelines and the evaluation of health improvement programmes: are health improvement programmes really too complex to assess their effectiveness?
Imagine an intervention whose effects vary within and between
individuals, depend on subtle interactions between those providing and
receiving the intervention, and where the true level of exposure is hard
to assess. In these circumstances, given all this complexity, who would
ever contemplate conducting a randomised controlled trial (RCT)?
In fact, these are all issues that must be dealt with in drug trials,
let alone in trials of surgery, psychosocial therapies and other more
obviously complex interventions of health protection (e.g. hand washing in
the slums of Pakistan ), patterns of care (e.g. midwife versus
obstetrician care [2 3]), upstream determinants of health (income
supplementation , rehousing ), and health care systems (e.g. in
Mexico ). Mackenzie et al  argue that health improvement is a
special case. They use the specific problems associated with the
implementation of Keep Well to argue that RCTs or designs involving some
form of counterfactual are inappropriate or impossible for evaluating
health improvement interventions in general. In doing so, they ignore the
many successful RCTs of health improvement interventions and misrepresent
the MRC Guidance for the Development and Evaluation of Complex
The MRC guidance on evaluating complex interventions  recommends
that evaluation methods should be chosen according to specific features of
the intervention. RCTs should be considered where the relative harm and
benefits of an intervention are uncertain, there is a high risk of
selection bias, and the additional cost of an RCT is justified by the need
for better quality evidence. Alternatives to RCTs should be considered
when “the intervention is irreversible, necessarily applies to the whole
population, or because large scale implementation is already under way.”
The last of these applies to Keep Well, but the first two do not. The
guidance also recognises that complete standardisation and fidelity to a
rigid protocol is often impractical and inappropriate, and emphasises the
need for thorough process evaluation, and a good theoretically-informed
understanding of the process of change.
There are now enough exemplary RCTs of health improvement
interventions, within the health service (e.g. PROACTIVE ), in other
areas of social policy such as education (e.g. Gatehouse Project ,
SHARE ) and housing (e.g.Moving to Opportunity ), and across
sectors (e.g. Well London ) to suggest that there is no need to treat
health improvement as a special case. On the other hand, the repeated
finding from systematic reviews [14-16] that there is a dearth of good
quality evidence about the health impact of many costly interventions
strongly suggests the need to improve future evaluations.
Keep Well clearly has many features that make evaluation difficult.
But shifting the focus of the evaluation away from effectiveness towards
variability in implementation is a mistake. Why should decision-makers
want to know whether Keep Well was implemented as planned, reached
disadvantaged populations, or became part of normal practice, unless they
also know whether it is effective? Such questions are ‘highly pertinent’,
but only in the context of good information about outcomes.
Arguments that trials are inappropriate to particular settings and
types of intervention used to be commonplace, but have been refuted
convincingly in relation to surgery , psychiatry , as well as
health improvement [19 20]. The MRC Guidance took a pragmatic approach,
warning researchers to beware of blanket statements prescribing specific
methods for settings, and drawing attention to successful and useful
applications of a wide range of non-experimental approaches. It is
misleading and unhelpful to class it as part of a powerful ‘lobby for
randomised trial designs.’
1. Luby SP, Agboatwalla M, Feikin DR, Painter J, Billhimer W, Altaf
A, et al. Effect of handwashing on child health: a randomised cotnrolled
trial. Lancet 2005;366:225-233.
2. Hundley VA, Cruickshank FM, Lang GD, Glazener CMA, Milne JM,
Turner M, et al. Midwife managed delivery unit: a randomised controlled
comparison with consultant led care. BMJ 1994;309:1400-1404.
3. Tucker JS, Hall MH, Howie PW, Reid ME, Barbour RS, Florey CdV, et
al. Should obstetricians see women with normal pregnancies? A multicentre
randomised controlled trial of routine antenatal care by general
practitioners and midwives compared with shared care led by obstetricians.
4. Connor J, Rodgers A, Priest P. Randomised studies of income
supplementation: a lost opportunity to assess health outcomes. Journal of
Epidemiology and Community Health 1999;53(11):725-730.
5. Leventhal T, Brooks-Gunn J. Moving to opportunity: an experimental
study of neighbourhood effects on mental health. American Journal of
Public Health 2003;93(9):1576-1582.
6. King G, Gakidou E, Imai K, Lakin J, Moore RT, Nall C, et al.
Public policy for the poor? A randomised assessment of the Mexican
universal health insurance programme. Lancet 2009;373:1447-1454.
7. Mackenzie M, O'Donnell C, Halliday E, Sridharan S, Platt S. Do
health improvement programmes fit with MRC guidance on evaluating complex
8. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew
M. Developing and evaluating complex interventions: the new Medical
Research Council guidance. BMJ 2008; 337:979-983.
9. Kinmonth A, Wareham NJ, Hardeman W, Sutton S, Prevost AT, Fanshawe
T, et al. Efficacy of a theory-based behavioural intervention to increase
physical activity in an at-risk group in primary care (ProActive UK): a
randomised trial. Lancet 2008;371:41-48.
10. Patton G, Bond L, Carlin JB, Thomas L, Butler H, Glover S, et al.
Promoting social inclusion in secondary schools: a cluster randomised
trial of the Gatehouse Project. American Journal of Public Health
11. Henderson M, Wight D, Raab G, Abraham C, Parkes A, Scott S. The
impact of a theoretically based sex education programme (SHARE) delivered
by teachers on NHS registered conceptions and terminations: final results
of a cluster randomised trial. BMJ 2007;334:133-135.
12. Howden-Chapman P, Pierse N, Nicholls S, Gillespie-Bennett J,
Viggers H, Cunningham M. Effects of improved home heating on asthma in
community dwelling children: randomised controlled trial. . BMJ
13. Wall M, Hayes R, Moore D, Petticrew M, Clow A, Schmidt E, et al.
Evaluation of community level interventions to address social and
structural determinants of health: a cluster randomised controlled trial.
BMC Public Health 2009;9:207 doi:10.1186/1471-2458-9-207.
14. Ogilvie D, Egan M, Hamilton V, Petticrew M. Promoting walking and
cycling as an alternative to using cars: systematic review. BMJ
15. Thomson H, Thomas S, Sellstrom E, Petticrew M. The Health
impacts of housing improvement: A systematic review of intervention
studies from 1887 to 2007. American Journal of Public Health
16. Bambra C, Gibson M, Sowden A, Wright K, Whitehead M, Petticrew
M. Tackling the wider social determinants of health and health
inequalities: evidence from systematic reviews. Journal of Epidemiology
& Community Health [Epub ahead of print]
17. McCulloch P, Taylor I, Sasako M, Lovett B, Griffin D. Randomised
trials in surgery: problems and possible solutions. BMJ 2002;324:1448-
18. Walwyn R, Wessely S. RCTs in psychiatry: challenges and the
future. Epidemiologia e Psychiatria Sociale 2005;14(3):127-131.
19. Moore L, Graham A, Diamond I. On the feasibility of conducting
randomised trials in education: case study of a sex education
intervention. British Educational Research Journal 2003;29(5):673-689.
20. Rosen L, Manor O, Engelhard D, Sucker D. In defense of the
randomized controlled trial for health promotion research. American
Journal of Public Health 2006;96:1181-1186.
Competing interests: No competing interests