Intended for healthcare professionals

Rapid response to:

Analysis

Do health improvement programmes fit with MRC guidance on evaluating complex interventions?

BMJ 2010; 340 doi: https://doi.org/10.1136/bmj.c185 (Published 01 February 2010) Cite this as: BMJ 2010;340:c185

Rapid Response:

Evaluation of Complex Initiatives - Confusing what is with what could be

The authors have confused a range of different issues facing an
evaluation of a community wellness programme. Whatever the problems of
randomisation and standardisation, all kinds of detailed monitoring would
provide some valuable research data, even with imperfections. To take
just one example, different projects, the authors tell us, used different
methods to contact their target population groups. Could they document
how they did it and the response rate? Is that too much to ask? At least
we would know something about getting responses to initiatives of this
kind.

In my experience, there are several key problems in this area. One
is that, while I have never met anyone in the NHS who argues that every
drug is 100 per cent effective, those involved in community initiatives
seem generally convinced that their interventions work, so unless made to
evaluate thoroughly by funding bodies, evaluation is not high on their
agenda. This often stems, quite legitimately, from a concern to help
groups with poor health and poor life chances. But if resources are
limited, we need to find the best ways of helping such groups, not assume
that we already know this. A second problem is that existing services,
already in place and not part of any quasi-experiment or test of
effectiveness, frequently collect almost no good quality data. As a
result, if we evaluated new ideas effectively we would still be stuck with
potentially ineffective services. A related issue is that the culture of
data collection and evaluation is poorly developed, making data collection
on new initiatives harder.

Services to improve population wellness should be evaluated as
thoroughly as possible, so that even if some limitations apply to the
approach and methods, we do the best job we can to learn what works and
what does not. Multi-site initiative provide at least a natural
experiment and if we only took the time and trouble to measure what
happens, we would be better equipped in future to decide on the priority
of these services relative to others.

Competing interests:
Peter West is a health economist and health service researcher who has attempted to evaluate a range of community interventions, usually without much success!

Competing interests: No competing interests

24 February 2010
Peter A West
Health Economist
Retiring but available for hire!