Monitoring global health: time for new solutions
BMJ 2004; 329 doi: https://doi.org/10.1136/bmj.329.7474.1096 (Published 04 November 2004) Cite this as: BMJ 2004;329:1096
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
We agree with Murray et al that "enhanced global reporting will
increase government commitment to collect high quality data"(1); WHO and
UNICEF have, since 2001, engaged in a multi-stakeholder improvement
process with countries that has had precisely this result.
As immunization programmes were maturing in countries, the
development of national monitoring systems lagged behind ; recognizing
the artefacts and inaccuracies in immunization coverage figures annually
reported to WHO and UNICEF, the two agencies undertake an annual
systematic review of all available country data, including reported
coverage, independently available survey data, as well as demographic,
programmatic, and socio-political factors that might affect coverage and
accuracy. The results are a transparent attempt to best reflect actual
coverage levels and trends (2). As a result of this process, many
countries have come to recognize the need to strengthen their routine
monitoring systems. WHO, UNICEF, and partners are actively providing
technical support to help countries achieve this.
Simultaneously, the need for accurate data has been signalled by the
Global Alliance for Vaccines and Immunizations (GAVI) reward payment
system, which requires independent verification of reported data.
Between 2001-2004, 45 independently-conducted Data Quality Audits (DQA)
were implemented in 35 countries to identify both the reliability of
reported figures and the causes for inaccuracies (3). Reporting on
whether or not there was evidence of deliberate misreporting is a
requirement of the DQA. To date, none have registered such a finding.
GAVI partners provide follow-up support to address the problems
identified through the DQA; new guidelines and tools to improve recording
and reporting within countries have been developed, including a data
quality self-assessment (DQS) tool targeted at district-level staff (4).
Additional stimuli to improve data accuracy are provided through refresher
training for health staff (5), emphasizing the use of monitoring data to
improve immunization services delivery.
The availability of accurate data is a function of whether the data
are useful at local levels. WHO, UNICEF and other partners actively
promote good monitoring practices and data use. These are essential
components of successful health delivery systems, and this type of
constructive engagement is the only approach sustainable in the long-term.
What is needed now is not the imposition of an arbitrary gold standard
from some new agency, but the continuation of the existing collaboration
of WHO, UNICEF, and partners working together with countries to find
solutions to building strong, sustainable monitoring systems from the
ground up.
Footnotes:
1. The claim made by Murray et al (1) that "no relation existed" between
reported coverage figures and those reported in DHS surveys is a spurious
observation, based on comparing all immunizations reported as administered
against a model-based prediction of only those immunizations delivered at
the correct time and age intervals; it is to be expected that a comparison
of that nature would find differences, and such a difference does not
actually reflect defects in the administrative reports.
References:
(1) Murray CJL, Lopez AD, and Wibulpolprasert S, "Monitoring global
health: time for new solutions", BMJ 2004: 329, 1096-1100.
(2) WHO & UNICEF estimates of National Immunization Coverage,
http://www.who.int/vaccines-surveillance/WHOUNICEF_Coverage_Review/,
Accessed 10 November 2004.
(3) DQA reports are available at
http://www.vaccinealliance.org/home/Support_to_Country/Monitoring_Evalua...,
accessed 10 November 2004.
(4) "DQS Used Systematically by at least 10 countries", GAVI
Workplan,
http://www.vaccinealliance.org/home/General_Information/About_alliance/w...,
accessed 10 November 2004.
(5) World Health Organization Department of Immunization, Vaccines
and Biologicals, "Immunization in Practice: A practical resource guide for
Health workers 2004 update", WHO/IVB/04/06, http://www.who.int/vaccines-
documents/DoxTrng/h4iip.htm, accessed 10 November 2004.
(6) Murray CJL, Shengelia B, Gupta N, Moussavi S, Tandon A, Thieren
M. Validity of reported vaccination coverage in 45 countries. Lancet
2003;362: 1022-7.
L.J. Wolfson, O. Ronveaux, M.E. Birmingham, J-M. Okwo-Bele, World
Health Organization, Department of Immunization, Vaccines and Biologicals,
Geneva
T. Croft, M. Neill, United Nations Children's Fund, Division of
Evaluation, Policy and Planning
T. Godal, Executive Secretary, The Global Alliance for Vaccines and
Immunizations
Competing interests:
All authors are involved in global monitoring of immunization coverage and providing support to countries to strengthen routine monitoring systems.
Competing interests: No competing interests
In their critique of WHO procedures for analysing and presenting
health statistics, Murray et al make a series of misleading statements
about TB monitoring and evaluation. Ironically, part of the reason that
they can criticize the WHO's TB statistics is that, by design, WHO is
completely open about the process of gathering, analysing and presenting
data. We are transparent so as to invite constructive criticism, because
that leads to better surveillance and assessment. The TB statistics, like
all other global health statistics, are imperfect, but the process by
which they are assembled is not the "serial guessing" that Murray claims.
We take issue with several points. First, it is untrue that "no
affordable and feasible methods are currently available to assess
tuberculosis in a community". For example, China has carried out a series
of national disease prevalence surveys that demonstrate the impact of
their DOTS programme (1). And India has just completed an excellent
national survey of TB infection (ref. 2 among others). With WHO's support,
such national surveys are now being done more frequently, and the
resulting data are being used to make better assessments, not only of case
detection, but also to measure impact and progress towards the Millennium
Development Goals (MDGs).
Second, after years of exposure to these statistics, the authors
still do not seem to understand why the calculated case detection rate can
exceed 100%, and how this index is used in planning and evaluation. Any
reader of the methods section in our report (page 13) will see that case
detection is defined as the ratio of smear-positive cases notified in any
one year to the estimated incidence in that year. We point out in the same
place that "the calculated detection rate can exceed 100% if case finding
is intense in an area that has a backlog of chronic cases, if there has
been over-reporting (e.g. double counting) or over-diagnosis, or if
estimates of incidence are too low." Whenever this index comes close to or
exceeds 100%, WHO investigates, as part of the joint planning and
evaluation process with national TB control programmes, which of these
explanations is correct.
Third, we reject absolutely the suggestion that WHO manipulates
global TB statistics so as to advocate for the DOTS strategy, and fails to
expose weaknesses in the data. With reference to Mozambique, the example
chosen by Murray et al, we said in our 2004 report (3) that "case
detection by the DOTS programme was estimated to be 45% for 2002 but,
because the underlying TB incidence is uncertain (as for other countries
in the region), so too is the estimate of case detection." We further
indicated that there was no significant trend in case detection. Small
variations around the estimate of 45% (shown in their graph) are beside
the point; the real and immediate challenge for Mozambique is to gather
better data to confirm that case detection lies in the range 40-50% and
not, say, 20-30%. The longer-term challenge is to measure indicators other
than case detection, such as prevalence and deaths, in order to monitor
progress towards the MDGs.
WHO now routinely collects TB statistics from 200 countries, and the
quality and diversity of the data (including those from Mozambique) are
improving each year. We do not need another global health monitoring
organization that would dilute this effort, and which would do little to
enable countries to use their data to correct their own problems. We need
instead further international support for WHO and its established
partners, who collectively have the mandate and expertise to carry out
this work.
1. China Tuberculosis Control Collaboration. The effect of
tuberculosis control in China. Lancet 2004;364:417-422.
2. Chadha VK, Kumar P, Gupta J, et al. The annual risk of tuberculous
infection in the eastern zone of India. International Journal of
Tuberculosis and Lung Disease 2004;8:537-544.
3. World Health Organization. Global Tuberculosis Control: Surveillance,
Planning, Financing. Geneva: WHO, 2004.
Competing interests:
We work for WHO
Competing interests: No competing interests
Murray et al. are correct to note that better health information is
needed to respond to health challenges, but misguided in their solution.
The problem is most acute in countries with large burdens of death and
disease where unavailability or poor quality of data leaves decision-
making in a vacuum. Is the solution to this information paradox to be
imposed from above or generated from below?
Murray et al. propose establishing a richly funded 'independent'
health monitoring organization which is primarily concerned with global
monitoring. It is difficult to see how the body could be useful to
countries. Murray et al. assume a priori that "... addressing the problems
with global reporting will also fuel greater commitment among countries to
strengthening national health information systems". Countries could
equally reject the findings of such a body as irrelevant. Indeed, we have
evidence that this is the more likely scenario. The proposed top-down
solution would look very much like WHO’s work on global burden of disease
and health system performance which Murray and Lopez led. The emphasis on
global comparability at the expense of country ownership, resulted in a
huge model-building effort with only tenuous links to empirical evidence
and a notable absence of country involvement. Not only did this do little
to address the information paradox, it actually diverted WHO efforts away
from building country capacity to generate, analyze, disseminate and use
information. WHO is now working to redress this imbalance without
neglecting its global monitoring role .
We contend that a bottom-up approach is more likely to be successful.
This is the approach taken by the Health Metrics Network, a global
alliance aiming to increase the availability and use of sound data through
the reform of country health information systems.
(www.who.int/healthmetrics). The premise is that the information systems
needed for country-level decision making will also generate the data
required by donors and international organizations. HMN, which starts off
with a seven year, $50 million grant from the Gates Foundation and smaller
pledges from bilateral donors, has both normative and technical support
functions and will certainly not limit itself to working in a few
countries as stated by Murray et al.
Examples abound of productive interaction between global monitoring
and country health information systems. UNAIDS and WHO produce estimates
of the AIDS epidemic based on methods developed by leading experts
(HIV/AIDS Epidemiological Reference Group). In 2003, participants from
over 100 countries were trained to use the methodology. In addition to
empowering countries to produce their own estimates and stimulating the
use of the data for health action, this approach has demonstrated the
importance of solid surveillance systems. Thus, country capacity building
is producing not only better country data but also better global
monitoring.
As pointed out by Murray et al. the provision of sound and comparable
data by international organizations is essential to maintain credibility.
Credibility is enhanced when all those involved in generating or using
data work together to harmonize estimates using the best possible methods.
During 2004, WHO, UNICEF, the World Bank and the UN Statistics &
Population Divisions collaborated to establish consensus around estimates
of child mortality. The World Bank and WHO are leading an international
household survey platform that includes a central register of surveys with
decentralized microdata repositories as well as a survey data
dissemination toolkit to assist countries (www.surveynetwork.org/home).
Partnerships such as these help reduce overlap and duplication resulting
from externally imposed reporting burdens.
Global number production is a crowded arena, especially in the
context of the Millennium Development Goals. Adding another large
'independent' and competing player does not seem the best route to
improving health information either globally or in countries. Instead, it
would be wise to build upon the current momentum of partnership and
collaboration led by the Health Metrics Network and WHO.
Ties Boerma
Director, Measurement and Health Information Systems, WHO, Geneva
Carla Abou-Zahr
Interim Executive Secretary, Health Metrics Network, Geneva
Competing interests:
TB and CAZ work for the World Health Organization in Geneva and are involved in the development of the Health Metrics Network.
Competing interests: No competing interests
There is no doubt about the need to develop and maintain robust
health information system that is free from political influence. The
Health Promotion Field of the Cochrane Collaboration has also recognized
this need and has identified priority areas of global importance.1 The
Field is in the process of commissioning these reviews. Murray et al
suggestion of an independent global organization sounds similar to the
ideals of the Cochrane Collaboration but seems to have a wider scope2.
One of the problems with these organizations is that these centers are
usually based in a developed country. For example, majority of the
Cochrane collaboration centers are located in the developed world. An
alternative to creating a new body looking at health information could be
to help support establishment of regional or national centers of this
collaboration in the developing world; rather than to have yet another
organization based in a developed country. In my view, this is more likely
to contribute to the development of a robust global health information
system.
1. Systematic reviews of public health in developing countries are in
train. Elizabeth Waters and Jodie DoyleBMJ, Mar 2004; 328: 585.
2. Monitoring global health: time for new solutions Christopher J L
Murray, Alan D Lopez, Suwit Wibulpolprasert BMJ 2004;329:1096-1100,
doi:10.1136/bmj.329.7474.1096
Competing interests:
IM has an active interest in setting up an independent center of research evaluation and dissemination in South Asia.
Competing interests: No competing interests
Murray et al [BMJ 2004; 329: 1096-100] rightly urge for improved
global reporting of health data. They also highlight the “difficulty of
showing national decision makers that good health data can strongly
support decision making”. Why is this?
Perhaps, because data are always subject to interpretation. Their
article is a case in point. They want to establish “an independent
monitoring organisation [costing]…of the order of $50m-70m…per year…to
report on what is spent on health, what health services are delivered, and
the impact of these efforts on population health.” But, there is already a
global health body that undertakes these functions – the World Health
Organization (WHO).
Murray and al argue that WHO is not fit to undertake the job since it
has “too many roles” leading to “tension between advocacy, monitoring, and
evaluation.” But, evidence that this tension prevents WHO monitoring is
weak, at best. Nor, is there an adequate case made for strengthening
WHO’s role in global health monitoring to address the known weaknesses in
monitoring.
In reality, the problems of global health reporting are primarily
national (and sub-national). The fact that the sum of deaths claimed by
different programmes is greater than the actual number of deaths reflects
epidemiological methods and the current limitations in global health
reporting, more than any WHO failure. The ‘global burden of disease’
project, was itself limited by these issues, but it was undertaken as
effectively in WHO as in any other organisation.
Not only are there problems in capacity, understanding the value of
surveillance, and coverage of health services. The reporting of health
events can be politically sensitive or even economically damaging.
Therefore, it is inevitable that political interference will continue
whoever collects the data, unless governments can be shown that ‘honesty
pays’. It is perhaps this failure of WHO that needs addressing. Setting
up yet another organisation needs a stronger analysis of the reasons (and
locus) for failures in global health reporting.
Competing interests:
Both the authors have worked fro WHO
Competing interests: No competing interests
Global Health Monitoring and Evaluation: Should the Developing World Oblige?
The crying need for health information from the under-developed
region of the world is pronounced in the name of making global health
monitoring more effective. However, global needs and priorities are well
grounded on a distinct agenda from that of the local one which is
confronted with insufficient provision against an enhanced need for
multiplicity of health services. In such a circumstance, how does one
rationalise global needs and local provision of information? Concerns are
raised in relation to having global comparability which contradicts the
existing differences of identifying a disease across regions of the world.
A better example to this effect could be in terms of identifying Malaria
or TB in the most-vulnerable prone regions of the world where it is
referred to as `Malaria like fever’ or `chronic coughing’. In such
circumstances a solution to having reliable monitoring of these diseases
could be neither through community-based surveillance nor clinical
testing. The best option is to derive clues from community based surveys
where the information accuracy regarding the disease may be limiting but
the associated conditionings of the disease could be appropriated for its
alleviation. The other issue referred to as a limitation in global
monitoring of health points at incomplete death registration which is a
long awaited phenomenon to be rectified, overall mortality levels have
it’s corresponding implication on adult mortality levels excepting in rare
circumstances of adult-specific disease vulnerabilities. The discipline of
Demography has enough tools for providing indirect estimates and to test
their validity too. Finally development of health information system in
developing countries will perhaps yield the expected result provided
internal incentive structures are put in place for its promotion apart
from its global requirements and priorities. In the current global divide
on priorities of health and health intervention, global norms and
standards of health monitoring is less probable to be free of biases and
subjective assessments.
Competing interests:
None declared
Competing interests: No competing interests