Not playing with a full DEC: why development and evaluation committee methods for appraising new drugs may be inadequate
BMJ 1999; 318 doi: https://doi.org/10.1136/bmj.318.7196.1480 (Published 29 May 1999) Cite this as: BMJ 1999;318:1480
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
EDITOR - Freemantle and Mason are quite right to explore the
difficult task which NICE will face in appraising new drugs. However, it
is disappointing that there was so much in this article which
misrepresented the work of our Development and Evaluation Committee (DEC)
and no recognition of the considerable DEC activity relating to different
interventions which are not new.
One of the main Summary Points stated that: "New drugs should be
appraised in terms of physical outcomes that mean something to doctors and
patients". This principle is a very major influence in all the
deliberations of DEC. For example, in the recent appraisal of beta
interferons for multiple sclerosis, our discussions gave far more weight
to possibly becoming wheelchair bound than to proxy measures such as
changes on the MRI scans. The debate also considered the impact of
relapses on patients and their families, together with a whole range of
issues relating to carers, health service delivery, and the complex matter
of both documented and hidden costs. This breadth of discussion is
exactly what Freemantle and Mason are proposing in their section on
"systematic profiles" and is a routine part of DEC appraisal. They also
advocate "including the right people" - for example general practitioners
"....if a drug will mostly be prescibed in primary care". Our Committee
includes two general practitioners and a lay member, because we believe
that their views are important across the whole range of interventions.
The NICE Appraisal Committee is likely to do the same, including a broad
range of interests.
With regard to getting all the information about new drugs, DEC aims
to address focussed questions about the application of new drugs within
quite a short time frame, pointing out areas of uncertainty in the
published literature, and giving an early "update category" when further
information is likely to become available. It has been our experience
that pharmaceutical companies have been rather slow to release
information, which is often presented in a somewhat selective form, and
which has not been subjected to peer review. Fot these reasons the
Committee is cautious about making decisions that may affect allocation of
NHS resources on the basis of unpublished data.
Estimates of cost effectiveness are difficult, and "cost per QALY"
ranges may be very wide in an attempt to allow for all the possible
variables and assumptions. However, with the explicit "cost per QALY"
thresholds (1.)which our DEC uses, a wide range does not influence our
judgement unless it crosses one of the threshold values: in practice this
seldom happens.
We hope that NICE will build on the experience of the DECs in their
appraisal work.
(1.) Stevens A, Colin-Jones D, Gabbay J. "Quick and Clean":
authoritative health technology assessment for local health care
contracting. Health Trends 1995;27:37-
42
Competing interests: No competing interests
Freemantle and Mason's paper on development and evaluation committees
(DECs)1 - more accurately, development and evaluation systems - contains
several important errors and omissions. This perhaps is a reflection of
their lack of involvement in the work of the DECs and failure to consult
with those that are either producers or consumers of the DEC reports.
First, the DECs were designed for, and have fitted excellently, a
particular purpose, namely to inform difficult commissioning decisions
that need to be taken quickly and cannot await long-term research2.
Without DEC reports such decisions would otherwise have been made with
little or no supportive information. Despite that, almost none of the 100
reports produced so far have been contradicted by later evidence. The DEC
conclusions on Olanzapine and Donepezil are likely to be as durable, (and
are scarcely recognisable in the account that Freemantle and Mason present
in their article). 3,4,5
Second, the suggestion that we need to know the quality of the
information is in fact central to DEC methods, in which systematic search,
synthesis, modelling and sensitivity analysis is always accompanied by a
thorough assessment of information quality. Two of the DEC Committees
formally use a grid in which grading the information is a key
dimension2,6. Furthermore the reports include a sensitivity analysis the
breadth of which reflects the quality of the existing evidence (as in the
donepezil and olanzapine cases). These procedures are at the core of
openness and evidence-based practice.
Third, the point that the overall cost, opportunity costs and
benefits are what counts in appraisal, is also central to the DEC process
and will hardly be news to anyone involved in setting up NICE.
Fourth, while the reports for DECs try, wherever possible, to
determine cost-effectiveness or cost-utility, this is far from being their
only assessment. Cost-utility has been modelled in about two-thirds of
reports. Costs, benefits and risks are always listed ("profiled"). Cost
per QALY estimates (always risky, and not undertaken lightly) are
included, with explicit assumptions, to protect appraisals from those
whose vested interests would wish to distort the results - something quite
easily done with a "ball-park" profile not backed up by cost-utility.
Fifth, DEC reports are produced not by a committee, but by teams
(three in fact) of reviewers representing expertise from public health,
health economics and operational research, and relevant specialist
clinicians. These three groups have formed a collaboration - InterDEC
[http://www.soton.ac.uk/~interdec/] - which importantly avoids
duplication, and continually refines and develops review methodology,
including taking into account a view on the balance between speed and
detail which will be needed when the work is done for a high profile
national decision making body.
Sixth, those producing reports for DECs have been successful in
obtaining unpublished information from, for example, pharmaceutical
companies and we recognise that published data are often insufficient. To
date even where only published information has been used, the consumers of
DEC reports (usually health authority commissioners of services) have been
much better informed by DEC reports than they could have been acting
individually.
Seventh, Freemantle and Mason's criticism that reports for DEC are
insufficiently extensive in coverage is misleading. When appropriate we
frequently examine a range of interventions. However, given that the
consumers of the reports - policy makers and commissioners of services -
usually need an answer rapidly, the value of a report that waited until a
class of drugs had emerged might be lost through the delay. In the real
world of NHS decision making judgements also need to be made about the
disutility of information that arrives too late to serve its purpose.
Yours faithfully
Andrew Stevens
Nick Payne
Ruairidh Milne
Ron Akehurst
Amanda Burls
John Gabbay
1 Freemantle N, Mason J. Not playing with a full DEC: why development
and evaluation committee methods for appraising new drugs may be
inadequate. BMJ 1999;318:1480-2.
2 Stevens, A, Colin-Jones, D, and Gabbay, J. 'Quick and clean':
authoritative health technology assessment for local health care
contracting. Health Trends. 27:37-42, 1995.
3 Cummins C, Stevens A, Kisely S. The use of olanzapine as a first
and second choice treatment in schizophrenia. DPHE report no 10, The
University of Birmingham, 1998.
4 Stein KW. Donepezil in the treatment of mild to moderate senile
dementia of the Alzheimer type (SDAT). Report to the Development and
Evaluation Committee. Bristol: NHS Executive South and West, 1997. (Report
No. 69).
5 Pitt FA, Chilcott J, Golightly P, Sykes J, Whittingham M. A review
of the use of donepezil in the treatment of Alzheimer's disease.
Sheffield: Trent Institute for Health Services Research, Universities of
Leicester, Nottingham and Sheffield, 1997. Guidance Note for Purchasers,
series editor Nick Payne: 97/09.
6 Bandolier. Decisions, decisions: QALYs etc. Bandolier. April 1999
[http:www.jr2.ox.ac.uk/bandolier/] Oxford.
Competing interests: No competing interests
Freemantle and Mason's paper contains several unsubstantiated
statements about horizon scanning for new and emerging health
technologies. Firstly they only mention horizon scanning for
pharmaceuticals whereas the activity of the National Horizon Scanning
Centre (contracted to provide advance notice to the Department of Health
of significant emerging technologies), covers diagnostic tests, imaging
techniques, genetic testing and therapies, biotechnology, biomaterials,
medical implants and other devices, surgical techniques and non-surgical
therapies among others.
The statement that horizon scanning may be an inefficient use of
resources would be better informed if the authors knew anything about its
costs or benefits. Regarding the opportunity costs the idea that such
resources could be better deployed in the rapid appraisal of
pharmaceuticals, does not allow for the fact that you need to know what to
appraise before you can commission such work. The suggestion made that
NICE becomes reliant on pharmaceutical companies informing them of
significant new products and preparing in-house cost impact assessments
not open to independent review, is illogical given the authors view on the
involvement of health service personnel and the companies' likely bias.
Horizon scanning is especially important in relation to non-
pharmaceutical advances where there is no mechanism for their co-ordinated
early identification and their development may be rapid in comparison to
new pharmaceuticals. Non-pharmaceuticals are also developed by a wide
variety of different organisations and effective developments may not be
recognised without special attention being given to them.
Many other countries involved in health technology assessment have
also seen the need for the early identification and prioritisation of
emerging technologies, some are joining a developing European
collaborative working group, EuroScan, to exchange information. In
addition the methods used within the Horizon Scanning Centre build on
processes used within the UK as well as overseas, are increasingly
sophisticated and under continual scrutiny and revision.
The problem of devoting time to new pharmaceuticals, such as
lubeluzole, that are subsequently withdrawn or suspended is a recognised
hazard for all early identification systems. We believe that spending
time investigating advances that are subsequently halted is preferable to
missing the window of opportunity for important topics. The early
identification of the neuraminidase inhibitors, zanamivir and Ro64-0796,
for the prevention and treatment of influenza, has facilitated the
development of a proposal for a systematic review as well as allowed early
consideration of its potential impact on the current immunisation
programme. Without this early identification all levels of the NHS would
be unprepared for their use.
Yours faithfully
Dr Claire Packer
Senior Lecturer in Public Health Medicine
Dr Dogan Fidan
Health Economist
National Horizon Scanning Centre
Dept Public Health & Epidemiology
University of Birmingham
Egbaston
Birmingham B15 2TT
1 Freemantle N, Mason J. Not playing with a full DEC: why
development and evaluation committee methods for appraising new drugs
maybe inadequate. BMJ 1999;318:1480-2.
2 Stevens A, Robert G, Gabbay J. Identifying new health care
technologies in the United Kingdom. International Journal of Technology
Assessment in Health Care 1997;13(1):59-67.
3 Trindade E, Topfer L-A, De Giusti M. Internet information sources
for the identification of emerging health technologies: a starting point.
International Journal of Technology Assessment in Health Care
1998;14(4):644-651.
4 ten Velden GHM. The identification of new health care
technologies by the Health Council of the Netherlands. International
Journal of Technology Assessment in Health Care 1998;14(4):671-679.
5 Carlsson P, Hultin H, Törnwall J. The early experiences of a
national system for the identification and assessment of emerging health
care technologies in Sweden. International Journal of Technology
Assessment in Health Care 1998;14(4):687-694.
Competing interests: No competing interests
Freemantle and Mason's1 discussion of the DEC system misrepresents
the olanzapine report 2 and the Wessex DEC Committee decision. The
Committee did indeed conclude that olanzapine was "value for money"3, but
cautioned evidence was restricted to short term studies, further
comparative trials were needed, and savings, dependent upon favourable
effects on health service use, may not be realised. This decision was
based upon a report which included a systematic review and an economic
analysis. The review highlighted the poor quality of the longer-term
evidence, while the economic analysis warned potential savings from
reductions in inpatient and intensive community care would not be
realised, and gains in QALYs would be small. These conclusions were in
the report's executive summary. Had Freemantle and Mason read it?
Freemantle argued for consideration of all new antipsychotics in the
same report. The Cochrane schizophrenia group, however, has reviewed
individual drugs. The propensity for new antipsychotic drugs to be
withdrawn, highlighted in the report and confirmed by the withdrawal of
sertindole, indicates that atypical antipsychotics do not all share the
same potential to benefit or harm. Variability of patient response
effectively means that the question to be answered is whether a new
antipsychotic is a useful addition to existing treatment possibilities, as
many effective and safe drugs will have a role for some patients4. DEC
reports are requested by purchasers: in this case concern centred on a
particular newly introduced drug likely to have a sizeable impact on
prescribing costs.
As many antipsychotic drugs are used in clinical practice, cost
effectiveness studies will need to model the use of different
antipsychotics. The weakness of the currently available evidence,
particularly in the longer term, indicates that trials of cost
effectiveness are required. Naturalistic trials of the effectiveness and
cost effectiveness of atypical antipsychotics are in progress, but will
not report for some time. Both purchasers and clinicians need an interim
guide to the available evidence.
The two reports singled out by Freemantle concern psychiatric
services, a notoriously difficult area for health technology assessment,
where resource allocation has been a health policy issue for over 30
years. The value of QALYs lies in the explicit attempt (with explicit
assumptions) to present interventions benefiting different patient groups,
including the mentally ill, on a level playing field. Equity, as well as
cost effectiveness, will be an issue for NICE.
Yours faithfully
Carole Cummins
Lecturer
1 Cummins C, Stevens A, Kisely S. The use of olanzapine as a first
and second choice treatment in schizophrenia. DPHE report no 10, The
University of Birmingham, 1998.
2 Freemantle N, Mason J. Not playing with a full DEC: why development
and evaluation committee methods for appraising new drugs may be
inadequate. BMJ 1999;318:1480-2.
3 Stevens, A, Colin-Jones, D, and Gabbay, J. 'Quick and clean':
authoritative health technology assessment for local health care
contracting. Health Trends. 27:37-42, 1995.
4 Steering Committee on Practice Guidelines. Practice guidelines for
the treatment of patients with schizophrenia. Am J Psych 1997;1977
supplement:1-57.
Competing interests: No competing interests
The task for the National Institute for Clinical Excellence (NICE), or any other body which attempt to evaluate the efficacy of a new form of treatment is heavily confounded by publication bias. Some trials started don’t get finished, some finished trials don’t get submitted, some submitted papers don’t get published. Everyone from the research worker to the reader of tabloid newspapers prefers certainty to uncertainty, and is more excited by the ‘new breakthrough’ then the equivocal or the ‘not-proven’.
If NICE were to insist on notification of all proposed trials before they commenced, and only to consider in their evaluation process trials about which they had had prior notification, they would at least know about the trials that were not completed, or not published. They would then have the opportunity at least partly to overcome the effects of publication bias.
Competing interests: No competing interests
The authors' reading was not up to the task
Freemantle and Mason (1) cite the 1997 South & West Development
and Evaluation Committee report on donepezil (2) as an example of the
limitations of the Committee's methods. However, it is clear from their
assertions that their reading was not up to the task.
They imply that information on quality of life was not made available
by the manufacturers. In fact, the DEC report makes clear that we
obtained considerable unpublished information from Eisai and Pfizer. The
quality of life scales used in the trials have very low face validity and
were considered unfit for the purpose of informing the cost utility
analysis. Direct measurement of quality of life in dementia is an area of
great uncertainty (3).
In deriving a cautious estimate of the possible cost utility of
donepezil, we estimated duration of benefit in two ways. Firstly,
directly from the results of the phase III trials, which showed a six
months absence of decline in cognitive function (the limitations of this
as a proxy for disease status were discussed). Secondly, the difference
in the cognitive scores between treatment and control groups were
considered in the context of a longitudinal study of cognitive decline
which confirmed the probable duration of benefit in a less differentiated
population (4).
We then considered what the prevention of cognitive decline for six months
might mean for patients, using the Index of Health Related Quality of
Life. This calculation is acknowledged as being speculative, but amidst
the hype that surrounded the release of donepezil (5) made a useful and
explicit contribution to the debate. Seven clinical experts and the
Alzheimer's Disease Society provided advice and comments during the
preparation of the report, for which the DEC itself acted as a final peer
review.
Freemantle and Mason are therefore wrong in stating that "it is
simply not possible to square the arbitrary costs per QALY estimates in
the conclusion of the Committee's report with the results provided by the
trial". It is clear from the evidence that donepezil provides, on
average, a modest benefit at high cost and that considerable uncertainty
should surround this. What we were successful in doing for the DEC was to
clarify the evidence and, where evidence was lacking, to make judicious
and explicit use of reasonable assumptions in order to inform the
decisions of commissioners and clinicians at the time at which they were
being taken (6).
Yours sincerely
Dr Ken Stein
Consultant in Public Health Medicine
Formerly, Senior Lecturer in Public Health Medicine, University of
Southampton
REFERENCES
1. Freemantle M, Mason J. Not playing with a full DEC: Development
and Evaluation committee methods for appraising new drugs maybe
inadequate. BMJ 1999;318:148-2.
2. South & West Development and Evaluation Committee. Donepezil
in the treatment of mild to moderate senile dementia of the Alzheimer type
(SDAT). Bristol: NHS Research & Development, 1997 (report no. 69).
3. Howard K, Rockwood K. Quality of life in Alzheimer's Disease.
Dementia. 1995;6:113-6
4. Stern Y, Hesdoffer D, Sano M, Mayeux R. Measurement of
functional capacity in Alzheimer's Disease. Neurology. 1992;42:1689-96
5. Stein K, Milne R, Best L. Advertisements for donepezil. BMJ.
1997;315:1623
6. Best, L., Stevens, A., and Colin-Jones, D. Rapid and responsive
health technology assessment: the development and evaluation process in
the South and West region of England. J.Clin.Effect. 2(2):51-56, 1997
Competing interests: No competing interests