Rapid Responses to:
|
|
Rapid Responses: Rapid Responses published:
-
![[Read Rapid Response]](/icons/misc/sectionDOWN.gif)
Aiming at the right target?
- Wen Bin Liang (24 June 2005)
-
Meningococcal Monsters
- Ron Law, Barbara Sumner Burstyn (24 June 2005)
-
Re: Aiming at the right target?
- Nelly Ninis (27 June 2005)
-
Is the methodology adequate?
- Shane M Tibby (29 June 2005)
-
Is the methodology adequate- a response from the authors
- Nelly Ninis, On behalf of the authors (1 August 2005)
|
|
||||||||||||||||||||||||||||||||||
|
Wen Bin Liang, taking master of public health Curtin University of Technology
Send response to journal:
|
In this study, diagnosis and treatments of meningococcal disease given by ‘hospital team’ is compared with the ‘standard’ practice under ‘ideal’ circumstance.[1] The circumstance/conditions when treatments and diagnosis decisions were made in emergency care department and the circumstance when treatments diagnosis decisions were made by ‘panel members’ who were not under heavy workload, nor having limited time for decision making, while even noticing the kind of diseases which they were evaluating, are significant different, and this is very likely to worsen the image of the emergency practice on diagnosis and on providing treatment for meningococcal disease (the health care professionals may have sufficient training but ,for example, they may not have enough time to look after the patients). The outcome / difference between original decisions on diagnosis and treatments and the ‘standard’ decisions on diagnosis and treatments is at least due to 1) the skill of the emergency professionals on diagnosis and treatment of the disease.[1] 2) the circumstance affects the diagnosis methods in an emergency health care department—with limited time to consult/ observe the patients as children especially young children always could not express their feelings, and high workload. As been described in the article, meningococcal disease may develop fast, and correct diagnosis the disease and the stage of the disease on time is crucial for the outcome of the patients. [1] However there are many factors may affect the diagnosis and treatments provided by the health care professionals, thus in order to improve the health care service it is necessary to identify the key factors which influence the practice.—are there enough doctors or now it is already the most cost- effective way to run the health care system? Reference: 1 Ninis, N. et al The role of healthcare delivery in the outcome of meningococcal disease in children: case-control study of fatal and non-fatal cases BMJ VOLUME 330 25 JUNE 2005 Competing interests: None declared |
||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
|
Ron Law, Independent Risk & Policy Analyst New Zealand, 1008, Barbara Sumner Burstyn
Send response to journal:
|
Statements dealing with diseases that ebb and flo such as meningococcal disease deserve up to date references. The phrase, "remains the most common infectious cause of death in children in many developed countries," warrants current references, not ones 5-10 years old. A debate rages in New Zealand regarding the use of out of date information, exaggerated epidemiology and attenuated risks and exaggerated benefits of an experimental vaccine being used to vaccinate a nations entire population of children. {1} A point that the authors could have mentioned regarding poor treatment for meningococcal disease is that the disease is so rare that most doctors have never seen a patient with meningococcal disease. Even in New Zealand where officials and meningococcal researchers would have the nation believe there was a 'Meningococcal Monster' on the loose, {2} an average GP would see one case of meningococcal disease every nine years based on current levels reported disease. {1} Full Coverage: Meningococcal Gold Rush - 2nd Ed., http://www.scoop.co.nz/stories/HL0506/S00192.htm {2} Immunise Against Meningococcal Monster; Press Release: Ministry of Health Wednesday, 16 February 2005, http://www.scoop.co.nz/stories/GE0502/S00054.htm Competing interests: Independent Risk & Policy Analyst Member of Ministry of Health expert working group that advised the Ministry of Health on the reporting and management of medical injury in the public health system. |
||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
|
Nelly Ninis, Registrar in paediatric infectious disease Great Ormond Street Hospital
Send response to journal:
|
The points raised are important. Reviewing clinical data on patients after the event cannot be the same as being there and of course there are many pressures on those at the coal face that we could not appreciate as we were not there. However our panel was not asked only to diagnose meningococcal disease, a relatively rare condition. They were asked to diagnose the time that children met the criteria for disease complication/organ failure. These criteria were very basic ( as shown in table 1, full text, online) and for the following organ failures; shock, respiratory failure, coma, raised intracranial pressure, and a non-blanching rash. All these disease complications other than the rash are not only seen in meningococcal disease but are seen in other life threatening paediatric conditions. This is important , the lessons from this study are likely to be important for all acute paediatrics. It seems unlikely that the increased risk of death we found associated with children being looked after by non-paediatric trained doctors or junior doctors was simply due to medical staff being too busy to act appropriately. During this study we saw cases that were intensively monitored and had plenty of time lavished on them but inapropriate therapy delivered by those unfamiliar with managing sick children. Improved training and supervision would lead to better recognition of problems in less time, after all that is what experience is. Competing interests: None declared |
||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
|
Shane M Tibby, Consultant Paediatric Intensivist Guy's and St Thomas' NHS Trust, SE1 9RT
Send response to journal:
|
Ninis and colleagues address an important issue: namely whether timely, appropriate and supervised resuscitation of children with meningococcal disease improves outcome.[1] Although the conclusions seem logical, there appear to be multiple anomalies in variable selection, classification and analysis, which may call the conclusions of this study into question. 1. Matching Although cases and controls were matched for age and serotype, they were not matched for disease severity. The authors acknowledge the huge disparity between groups in terms of acuity as measured by the Glasgow meningococcal septicaemia prognostic score (GMSPS). Of note, the proportion of patients with severe illness (GMSPS > 10) was 44% for cases and 8% among controls. They adjust for this in the final model in two ways: as a dichotomous variable (<8, >8) and as an ordinal variable (0-5, 6-10, 11-15). Unfortunately it is unlikely that adjustment will be adequate for a variety of reasons. First, one presumes the categories were chosen on the basis of historic data which show a stepwise increase in likelihood of death with increasing score: the positive predictive value for death ranges from 30 to 74% for a GMSPS >8, and 42 to 93% for a GMSPS >10. [2,3] Contemporary data has highlighted that GMSPS is overly sensitive (overpredicts death); for example the positive predictive value for a score >10 is closer to 10%. [3,4] This weakness in GMSPS has been acknowledged previously by one of the authors as a possible factor in the underpowering of the bacteriocidal permeability increasing protein trial. [5] Second, there is nonconformity to a linear gradient within the three ordinal categories (i.e. the risk of death does not increase evenly between categories). [6] Third, there is also significant prognostic variation within the severe (10-15) category, where death rates increase significantly with scores >12. [3] Heterogeneity of distribution between cases and controls within this category will thus be problematic and may obscure the adjustment even further; perhaps a better dichotomous cut-off would be a GMSPS score of 12. Fourth, the validity of this score when calculated retrospectively is questionable. This point is acknowledged by the authors in the discussion “vital signs were often inadequately documented”. It is also reinforced by the fact that a GMSPS appears to have been calculated in only 93% (133/143) of cases and 86% of controls (306/355, table 3), further weakening the power of model. 2. Variable selection and definitions used As shown by the GMSPS, cases were sicker than controls and were thus likely to require more treatments. This introduces two problems. First, an interaction effect is likely between treatment failure and disease severity; this does not appear to have been examined in the model (in fact there were many more potential interaction effects in this model). [7] Second, and more importantly, by using overly subjective definitions of “treatment failure” the likelihood of this occurring in the case cohort increases, thus producing a self-fulfilling prophecy. The authors state that treatment failure was based upon three references designated as “published protocols of management”. Two of the three references quote articles published by the authors, the third cites the latest Advanced Paediatric Life Suport manual. It is disappointing that neither the European Paediatric Life Support guidelines nor the American consensus conference guidelines were considered when defining standard treatment. [8,9] The latter guidelines were compiled following an extensive literature review by thirty experts, and have been validated, albeit retrospectively.[10] The subjectivity of the management definitions in the present study means that “treatment failure” could be designated for (a) any patient in warm shock with preserved cardiac function who receives noradrenaline as the sole vasoactive agent, (b) patients who receive hypertonic saline rather than mannitol as osmotherapy, and (c) any patient with a reduced Glasgow coma score who has a jugular rather than a femoral central venous line inserted (this allows monitoring of central venous oxygen saturation, and is one of the only monitoring modalities of proven benefit in septic shock). [11] Indeed under these definitions, many of the patients managed in our own intensive care unit (and likely throughout the rest of the UK) would be classed as treatment failures. 3. Overfitting Overfitting refers to creation of a model that is study specific, and not generaliseable to the wider population. The commonest cause for overfitting is inclusion of too many predictor variables relative to the number of outcomes (death); this can be measured by quantifying the degree of shrinkage. [7] This is not the case in the present study; here overfitting is likely to result from poor variable selection, as defined above in points 1 and 2. Clues to overfitting in this study are the extremely wide confidence intervals seen with several of the predictor variables (table 5), and the extraordinarily high r-squared values (0.68 to 0.79). The latter means that death can almost be entirely explained by factors within the model; the corollary being that other variables, such as provision of paediatric intensive care, are largely irrelevant to outcome. Lastly, this study has undoubtedly collected invaluable data about this patient group. It would be of great interest for the reader to see many of the data concerning the presentation and treatment of these children in quantitative rather than qualitative form. Examples include the time elapsed between registration in A&E and assessment by a clinician, timing and content of nursing observations performed, time taken to administer antibiotics, and actual amount of fluid given. I am rather surprised that the BMJ saw fit to publish this study in its current form. Perhaps this is because it tells us what we want to hear, namely that critically ill children must always be managed by a paediatrician. This is at odds with my own clinical experience, and denigrates the role of other disciplines such as anaesthesia and accident and emergency. Shane Tibby Consultant Paediatric Intensivist Guy’s and St Thomas’ NHS Trust References 1. Ninis N, Phillips C, Bailey L, Pollock JI, Nadel S, Britto J, Maconochie I, Winrow A, Coen PG, Booy R, Levin M. The role of healthcare delivery in the outcome of meningococcal disease in children: case-control study of fatal and non-fatal cases. BMJ 2005;330(7506):1475-82. 2. Castellanos-Ortega A, Delgado-Rodriguez M. Comparison of the performance of two general and three specific scoring systems for meningococcal septic shock in children. Crit Care Med 2000;28:2967-73. 3. Peters MJ, Ross-Russell RI, White D, Kerr SJ, Eaton FE, Keengwe IN, Tasker RC, Wade AM, Klein NJ. Early severe neutropenia and thrombocytopenia identifies the highest risk cases of severe meningococcal disease. Pediatr Crit Care Med 2001;2:225-231. 4. Castellanos-Ortega A, Delgado-Rodriguez M, Llorca J, Sanchez Buron P, Mencia Bartolome S, Soult Rubio A, et al. A new prognostic scoring system for meningococcal septic shock in children. Comparison with three other scoring systems. Intensive Care Med 2002;28:341-51. 5. Giroir BP, Scannon PJ, Levin M. Bactericidal/permeability- increasing protein--lessons learned from the phase III, randomized, clinical trial of rBPI21 for adjunctive treatment of children with severe meningococcemia. Crit Care Med 2001;29(7 Suppl):S130-5. 6. Concato J, Feinstein AR, Holford TR. The risk of determining risk with multivariable models. Ann Intern Med 1993;118:201-10. 7. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361-87. 8. European Resuscitation Council. European Paediatric Life Support Course Manual. London:Resuscitation Council (UK), 2003 9. Carcillo JA, Fields AI; American College of Critical Care Medicine Task Force Committee Members. Clinical practice parameters for hemodynamic support of pediatric and neonatal patients in septic shock. Crit Care Med 2002;30:1365-78. 10. Han YY, Carcillo JA, Dragotta MA, Bills DM, Watson RS, Westerman ME, Orr RA. Early reversal of pediatric-neonatal septic shock by community physicians is associated with improved outcome. Pediatrics 2003;112:793-9. 11. Rivers E, Nguyen B, Havstad S, Ressler J, Muzzin A, Knoblich B, Peterson E, Tomlanovich M; Early Goal-Directed Therapy Collaborative Group. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 2001;345:1368-77. Competing interests: None declared |
||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||
|
Nelly Ninis, clinical research fellow Imperial College of Science, Technology and Medicine, London W2 1PG, On behalf of the authors
Send response to journal:
|
Is the methodology
adequate- a response from the authors. Dr Tibby has raised serious
concerns about the design, analysis and
interpretation of our study and we welcome the opportunity to explain our
conduct. His conclusion that we are reporting what we would like to hear does
not reflect the facts. Our study addressed the question of what action can be taken in order to improve the care of children with meningococcal disease. We have already dealt with most of the problems raised by Dr Tibby by 1) devising objective methodology before the data collection, 2) collecting 3 controls (survivors) per case (death), 3) dealing with potential confounders by means of multivariate regression (with more than one severity variable) and the matched design, and 4) adopting a blinded approach in the evaluation of failure. We are the first to acknowledge that such a study is fraught with potential problems such as overfitting. We challenge Dr Tibby and colleagues to do better. Our study has addressed a most important and highly sensitive issue: does failing or inadequate healthcare delivery contribute to poor outcome in this devastating infection. We too have anecdotal evidence of good and bad care. The fact remains that for many years the overall case fatality rate for this disease has remained unchanged despite great advances in the understanding of the pathophysiology and despite numerous scoring systems predicting outcome having been devised. Meningococcal disease seems to be an emotive subject for many. For all these reasons we set out to conduct this study in a sound and objective way and ensure the findings were robust. The only author who went through all 500 sets of notes was Dr Ninis who also interviewed most of the bereaved families. She played no part in judging the quality of care the children received. This was done on anonymised data by a panel blinded to outcome. Quantification of quality of care is difficult. The study
team spent many months devising a methodology
for this purpose as, at the time, there was no prior model objective enough available
for us to use. We had to have a defined standard of care. As the panel procedure
started in 1999 we used standards available to us then. The standards quoted by
Tibby in his response were unavailable at that time as
they were published in 2003 and 2002. (In fact one of our authors edited the
European Paediatric Life Support manual and the paper by Carcillo
et al cites recommendations made by our group). Hence, we make no apology for
using a protocol written by some of our authors. They drew on extensive
knowledge of meningococcal disease and at the time they were the only group to
publish a protocol based on their experience. This protocol for management was
widely in use in district general hospitals. We used the advanced paediatric
life support manual as a standard text for middle grade paediatricians. As our panel included 3 paediatricians with intensive care
experience we were able to decide whether the type of detailed intensive care
decisions described by Tibby were appropriate and did
not necessarily automatically score these details of care as treatment
failures. It was very uncommon for the panel to have to judge subtleties of
PICU care, only half of all deaths got to a PICU and most treatment failures
occurred in basic management. The analysis of this data set was made complex by the
potential differences in disease severity between the two groups. For this
reason alone we sought three controls per case. Deaths and survivors were not
matched for disease severity on initial selection as this would have been
impossible on a national scale. We must point out that we did not set out evaluate
the GMSPS score, simply to use it as a marker for severity. Although we agree it is overly sensitive in
predicting death it has been the most widely used tool for evaluating severity
of meningococcal disease and has performed extremely well in relationship to
other markers of severity in previous studies. The score was calculated from
the available data on admission for those children for whom it was appropriate
(ie those with a rash). It was not possible to use the newest scoring
for severity quoted as often full blood counts were not done or neutrophil counts recorded even in the prescence
of a non blanching rash. We did not rely entirely on GMSPS to control for
severity. The multivariate analysis
included other factors associated with severity: disease presentation (
septicaemia vs meningitis), serogroup and the presence of organ failure. We did indeed
recognise that those who had more treatments had a greater opportunity for
treatment failures and so only examined failings in individual managements in patients
that had organ
failure. This was done by including the variable ‘needed treatment’ in the
multivariate analysis (For example failure to administer inotropes
was only assessed in a multivariate model that included a variable for ‘being in shock’). We have conducted a previous analysis not shown in this
paper using the subgroup of controls who had organ
failure. This subgroup
represented the sickest controls with disease severity similar to the cases. In
multivariate analysis failure to recognize
disease complications, failure of supervision, absence of paediatric care and
inadequate inotropes were independent risk factors
for death (Table 1). Failure to recognize disease complications and failure in
supervision were highly correlated (logistic regression model correct
predictive classification = 73 to 83%, Fisher's exact test p < 0.001). These two variables
were therefore not completely independent of each other and their significances
as risk factors were reduced in a multivariate model that included them both.
When either was used in the final multivariate model, failure of recognition of
complications resulted in OR = 2.98 (p = 0.005) and failure in supervision resulted
in OR = 3.61 (p = 0.003). Table 1 with these results is included below. The
results are very similar to those published. We have welcomed this
opportunity to respond to Dr Tibby’s comments. Aside
from the obvious error in his response such as asking us to use standards of
care three years before they were published he has also made statistical criticisms
we do not agree with. Dr Tibby suggests we used an
oversimplified multivariate model (many potential interactions were not
included). Then he states that the model was too complex for the available data
(resulting in overfitting). These statements are exaggerated
and potentially contradictory: is Tibby saying that
quality of care is intrinsically impossible to study even with a large study such
as ours? We think not. Although we acknowledge there are many potential problems in a study like this, we feel we had addressed them and stand by our conduct and our findings. The suggestion that paediatricians look after sick children, junior doctors be supervised and protocols followed are not as controversial as Tibby would have us believe. We are grateful to all parents and doctors who made this study possible and to the BMJ for publishing it. The authors TABLE 1 :
Identification of independent risk factors for fatal outcome in multivariate
analysis in all cases and controls and in the subgroup with organ failure.
** Controlling for needing inotropes + These 2 management failures are correlated therefore reducing their independent significance when both in the multivariate model |
||||||||||||||||||||||||||||||||||