Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study
BMJ 2013; 346 doi: https://doi.org/10.1136/bmj.f1706 (Published 02 April 2013) Cite this as: BMJ 2013;346:f1706All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Nijdam and colleagues present a derived and externally validated prediction model for serious bacterial infection (SBI) in febrile children.(1) The title of this article suggest the presented model will help my decision making as an emergency physician. This detailed and very well described study has many strengths, but there are five reasons why the model may have limited clinical use.
First, the authors correctly identify the need to distinguish different types of SBI, with a focus on pneumonia. However, pneumonia is mainly a clinical diagnosis, where the chest radiograph should aid diagnosis making, especially in view of the large inter-rater variability.(2) In this study, the chest radiograph forms the sole basis of the diagnosis as defined by the authors. Furthermore the authors argue that a different validation population is a great test of external validity, however the definitions for (types of) SBI should be the same. Now the definition for pneumonia is different for the derivation and validation groups. More-over, many infiltrates on chest radiographs, especially in young children may represent self-limiting viral pneumonias.
Second, a cut-off comparison for children above and below 1 year is described, which is not congruent with clinical practice. Most clinicians, supported by guidelines, will consider a more extensive workup for young infants (under 90 days) than those that are older than 90 days.(3)
Third, clinical decision rules should be assessed against physician judgment to decide the added value of the rule. Most data in decision rules will need a degree of clinician interpretation for variables in the model, which will be subjective in nature.(4) The marker ‘ill appearance’ (measured differently in derivation and validation cohorts) was 7% and 30% in the two derivation cohorts and 73% in the validation cohort, where the proportion of SBI was 12%, 13% and 25%. This discrepancy suggests that the population were likely differently assessed or the clinician ‘Gestalt’ was more often wrong than right, which would be at odds literature, where ‘Gestalt’ usually performs equivalent or better than decision rules.(5) A clinician judgment comparison (pre-test probability assessment arm before diagnosis of SBI was confirmed) would have had great value in interpreting this data.
Fourth, the difference between clinician assessment and decision rule outcome may have lead to a different practice in test ordering, including C-reactive protein (CRP). The authors acknowledge this in the discussion. However, this leads to an concern that is not discussed – in patients with a fever and a clinical scenario that is suggestive of a UTI or pneumonia in combination with a positive urine microscopy or positive chest radiograph, the additional benefit of a CRP will be very limited. The current message of the manuscript suggests that a CRP is helpful, but this is probably only the case in selective cases where a child is ill-appearing with negative urine and chest radiograph and no other obvious focus for infection. In those situations a CRP may lead to further investigations to look for meningitis, osteomyelitis or septic arthritis.
Lastly, despite the very technical analysis, most clinicians would argue that a sensitivity of 90% in the derivation population (10% false negatives) or even 80% in the final model (20% false negatives) for a low- risk treshold (2.5%) is not acceptable if a clinician wants to exclude meningitis or osteomyelitis.
1. Nijman RG, Vergouwe Y, Thompson M, van Veen M, van Meurs AH, van der Lei J, Steyerberg EW, Moll HA, Oostenbrink R. Clinical prediction model to aid emergency doctors managing febrile children of serious bacterial infections: diagnostic study. BMJ. 2013 Apr 2;346:f1706. doi: 10.1136/bmj.f1706.
2. Levinsky Y, Mimouni FB, Fisher D, Ehrlichman M.Chest radiography of acute pediatric lower respiratory infections: experience versus interobserver variation. Acta Paediatr. 2013 Apr 9. doi: 10.1111/apa.12249
3. Paquette K, Cheng MP, McGillivray D, Lam C, Quach C. Is a lumbar
puncture necessary when evaluating febrile infants (30 to 90 days of age) with an abnormal urinalysis? Pediatr Emerg Care. 2011 Nov;27(11):1057-61. doi: 10.1097/PEC.0b013e318235ea18.
4. Schriger DL, Newman DH. Medical decisionmaking: let’s not forget the physician. Ann Emerg Med. 2012 Mar;59(3):219-20. doi: 10.1016/j.annemergmed.2011.08.015
5. Penaloza A, Verschuren F, Meyer G, Quentin-Georget S, Soulie C, Thys F, Roy PM. Comaprison of the unstructured clinician Gestalt, the Wells Score and the Revised Geneva Score to estimate prestest probability for suspected pulmonary embolism
Ann Emerg Med. 2013. doi:pii: S0196-0644(12)01718-0. 10.1016/j.annemergmed.2012.11.002
Competing interests: No competing interests
At the outset, I would like to congratulate BMJ and authors for a wonderful study. There are a few important principles while developing predication models - Multi-collinearity (two or more predictor variables in a model are highly correlated) and selection of variables as per statistical contribution. In this model many of the variables appear to be correlated and some of the variables may not play important role in the prediction. Some of the variables are subjective like ill appearance. Variance inflation factor (VIF) may be used for assessing the multi-collinearity. The issue can be handled by
1. Increasing the sample size
2. Removing the most intercorrelated variables from analysis
3. Combining variables into a composite variable
4. Using centering
All these issues are important and aim should be to develop a parsimony model.
Competing interests: No competing interests
Re: Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study
Dear editor,
In his letter, colleague Dr. Keijzer raised some valid and important issues concerning the clinical usefulness of our decision model discriminating between serious bacterial infections and self-limiting diseases in febrile children. We thank him for his comments and the invitation for further ongoing discussion.
First of all, he questioned the reference standard used for diagnosing pneumonia in febrile children. He correctly claimed that pneumonia is mostly a clinical diagnosis, and chest radiograph merely an auxiliary diagnostic test aiding decision making. The role of using chest radiograph to identify children with bacterial pneumonia is indeed becoming more and more disputed, and several studies have shown that using chest radiograph for discriminating viral and bacterial pneumonia might not be the ideal gold standard [1-3]. Both interrater variability and abnormalities on chest radiograph caused by viral pneumonia effect the validity of using chest X ray to define bacterial pneumonia. Therefore, its clinical role as a diagnostic tool is probably limited to complicated lower respiratory tract infections such as suspected empyema [4]. However, in the absence of a valid, and ethical justifiable, gold standard for bacterial pneumonia in febrile children at an emergency department, many studies have agreed on focal infiltrates on chest radiographs in combination with suiting clinical features as the best proxy for a reference standard in this field of diagnostic research. [5] At least this definition includes all children with definite lower respiratory tract infections optimizing sensitivity. [6] We feel that the differences in our applied definitions of ‘pneumonia’ between the derivation and validation populations are minor and of very limited clinical consequence, and support external validity of the model. In both derivation studies it was, at the time of study, common practice to perform chest X ray in children with suspected lower respiratory tract infections, and the standardized follow-up procedure ensured limited verification bias. If we have classified some children with viral pneumonia as bacterial pneumonia, this resulted in reduced diagnostic performance of the model. We agree with Keijzer that future validation studies may focus on the indication for antibiotics in those with a raised predicted probability for pneumonia, rather than on the indication for performing a chest radiograph.
A second issue addressed concerns age-specific risk predictions of SBI. We used a linear piecewise approach for modelling age, meaning that we used age as a continuous variable, taking into account a change in risk predictions for SBI at the age of one year [7]. Hence, the age cut-off of 1 year is not a dichotomisation, but rather the age cut-off at which we observed an alteration of the risk prediction pattern. In our validation studies we found a similar behaviour of age as predicting variable. For other SBI, including urinary tract infections, risk was highest in young infants and decreased in the first life-year; in children over the age of one year the risk of having an other SBI then slowly rised again, without getting as high as in young infants. For pneumonia the risk increased significantly in the first life-year, only to increase further only marginally after the age of one year. As our populations included children until the age of 16, only 138 children were aged below 3 months (5% of the total derivation populations, including 4 children with pneumonia and 13 children with other SBI), and hence we might have been underpowered to detect specific additional changes in risk prediction at the age of 3 months. Modelling in a piecewise linear fashion is perhaps an advanced and not easy to grasp method of modelling but ensures maintaining the continuous nature of an important predictor of age, rather than dichotomising. Perhaps surprising was that the overall risk of SBI increased after the age of 1 year old, possibly indicating a selective population being referred to secondary care from primary care: e.g. younger febrile infants will more likely be referred to secondary care and thus constitute a more complete sample, whereas older febrile children will only be referred if warning signs are present and thus constituting a sample with a higher underlying risk of SBI.
In response to the third issue, a large observational study among febrile children showed that ‘clinician’s impression’ performed worse than a model consisting of 26 clinical signs and symptoms. [8] Another study also showed that biomarkers performed much better than a VAS score of the clinician’s impression of the risk of having SBI.[9] These findings were among the main reasons for constructing a decision model including clinical signs, clinician’s impression and a biomarker. Moreover, the variable ‘ill appearance’ as included in the model to reflect the physician’s judgement, differed considerably between the centres, as stated by colleague Dr. Keijzer. This may illustrate how clinical features are elicited and understood in different ways [10], and underlines the necessity to validate prediction rules in various settings, as we did. Nonetheless, despite the wide variability of this variable the model still performed well in validation studies, confirming both the need for validation studies and for decision models incorporating several variables that are interrelated to one another.
Fourthly, considering the dilemma of unnecessary CRP testing, we would like to emphasize that not only the rule-in value of a decision model but also the rule-out value is important. A high sensitivity, as reflected by a low risk estimate, will enable physicians to safely rule out urinary tract infections, pneumonia, or other SBI without any additional diagnostic testing. To rule in SBI, a high risk estimate could lead to a most probable diagnosis of a urinary tract infection or pneumonia without ordering additional testing, and enabling early antibiotic treatment. Additional testing in these children should be performed not solely for establishing a diagnosis, but rather for finding a focus, determining antibiotic resistance pattern, and excluding clinical complications such a empyema. As colleague Dr. Keijzer suggests, a high CRP in ill-appearing febrile children without apparent focus is probably most useful, indicating the need for further diagnostic evaluation. In addition, a model including CRP performed better than a model with clinical signs and symptoms only, and including CRP in decision models has become feasible with minimally invasive and validated bedside tests.
Finally, the topic of risk thresholds is a difficult one, as there are no acceptable risk thresholds for deciding on additional diagnostic testing or initiating treatment. Such thresholds need to be agreed upon in future studies. It also relates to the discussion of the limited predictive value of the physician’s judgment, which could very well reflect thresholds for additional diagnostic testing and initiating therapy rather than the absolute risk of an underlying SBI. Diagnostic models will inherently imply a trade-off between identifying all children with an SBI without performing extensive diagnostic testing in, or treating or hospitalising all febrile children, as illustrated by the presented thresholds of our rule. We would like to point out that our rule, or any other diagnostic decision model, is not intended for the obviously ill appearing children, or for the evidently non-ill appearing children. Decision rules are intended for the children in the diagnostic ‘grey area’ in particular. The model is not perfect, and both a proper safety netting protocol as well as improvements to this model, for example by extending it with other biomarkers such as PCT and, as proposed by colleague Keijzer, a more sophisticated variable for physician’s judgement, should be considered.
References
1. McIntosh, K., Community-acquired pneumonia in children. N Engl J Med, 2002. 346(6): p. 429-37.
2. Michelow, I.C., et al., Epidemiology and clinical characteristics of community-acquired pneumonia in hospitalized children. Pediatrics, 2004. 113(4): p. 701-7.
3. Wilkins, T.R. and R.L. Wilkins, Clinical and radiographic evidence of pneumonia. Radiol Technol, 2005. 77(2): p. 106-10.
4. Bradley, J.S., et al., The management of community-acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis, 2011. 53(7): p. e25-76.
5. Lynch, T., et al., A systematic review on the diagnosis of pediatric bacterial pneumonia: when gold is bronze. PLoS One, 2010. 5(8): p. e11989.
6. Nijman, R.G., et al., Derivation and validation of age and temperature specific reference values and centile charts to predict lower respiratory tract infection in children with fever: prospective observational study. BMJ, 2012. 345: p. e4224.
7. Steyerberg, E.W., Clinical prediction models. A practical approach to development, validation, and updating. Statistics for Biology and Health. 2009, New York: Springer.
8. Craig, J.C., et al., The accuracy of clinical symptoms and signs for the diagnosis of serious bacterial infection in young febrile children: prospective cohort study of 15 781 febrile illnesses. BMJ, 2010. 340: p. c1594.
9. Manzano, S., et al., Markers for bacterial infection in children with fever without source. Arch Dis Child, 2011. 96(5): p. 440-6.
10. Oostenbrink, R., et al., Barriers to translating diagnostic research in febrile children to clinical practice: a systematic review. Arch Dis Child, 2012. 97(7): p. 667-72.
Competing interests: No competing interests