Papers

Video assessment of simple respiratory signs

BMJ 1996; 313 doi: https://doi.org/10.1136/bmj.313.7071.1527 (Published 14 December 1996) Cite this as: BMJ 1996;313:1527
  1. Mike English, lecturer, Nuffield department of medicine, Oxforda,
  2. Laura New, lecturer, Nuffield department of medicine, Oxforda,
  3. Norbert Peshu, directora,
  4. Kevin Marsh, honorary consultant, Nuffield department of medicine, Oxforda
  1. a Clinical Research Centre, KEMRI Kilifi Unit, PO Box 428, Kilifi, Kenya
  1. Correspondence to: Dr Mike English, 10 Beauchamp Lane, Oxford OX4 3LF.
  • Accepted 6 September 1996

Case management algorithms based on simple symptom histories and physical signs are seen as one means of reducing childhood mortality in developing countries. Simple respiratory signs, often accompanied by metabolic acidosis, are important indicators of potentially life threatening malaria.1 Some are also characteristic of severe, acute respiratory infection.2 It is usually assumed that clinicians share a common understanding of these simple signs. Differences in interpretation might, however, both contribute to the overlap apparent in published descriptions of these illnesses and in practice reduce the effectiveness of algorithms.

Methods and results

Twenty five video recordings of children (aged 3–50 months) admitted with acute respiratory infection or malaria to a hospital on the Kenyan coast were made with consent, compiled, and distributed to 30 clinicians for review. They included representatives from six different research units (see acknowledgements) and six government of Kenya clinical officers. A standard questionnaire giving details of the patients' ages and respiratory rates and asking for a “yes” or “no” response to the presence of the signs nasal flaring, indrawing, and deep (Kussmaul's) breathing was completed, independently, by each clinician. An answer of “don't know” could not be given but a record could be rejected on the basis of poor quality. No structured definition of the clinical signs was provided. No discussion of the records was allowed during viewing or between observers viewing at different sittings. All data were double entered and verified by using dBASE IV and analysed with SPSS.

Interobserver agreement was assessed with the (kappa) statistic, a measure of how much better agreement is than expected if observers simply answer “yes” or “no” at random. Values for (kappa) may vary from −1 to +1, with −1 indicating perfect disagreement, 0 the level of agreement expected by chance, and +1 perfect agreement; a (kappa) value >0.6 suggests substantial agreement.3 A (kappa) value cannot be calculated if the number of “yes” or “no” responses given is 0. The consensus opinion of the reference centre was defined as agreement by at least 5 of the 6 observers for the presence or absence of a sign. The responses of individual clinicians not from the reference centre were compared with this consensus response and the proportion with a (kappa) score >/=0.6 compared by the χ2 test.

Of the 2250 individual assessments of simple clinical signs, in only seven (<0.5%) was video information judged inadequate. Although borderline records for which consensus was not achieved by the reference group were excluded from analysis (probably artificially raising absolute (kappa) values), comparison of an individual's responses with the reference group's consensus response showed clear interobserver variability even for these “clearcut” records (fig 1). This was most noticeable for the sign deep breathing, when only 5/23 observers achieved a (kappa) >/=0.6 compared with 19/23 for indrawing (P<0.001) and 20/24 for nasal flaring (P<0.001). The differences remained significant (P<0.005) when only data from the 18 research clinicians were compared with the reference centre.

Fig 1
Fig 1

Agreement between individual observers and the consensus response from the reference centre (some (kappa) values are missing because they could not be calculated when the number of “yes” or “no” responses = 0)

Comment

The video recordings allowed us to assess interobserver variability among 30 clinicians from at least five countries exposed to exactly the same clinical information. Good general agreement between observers and a derived consensus response was found for nasal flaring and indrawing, although, as reported previously,4 this masks sometimes major differences between individuals. Given their use as indicators of disease severity,1 2 this is reassuring. However, the poorer agreement for deep breathing is of concern. Differences in interpretation of this sign—which should identify children with metabolic acidosis, a life threatening complication of severe malaria 1 5 could reduce the effectiveness of a management algorithm for this disease.

Improving interobserver agreement may improve the power of clinical studies, make studies more comparable, and make their results more generalisable. Video recordings may be useful in standardising definitions of simple respiratory signs and could be used to train health workers using algorithms, increasing their effectiveness.

This study is published with the permission of the director of the Kenya Medical Research Institute (KEMRI). We thank the nurses and staff of Kilifi District Hospital and the KEMRI research ward and Margaret Marr for help in making the video recordings and all the observers from the following units: KEMRI, CRC Kilifi, Kenya; KEMRI, CRC Kisumu, Kenya; University Department of Paediatrics/National ARI Programme, Nairobi, Kenya; MRC(UK), Fajara, The Gambia; Muhimbili Medical Centre, Dar es Salaam, Tanzania; CDC, Atlanta, USA. Particular thanks are owed to the following: Professor F E Onyango, Dr N Anstee, Mr E Shungu, Mrs K Amuma, Dr K Mulholland, Dr B Nahlen, Dr J Zucker, Dr S Redd, and Professor M Molyneux.

Footnotes

  • Funding KEMRI and the Wellcome Trust (040313). KM is a Wellcome Trust senior research fellow in clinical science (031342).

  • Conflict of interest None.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
View Abstract