Sensitivity and specificity of photography and direct ophthalmoscopy in screening for sight threatening eye disease: the Liverpool diabetic eye studyBMJ 1995; 311 doi: https://doi.org/10.1136/bmj.311.7013.1131 (Published 28 October 1995) Cite this as: BMJ 1995;311:1131
- S P Harding, consultant ophthalmic surgeona,
- D M Broadbent, research fellowa,
- C Neoh, registrar in ophthalmologya,
- M C White, consultant physician+b,
- J Vora, consultant physicianb
- aSt Paul's Eye Unit, Royal Liverpool University Hospital, Liverpool L7 8XP
- bDepartment of Endocrinology and Diabetes, Royal Liverpool University Hospital
- Correspondence to: Mr. Harding.+Michael White died on 13 August 1995.
Objective: To evaluate different methods for community based screening for sight threatening diabetic eye disease.
Design: Prospective study.
Setting: Mobile screening unit visiting inner city community clinics; hospital assessment clinic (tertiary centre).
Subjects: 395 diabetic patients registered with four general practices in an inner city location.
Interventions: Community based photography with mydriasis and direct ophthalmoscopy through dilated pupils by an experienced ophthalmologist, both compared with reference standard of slit lamp biomicroscopy by a consultant specialist in medical retinal disease.
Main outcome measures: Sensitivity and specificity of screening method and prevalence of sight threatening diabetic eye disease (moderate preproliferative retinopathy, circinate maculopathy, exudate within 1 disc diameter of fixation, other diabetes related eye disease).
Results: 358 subjects underwent photography, 326 attended hospital clinic for ophthalmoscopy, and six were ungradable on photographs and biomicroscopy, leaving 320 for analysis. Of these 295 (91%) attended clinic within four months of photography. Sensitivity of detection of eye disease by photography was 89% (95% confidence interval 80% to 98%), significantly better than for direct ophthalmoscopy (65% (51% to 79%)). Analysis of patients with false negative results indicated possible improvement of photographic sensitivity to 93% by addition of stereoscopic macular pair photographs. Specificity of detection of sight threatening eye disease was 86% (82% to 90%) for photography and 97% (95% to 99%) for direct ophthalmoscopy.
Conclusions: Since high sensitivity is essential for an effective screening programme, a photographic method should be considered as preferred option in national, community based screening programmes. Even in the hands of an experienced ophthalmologist, direct ophthalmoscopy is limited by weaknesses inherent to the instrument.
A cost effective method of screening for diabetic retinopathy has yet to be established, but high sensitivity and specificity are essential
We compared the effectiveness of two screening methods: community based photography with mydriasis, three overlapping fields, 35 mm transparencies, and a trained grader; and direct ophthalmoscopy performed by an experienced ophthalmologist
Photography achieved acceptable sensitivity of 89% and a specificity of 86%, whereas direct ophthalmoscopy achieved sensitivity of only 65% and a specificity of 97%
From this evidence photography is the preferred method for screening for sight threatening diabetic eye disease
Screening for diabetic retinopathy prevents blindness and is cost effective.1 2 3 4 5 Because of the inadequacies of current screening programmes, however, many diabetic patients never receive treatment before developing severe visual loss.6 Thus diabetic retinopathy remains the commonest cause of registrable blindness in people aged under 65.7 After appropriate screening, early laser photocoagulation prevents severe visual loss in at least half of cases of proliferative diabetic retinopathy8 and moderate visual loss in 50-70% of cases of maculopathy.9 10
Several alternative screening methods exist: direct ophthalmoscopy, various methods of fundus photography, slit lamp biomicroscopy, and fluorescein angiography. Establishing the sensitivity and specificity of each method is a prerequisite for making a realistic judgment about their value in a widespread screening programme.11 Some reports have suggested that direct ophthalmoscopy and non-mydriatic fundus photography fail to detect a large proportion of sight threatening retinopathy.12 13 14 A crucial omission in many published studies has been the lack of an analysis of the sensitivity or specificity of the screening method used.15 16 17 18 19 20 21
In Liverpool a research programme was established in 1991 to evaluate screening for sight threatening eye disease. In this report we present an evaluation of the sensitivity and specificity of two methods widely used in community based screening programmes for diabetic retinopathy, but with novel and unique enhancement of the photographic technique. For the first time in Britain, retinal photography was performed through dilated pupils, with more than one image of each eye, and recorded on 35 mm colour transparencies. Direct ophthalmoscopy was performed through dilated pupils by an experienced ophthalmologist.
Patients and methods
All diabetic patients attending four general practices in Liverpool with disease registers were examined. Community based retinal photography and hospital based direct ophthalmoscopy were performed. Patients already attending an ophthalmologist were included. Approval for the study was obtained from the local ethics committee.
A technician was specifically trained to instil eye drops, measure visual acuity, and perform photography. All patients' visual acuity was measured at 6 metres on a Snellen chart, followed by pupillary dilatation with tropicamide 1.0%. Retinal photography was performed in a mobile unit equipped with a Canon CR4-45NM fundus camera with 35 mm transparencies on Kodachrome 64 film: three overlapping, non-stereoscopic 45° photographs were taken of each eye (see figure). Patients' age, type of diabetes, and duration of diabetes (grouped into time bands) were recorded.
An experienced ophthalmic clinical assistant (DMB) graded the photographs (with arbitration in case of doubt by SPH) with a simplified version of the Wisconsin protocol.22 Modifications were made to the retinopathy indicators: weighting of venous signs and intraretinal microvascular anomalies was increased, weighting of cotton wool spots was reduced, exudates outside the macula were excluded, and no distinction was made between small haemorrhages and microaneurysms. Each of eight determinants of retinopathy were graded by greatest degree in any field, and an overall retinopathy score was assigned (table I). Grading of maculopathy was based on the presence of exudates in the macular region (table II). Other serious disease was included if it was related to the underlying diabetes (for example, central retinal vein or artery occlusion), but cataract and glaucoma were excluded.
A positive screening result (test positive for the purposes of calculating sensitivity and specificity) was defined as the presence of any of the following in either eye: retinopathy >/=level 40, maculopathy >/=level 3, visual acuity </=6/12, other diabetes related eye disease, and unobtainable or ungradable photographs. Sight threatening eye disease (true positive for the purposes of calculating sensitivity and specificity) was defined as the presence of any of the following in either eye: retinopathy >/=level 40, maculopathy >/=level 3, and other diabetes related eye disease.
All patients received a postal appointment for a hospital clinic within three months of photography. Patients who failed to attend received another three appointments and, finally, a telephone call. At the clinic an experienced registrar in ophthalmology (CN) performed direct ophthalmoscopy with mydriasis using a Welch Allen 3.5 V halogen rechargeable ophthalmoscope. The definition of a positive screening result was the same as for photography.
SLIT LAMP BIOMICROSCOPY
A consultant specialist in medical retinal disease (SPH) performed stereoscopic slit lamp biomicroscopy with 60 and 90 dioptre indirect lenses to act as a reference standard. A subjective assessment was made as to whether media opacity would interfere with fundus photography. Biomicroscopy findings were also graded against the simplified version of the Wisconsin protocol.
Data were recorded and extracted using the EpiInfo epidemiological package (USD, Stone Mountain, Georgia, USA). Sensitivity, specificity, and positive predictive value were calculated with standard formulae,23 and 95% confidence intervals were calculated using the formula 2 x (square root)pq/n, where p=sensitivity or specificity expressed as a fraction of 1, q=1 - p, n=number of observations.
Between 1991 and 1993, 395 diabetic patients were identified in four general practices in Liverpool and invited for community based photography at their local health centre. A total of 358 attended for photography and were subsequently called to the hospital clinic. Of these, 326 attended the hospital clinic and six were ungradable from photographs and biomicroscopy, leaving data on 320 for analysis. Of these 320, 233 were examined at the assessment clinic within three months of their photographs and 295 were examined within four months (range 26-191 days).
There was no significant difference in mean age, type of diabetes, and duration of diabetes between the 326 patients who completed both arms of the study and the 32 who did not: mean ages were 60.2 and 60.9 years respectively, the proportion treated with insulin was 24.9% and 25.8%, and there was no significant difference in the duration of diabetes (Mann-Whitney two tailed test, P>0.2). The prevalence of eye disease in our study population, based on the 320 gradable biomicroscopy examinations, was similar to that reported in previous studies24 25: retinopathy (>/=level 40) 4.7%, maculopathy (>/=level 3) 10.3%, and sight threatening eye disease 14.1%.
SENSITIVITY AND SPECIFICITY OF DETECTION
Tables III and IV show the sensitivity and specificity of photography and direct ophthalmoscopy for sight threatening retinopathy, maculopathy, other diabetes related eye disease, and overall sight threatening eye disease. One patient developed a branch vein occlusion in the time between photography and direct ophthalmoscopy, giving one more case in the “other eye disease” group in table IV.
Sensitivity of detection of sight threatening eye disease by photography was 89% (95% confidence interval 80% to 98%), significantly better than for direct ophthalmoscopy (65% (51% to 79%)). Compared with the reference standard, photography missed five patients with sight threatening eye disease, all with maculopathy: four were due to incorrect grading and one had an epiretinal membrane deemed undetectable by photography. Direct ophthalmoscopy missed sight threatening eye disease in 16 patients. Photography missed sight threatening retinopathy and maculopathy in eight and 13 patients respectively, compared with 11 and 10 cases missed by direct ophthalmoscopy. Patients with sight threatening retinopathy who were missed by photography tended to have peripheral venous beading or cotton wool spots with artefacts that rendered grading of peripheral retina difficult. The serious error rate for missed sight threatening eye disease was 1.5% (5/320) for photography and 5.0% (16/320) for direct ophthalmoscopy.
Specificity of detection of sight threatening eye disease was 86% (82% to 90%) for photography and 97% (95% to 99%) for direct ophthalmoscopy. After exclusion of patients with ungradable or unobtainable photographs the positive predictive value of photography was 94% (90% to 98%) and that of direct ophthalmoscopy was 93% (89% to 98%). With inclusion of ungradable patients, the positive predictive value fell to 51% (confidence interval >100) for photography and to 81% (75% to 87%) for direct ophthalmoscopy.
UNOBTAINABLE AND UNGRADABLE EXAMINATIONS
Six of the 326 patients were ungradable on photography, direct ophthalmoscopy, and slit lamp biomicroscopy. Photographs were unobtainable in a further 12 of the remaining 320 patients due to difficulties such as posture and tremor and were ungradable in a further 34, of whom 29 had media opacities. In 18 of these 34 patients media opacity was deemed sufficiently dense at biomicroscopy to prevent photographic screening. The prevalence of sight threatening eye disease was higher in these patients, occurring in four of the 12 with unobtainable photographs and six of the 29 with media opacity. Only seven of the 320 patients were scored ungradable by direct ophthalmoscopy.
Of the 40 patients with true positive results (excluding one who developed a branch vein occlusion between the two arms of the study), there was agreement between the two methods in 29. Of the 10 cases graded and missed by direct ophthalmoscopy, six were detected and four were ungradable by photography. One case of maculopathy that was missed by photography (grader error) was detected by direct ophthalmoscopy.
SUBANALYSIS OF PHOTOGRAPHY
A subanalysis was performed to measure the effect of reducing the level of retinopathy to level 30 (<6 cotton wool spots as surrogate marker for level 40) in the definition of photographic screen positive (table V). Four of the missed cases would have been detected, thereby increasing the sensitivity for retinopathy to 73% (67% to 79%) and reducing specificity to 97% (95% to 99%). As these four patients all had screen positive maculopathy there was no increase in the sensitivity of detection of sight threatening eye disease, but the specificity fell to 84% (80% to 88%).
The measurement of visual acuity during community based screening detected one patient with maculopathy that was missed by photography and increased sensitivity of detection of sight threatening eye disease by 2% to 91% (83% to 99%). However, another 28 false positives were added, and specificity fell by 10% to 76% (71% to 81%). The effect of measuring acuity with direct ophthalmoscopy was to increase sensitivity by 9% to 74% (62% to 86%), but the specificity fell by 20% to 77% (72% to 82%).
Photography as used in our community based screening programme achieved a high sensitivity of 89% and was significantly better than the 65% achieved with direct ophthalmoscopy. Even in experienced hands, direct ophthalmoscopy proved fallible at detecting early degrees of sight threatening maculopathy, accurately counting cotton wool spots, and picking up areas of venous beading and intraretinal microvascular anomalies.
Both screening methods had high specificity for retinopathy and maculopathy. The specificity and positive predictive value of direct ophthalmoscopy for detecting sight threatening eye disease was apparently higher than photography. However, the influence of unobtainable and ungradable examinations was greater on photography. Photographs were unobtainable in 3.7% of patients, and media opacity interfered with photography in 9%, a higher proportion than in a study by Klein et al.16 The prevalence of sight threatening eye disease was higher in these groups than in the rest of the study population, indicating the need for their further assessment by slit lamp biomicroscopy. The positive predictive value of photography at 51% was limited because of the prevalence of cataract in our study population. With these patients being referred for cataract surgery, however, it would be expected that during subsequent rescreening the positive predictive value would approach the 94% seen in patients after exclusion of those with ungradable or unobtainable results in our study.
Alternative reference standards, most notably seven field stereoscopic colour photography and fluorescein angiography, have been advocated as the standard against which research in diabetic retinopathy should be set and offer a permanent record.5 26 27 28 However, seven field photography misses about 17% of the retina, overreading by trained graders has been reported,21 and both methods are expensive and time consuming.12 The reference standard used in this study was not only less expensive, but in Britain it is the standard on which treatment decisions are made. Furthermore, slit lamp biomicroscopy provides better visualisation through cataract and is able to visualise all postequatorial retina.
Photographic screening has been widely advocated mainly using the non-mydriatic method and Polaroid film, but without essential information on sensitivity and specificity.13 18 24 29 30 31 A multicentre study of 3318 diabetic patients reported a disappointing sensitivity for detection of sight threatening disease ranging from 41% to 67%.12 In contrast high sensitivity and specificity was reported in a small study of 62 patients.13 In the absence of mydriasis autonomic neuropathy causes small pupils and intense bilateral pupillary constriction after photography of the first eye, giving unacceptable failure rates.15 19 Polaroid films are difficult to archive, and their cost and poorer detail have led us and others to prefer 35 mm transparencies.13 32
The performance of the photographic method used in our study might be improved by some modifications. The CR4-45NM camera is relatively easy to use so that an unskilled technician can be trained quickly.16 Our protocol was meant to optimise the use of this type of camera by using more than one field, dilated pupils, and 35 mm transparencies and has subsequently been suggested elsewhere.11 26 There is, however, a tendency to underreport retinopathy16 and a minimal stereoscopic capability. An alternative protocol based on two fields has been suggested,26 but this covers less retina and relies on a single image of the macula. Detection of four of the eight cases of level 40 retinopathy missed by photography in our study might have been possible with better magnification or stereoscopic macular pairs. Sensitivity of detection of sight threatening retinopathy improved to a more acceptable level by changing the criterion for a positive result to >/=level 30, although overall performance for sight threatening eye disease was unchanged because these cases screened positive on other parameters.
Our protocol relied on a combination of reduced visual acuity and the presence of exudates within 1 disc diameter of fixation to detect significant maculopathy, which was then assessed stereoscopically in the hospital clinic. Measurement of acuity has been advocated as an inexpensive additional tool in screening for diabetic maculopathy.26 27 In our study only one patient with maculopathy who was missed by photography was detected by having an acuity of </=6/12. There was a 2% increase in sensitivity but a 10% fall in specificity. Measuring the acuity did not appear to improve efficacy but did detect a large number of incidental cataracts. The five cases of maculopathy missed by our photographic technique might have been detected by the addition of a single stereoscopic pair of the posterior pole with a consequent rise in sensitivity to above 90%.
The weaknesses of direct ophthalmoscopy are inherent to the instrument though it has been advocated as a less expensive alternative.2 5 Ophthalmoscopic screening has been found to be ineffective by general practitioners,12 33 34 even after formalised training,35 and by untrained physicians.6 18 19 36 37 Unacceptable rates of serious error of 30-74%,6 26%,14 and 49%37 have been widely reported, even when performed by a highly trained retina specialist.21 In spite of 10 years' experience the ophthalmologist in our study was not able to achieve an acceptable sensitivity, though the rate of serious error was a more acceptable 5%. Direct ophthalmoscopy by optometrists has been suggested as a suitable method for screening, but data on sensitivity and specificity are not available.17 38 We believe that further efforts to involve optometrists in screening for diabetic retinopathy should be directed towards training in the use of stereoscopic biomicroscopy.
Recent publications have suggested that combining direct ophthalmoscopy with retinal photography might improve effectiveness,39 40 but adequate data on sensitivity and specificity are awaited. Among our 40 patients with true positive results, however, there was only one case of sight threatening disease missed by photography that was detected by direct ophthalmoscopy. We think that it should be possible to develop a single screening method with sufficient efficacy to avoid the need of introducing added complexity and sources of error.
Photographic screening in Liverpool has not been restricted to the four practices included in this study. To date 2096 individuals have been screened from 54 practices, indicating the general accessibility of the method. The programme has been funded as a research project, so costs of implementation as a screening service can be only estimated. However, running costs for the Liverpool model--based on staff, consumables, and vehicle costs and assuming a service related activity level of 6000 screen events a year--would be pounds sterling22.70 per screen event.
In 1989 the St Vincent Declaration set a five year target to reduce new blindness due to diabetes by one third or more.41 Our study shows the effectiveness and accessibility of a community based fundus photographic screening programme in detecting sight threatening diabetic eye disease. Since high sensitivity is a prerequisite for a screening programme, photography is significantly more effective than direct ophthalmoscopy. Refinements of the photographic method may further improve sensitivity, but improvements in specificity will be more difficult. Direct ophthalmoscopy has only a limited role in national screening programmes, possibly as a primary screening tool in a two tier strategy. We believe that, on the evidence currently available, a photographic screening protocol including at least dilated pupils, more than one field, and 35 mm transparencies is the method of choice for purchasers of health care.
We thank Drs EMI Williams and S Ellerby of the Department of Public Health, University of Liverpool, the doctors and staff at The Elms, Westway, Priory and Fairfield Medical Centres, and Mrs Y Swain, Mr R Jackson and Mr D Frame, staff of the Liverpool Diabetic Eye Study.
Funding The work was supported by grants from the British Diabetic Association and Liverpool Health Authority.
Conflict of interest None.