Editorials

Assessing operative skill

Needs to become more objective 

Pressure is mounting for surgeons to demonstrate that they can operate well, maintain their performance, and deliver acceptable results. Improved data collection after the Bristol affair may provide more information on the performance of individual surgeons, but a large number of failures are needed before statistical significance is reached,¹ and, for patients, this will be a case of shutting the stable door after the horse has bolted. We need to be able to measure operative skill, set standards, and assess surgeons before any damage is done.

Although many factors influence surgical outcome, the skill of the surgeon in the operating theatre is very important. A skilfully performed operation is 75% decision making and 25% dexterity²; in some specialties, such as minimally invasive surgery, dexterity becomes more important. Though surgeons have formal examinations in surgical knowledge, there is no such requirement to show operative dexterity. Common sense suggests that technical skill does affect outcome. However, despite variation in operative results between surgeons,³ it has been impossible to relate outcome to surgical dexterity. A major reason for this is that we have no way of reliably assessing operative skill. This deficiency in assessment needs to be addressed.⁴

Investigators have observed surgeons in the operating theatre and in the skills laboratory using both objective and subjective criteria. Operative speed is one objective measurement of technical skill and can be important. Robert Liston challenged observers, "Now gentlemen, time me" 28 seconds before placing an amputated limb in the sawdust.⁵ More recently, time has been used to quantify skill in junior⁶ and experienced⁷ surgeons. Measuring competence merely by setting time targets for certain procedures is, however, crude and probably unacceptable. A fast surgeon is not necessarily a good surgeon. Counting the number of procedures performed has also been used as a tool to accredit surgeons⁸ but tells us nothing about how well the surgeon operates.

Finding objective criteria for judging good surgical technique is difficult, and most assessments are purely subjective. Lord Lister was observed to have "none of the dramatic dash and haste of the surgeon of previous times ... he proceeded calmly, deliberately, and carefully." As he told his students, "Anaesthetics have abolished the need for operative speed and they allow time for careful procedure."⁵ Junior surgeons have been ranked using global scores based on subjective criteria, but multiple observers are needed to obtain acceptable reliability.⁹ Gathering panels of experts to watch videos or attend theatre may be possible for a research project but is expensive in manpower and time, and this limits its feasibility in real life.

Assessment may be easier in the surgical skills training laboratory than in theatre. Surgeons may behave differently under simulated conditions, but if the tasks are designed carefully to reflect real surgical practice such tests could fulfil the essential requirements of feasibility, reliability, and validity.¹⁰ Abstract tests of manual dexterity have not stood up to validation¹¹ and would appear to be so far removed from the act of surgery as to be unhelpful in selecting potential surgeons. Subjective methods using structured scoring systems have been shown to be reliable.¹² Although multiple observers were used to rate candidates in terms of "economy" and "fluidity" of movement, it was difficult to validate these scores with subjective rankings of residents in the operating theatre.¹³

Recent work has tracked the movement of laparoscopic surgical instruments in the laboratory. Objective measurements of economy of motion and number of movements made are generated by the assessment device. These criteria have been validated for tasks in both reality and virtual reality. ^{14 15} Devices that objectively and reliably quantify surgical dexterity could have advantages over traditional subjective evaluation, particularly as a screening tool.

A system that can provide unbiased and objective measurement of surgical precision (rather than just speed) could help training, complement knowledge based examinations, and provide a benchmark for certification. A specific and sensitive test of operative competence could also detect an important problems and might improve surgical outcome. Revealing underperformance early would allow for further training or career guidance towards other less practical specialties. The surgical profession needs a reliable and valid method of assessing the operative skill of its members. A driving test may not be a guarantee against accidents but it makes it less likely that you career off the road. Surgeons, the public, and politicians need reassurance.

Ara Darzi, Professor of surgery. 
Simon Smith, Clinical research fellow. 
Nick Taffinder, Specialist registrar. 

Division of Surgery, Anaesthesia, and Intensive Care, Imperial College School of Medicine at St Mary's, London W2 1NY (a.darzi{at}ic.ac.uk)

1. Poloniecki J, Valencia O, Littlejohns P. Cumulative risk adjusted mortality chart for detecting changes in death rate: observational study of heart surgery. BMJ 1998; 316: 1697-1700[Abstract/Free Full Text].
2. Spencer F. Teaching and measuring surgical techniques: the technical evaluation of competence. Bull Amer Coll Surg 1978; 63: 9-12.
3. Fielding LP, Stewart-Brown S, Dudley HA. Surgeon-related variables and the clinical trial. Lancet 1978; ii: 778-779.
4. Smith R. All changed, changed utterly. BMJ 1998; 316: 1917-1918[Free Full Text].
5. Ellis H. Famous Operations. Media, Pennsylvania: Harwal , 1984.
6. Van-Rij AM, McDonald JR, Pettigrew RA, Putterill MJ, Reddy CK, Wright JJAD. Cusum as an aid to early assessment of the surgical trainee. Br J Surg 1995; 82: 1500-1503[Medline].
7. Hanna GB, Shimi SM, Cuschieri A. Randomised study of influence of two-dimensional versus three-dimensional imaging on performance of laparoscopic cholecystectomy. Lancet 1998; 351: 248-251[Medline].
8. European Association of Endoscopic Surgeons. Training and assessment of competence. Surg Endoscopy 1994; 8: 721-722.
9. Winckel CP, Reznick RK, Cohen R, Taylor B. Reliability and construct validity of a structured technical skills assessment form. Am J Surg 1994; 167: 423-427[Medline].
10. Reznick RK. Teaching and testing technical skills. Am J Surg 1993; 165: 358-361[Medline].
11. Graham K, Deary I. A role for aptitude testing in surgery? J R Coll Surg Edinb 1991; 36: 70-74[Medline].
12. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997; 84: 273-278[Medline].
13. Faulkner H, Regehr G, Martin J, Reznick R. Validation of an objective structured assessment of technical skill for surgical residents. Acad Med 1996; 71: 1363-1365[Medline].
14. Taffinder N, Smith S, Jansen J, Ardehali B, Russell R, Darzi A. Objective measurement of surgical dexterity - validation of the Imperial College Surgical Assessment Device (ICSAD). Minimally Invasive Therapy and Allied Techniques 1998; 7(suppl 1): 11.
15. Taffinder N, McManus I, Jansen J, Russell R, Darzi A. An objective assessment of surgeons' psychomotor skills: validation of the MIST-VR laparoscopic simulator. Br J Surg 1998; 85(suppl 1): 75.