Simulated surgeries would have advantages

EDITOR—McKinley et al present cogent arguments for directly assessing the consultation competence of general practitioners undergoing revalidation.1 They identify the required attributes of an assessment as reliability, validity, acceptability, feasibility, and educational impact. They do not mention, however, that the methodology must also be capable of setting a consistent standard for pass/fail decisions for every candidate.

The model that they offer depends on the judgment of two observers watching 10 consecutive consultations, a task that they might undertake 12 times in a year. If the prevalence of insufficient competence was 2% they might encounter it once every four years. How could they apply a minimum standard reliably, along with their 1000 assessor colleagues? Just 10 randomly presenting cases will be too few to guarantee an adequate coverage of consultation skills, and in the absence of standardisation it will be hard to achieve a reliability score above 0.5. Furthermore, the patients will behave differently if they know that their doctor is under assessment, thus compromising validity.

The 12 case simulated surgery for the MRCGP (membership of the Royal College of General Practitioners) examination is a model that would overcome these difficulties. The cases are written to test specific skills; the patients are role players who are trained to present the same challenge to each candidate; each examiner observes the same case throughout the circuit and marks the candidate's performance on a predetermined schedule; each candidate is assessed by all 12 examiners, which minimises bias; and the pass mark is set by the aggregation of the judgments of all the examiners, based on the cases they have marked.

This examination has yielded reliability figures (coefficient α) in excess of 0.8 in all six administrations over three years. Face validity is high (“like a locum …