Papers

Identifying appropriate tasks for the preregistration year: modified Delphi technique

BMJ 1999; 319 doi: http://dx.doi.org/10.1136/bmj.319.7204.224 (Published 24 July 1999) Cite this as: BMJ 1999;319:224
  1. Jane Stewart, research associate (Jane.Stewart{at}ncl.ac.uk)a,
  2. Catherine O'Halloran, lecturer in medical educationa,
  3. Patrick Harrigan, associate postgraduate deana,
  4. John A Spencer, senior lecturerb,
  5. J Roger Barton, readerc,
  6. Stephen J Singleton, lecturer in epidemiologyd
  1. a Postgraduate Institute for Medicine and Dentistry, Newcastle upon Tyne NE2 4AB
  2. b Department of Primary Health Care, Medical School, Newcastle upon Tyne NE2 4HH
  3. c Regional School of Medicine, North Tyneside Hospital, North Shields NE29 8NH
  4. d Department of Epidemiology and Public Health, Medical School
  1. Correspondence to: J Stewart
  • Accepted 12 May 1999

Abstract

Objectives: To identify the tasks that should constitute the work of preregistration house officers to provide the basis for the development of a self evaluation instrument.

Design: Literature review and modified Delphi technique.

Setting: Northern Deanery within the Northern and Yorkshire office NHS executive.

Subjects: 67 educational supervisors of preregistration house officers.

Main outcome measures: Percentage of agreement by educational supervisors to tasks identified from the literature.

Results: Over 61% of communication items, 70% of on call patient care items, 75% of routine patient care items, 45% of practical procedure items, and over 63% of self management items achieved over 95% agreement that they should be part of the house job of preregistration house officers. Poor agreement was found for the laboratory and clinical investigations that house officers could perform with or without supervision.

Conclusions: The tasks of house officers were identified but issues in using this method and in devising a universally acceptable list of tasks for preregistration house officers were apparent.

Key messages

  • More than 100 activities were identified as potential tasks for house officers, and 11 personal abilities were identified as self management skills

  • The ability of preregistration house officers to perform all of the tasks independently would be restricted by their experiences and therefore may depend on the specialty in which they work

  • The deliberation over what are and are not “shared tasks” was evident; some educational supervisors wanted the house officer to be capable of, but not practise, some tasks whereas others did not believe these tasks were within the remit of the house officer

  • The Delphi technique is a useful method for gaining the autonomous opinions of individuals from a large group of geographically distant members

Introduction

The year's post of house officer is being recognised as a critical transitional period.1 2 The content of the house job is, however, difficult to analyse because of its complexity and constant development. We aimed to identify the tasks of preregistration house officers and to devise a method on the basis of these results for following future changes.

The range of work performed by house officers has been investigated by various methodologies including interviews,3 direct observation,4 diary keeping,5 postal questionnaire,6 7 and multiple data sources.8 However, there remains no complete guide to the tasks that should make up a house job. Previous studies have focused principally on what house officers do and not on what they should do. Identifying what doctors should do during their house jobs was necessary, as our study was the first stage in developing a self evaluation instrument for house officers to be used to support the planning of a personal education strategy.

The Delphi technique is a consensus method used to determine the extent of agreement on an issue. The technique involves asking a panel of experts—in this instance educational supervisors—to take part in a series of rounds to identify, clarify, refine, and finally to gain consensus on the particular issue.911 As the panel do not meet, individuals can express their opinion without being influenced by others. To reduce the number of rounds in our study, the tasks were generated from the literature rather than from an initial round of the Delphi technique.

Participants and methods

The panel

Our panel was derived by a two stage process. Firstly, we identified educational supervisors eligible for inclusion in the study, then we identified those within the eligible group who were willing to take part.

We asked all 18 clinical tutors whose NHS hospital trusts were responsible for the training and employment of preregistration house officers in the Northern Deanery to propose educational supervisors for the panel The clinical tutors were asked to include those who had at least 2 years' experience of supervising preregistration house officers, and who were considered to have particular insight into the educational as well as the service function of the preregistration year. We then invited eligible educational supervisors to become part of the panel.

A list of tasks was identified from the literature35 7 1218 and collated under the section headings: 1, communications; 2, on call patient care; 3, routine patient care; 4, laboratory investigations; 5, clinical investigations; 6, practical procedures; and 7, self management. We generated a datasheet of operational definitions, categories of tasks, and space for comments, and we posted this to members of the panel.

Task appraisal

The educational supervisors were asked to accept, reject, or question the inclusion of each task. They were invited to modify the statements and to add new tasks Judgments were made on the basis of whether house officers would be able to perform the task by the end of their preregistration year. Tasks were defined as “any activity carried out by a preregistration house officer and deemed to be appropriate for that grade.”19 The panel were asked to include those tasks that are the “essence” of the educational experience of being a preregistration house officer, and those tasks that should be carried out by preregistration house officers and not tasks that are performed by them because there is no one else available to do them. The educational supervisors were also asked to include those tasks that might not be performed routinely by a house officer, but which the house officers might be called on to do.

Statements that gained over 95% agreement in the first round (round 1) were deemed accepted and were not resubmitted in the second round (round 2). The remaining task statements from round 1 were modified in line with the comments of the educational supervisors. When several suggestions for one task were given, we used the most commonly suggested modification.

Table 1

Section 1: communication tasks. Values are numbers (percentages) of panel accepting statement

View this table:
Table 2

Section 2: on call patient care. Values are numbers (percentages) of panel accepting statement

View this table:

In round 2, the modified task statements were resubmitted to the educational supervisors along with all additional tasks suggested by individual consultants. We also included with this a summary of the results of round 1.

The educational supervisors were asked to return the completed datasheets within 4 weeks. We coded the responses and analysed them by frequency of response with SPSS for windows (version 6.0). The Delphi technique was conducted between April and June 1997. Our results therefore reflect the jobs of house officers as they existed at that time.

Results

Of the 113 educational supervisors (68 physicians and 45 surgeons) proposed by the clinical tutors, 10 (9%) refused to take part and seven (6%) did not respond. Of the remaining 96 (85%), 75 agreed to take part, and 21 asked to see the work before deciding. Overall, we sent out 96 forms (60 physicians and 36 surgeons) in round 1. Two consultants withdrew and so we sent out 94 forms (59 physicians and 35 surgeons) in round 2.

Overall, 74 forms were returned in round 1 of which 64 (67%) were processed (42 physicians, 22 surgeons), and 72 forms were returned in round 2 of which 67 (71%) were processed (45 physicians, 22 surgeons). We did not process datasheets returned after 4 weeks. Forty physicians and 18 surgeons replied to both rounds.

Teaching versus non-teaching hospitals

Analysis of non-respondents in both rounds by employment showed no significant difference between teaching hospitals and non-teaching hospitals (6of 20 (30%) v 26 of 93 (28%) respectively). Eighteen specialties were represented on the panel.

Tables 16 show the task statements from round 2 and those that achieved over 95% agreement in round 1. The items identified from the literature under section 7 (self management) were skills and not tasks. However, as the data from this section were dealt with by the same procedure as the others, we included them here (table 7).

Table 3

Section 3: routine patient care. Values are numbers (percentages) of panel accepting statement

View this table:
Table 4

Section 4: laboratory investigations. Values are numbers (percentages) of panels' response in round 2

View this table:
Table 5

Section 5: clinical investigations. Values are numbers (percentages) of panels' response in round 2

View this table:
Table 6

Section 6: practical procedures. Values are numbers (percentages) of panel accepting statement

View this table:
Table 7

Section 7: self management skills. Values are numbers (percentages) of panel accepting statement

View this table:

Round 2

As round 1 of the Delphi technique was concerned principally with refinement of the task statements, we focus on the data from round 2.

Section 1

In section 1 (communications), 13 (62%) statements achieved over 95% acceptance, and 16 (76%) achieved over 90% acceptance. Suggestions were givenon how to alter three of the five tasks (1.18, 1.19, 1.20) that achieved less than 90% acceptance (table 1).

Comments indicated that task 1.18 would have gained more acceptance if presented as “Giving simple health promotion advice to patients.” Comments on task 1.19 suggested that handling complaints, other than in limited instances, was not the duty of the house officer. No clear guidance for change was given for task 1.20 except that it needed to be made more “explicit.”

Section 2

Section 2 (on call patient care) generated few comments from consultants, with over 95% acceptance for 7 (70%) task statements and over 90% acceptance for all but task 2.3b. One comment on task 2.3b was that it was not a house officer task.

Section 3

In section 3 (routine patient care), 15 (75%) task items achieved over 95% acceptance with only three (15%) items under 90% acceptance. Consultants' comments suggested that task 3.13 should be limited to hand written discharge letters and therefore supported task 3.10b “Completing hand-written discharge forms,” which was accepted in round 1. Comments suggested that altering task 3.15 to “Create a provisional problem list and management plan” might have made this task statement more acceptable.

Sections 4 and 5

For sections 4 and 5 (laboratory and clinical investigations respectively) venous blood sampling, electrocardiography, and simple respiratory function tests all achieved over 70% acceptance as unsupervised tasks whereas lumbar punctures and urinalysis achieved over 70% acceptance as supervised tasks. No investigation achieved over 95% acceptance in either unsupervised or supervised categories, although venous blood sampling came near. Those educational supervisors who accepted the task but did not indicate whether supervised or unsupervised are not included in the results.

For some investigations, acceptance and rejection rates were similar for both unsupervised and rejected categories—for example, urine sampling and sputum sampling. Some rejected a task because they considered it to be a nursing activity whereas others believed the house officer should be capable of performing it. There was parity in responses over all categories for some investigations, for example, Doppler arterial assessment. Comments by consultants stated that experience provided by a job influenced whether the house officer could perform the task independently or not, for example, “… depends on experience, for example, skin biopsy in dermatology ward, Doppler arterial assessment in vascular job, urine microscopy—renal job.”

Some investigations in tasks 4.2 and 5.2 were not seen as within the remit of the house officer. This was also true for tasks 5.3b (computed tomography scan) and 6.5c (injecting: intra-articularly).

Sections 6 and 7

For tasks 6.1 to 6.13 (practical procedures), 9 (47%) achieved over 95% acceptance and 4 (21%) under 90% acceptance. In section 7 (self management skills) of round 1, some of the panel were unhappy to accept some skills as “fully developed” by the end of the preregistration year. Therefore in round 2 for all newly submitted skills the panel were asked to decide whether the skill should still be developing or fully developed by the end of the preregistration year.

Discussion

The findings

The panel believed that the house officer should be able to perform the majority of the identified tasks independently by the end of the year. However, our study also showed those tasks that consultants considered the house officer should perform only after consultation with the consultant or under direct senior medical supervision, or both.

Results suggest that in some jobs the house officer would gain enough experience to execute tasks independently whereas in others they would not. This may indicate that unsupervised execution of all but a few investigations may be dependent upon the specialty into which the preregistration house job is placed.

In round 2, the educational supervisors agreed that all the self management skills were required by preregistration house officers, but opinion differed on how well developed these skills should be by the end of the year. This response may be indicative of the differing values held by individual educational supervisors rather than reflecting judgments on the basis of their knowledge of the house job. It may also indicate that an increase in the number of response choices reduces the chance of agreement being achieved.

The Delphi technique

Although a significant number of tasks achieved a high level of agreement, literature on the Delphi technique does not stipulate at what level consensus can be deemed to have been reached. Therefore we set an arbitrary decision of 95% in round 1. To use this level of acceptance in round 2 would have removed from the final list all laboratory and clinical investigations and those tasks the house officer is said to find demanding, for example, breaking bad news.

The panel

The constitution of the panel depended on the clinical tutors selecting individuals who they believed to be well informed. Although this was thought the most appropriate way of identifying the “experts,” it is acknowledged that this, together with the non-responders, may have caused hidden bias. Insufficient data were available to perform analysis by specialty, and no statistically significant associations were found when analysis was performed by physician versus surgeon classification.

Conclusion

The Delphi technique was useful in gaining the opinions of educational supervisors on the tasks that should be included in the preregistration year, and this technique may prove a useful tool in monitoring future changes to the job. Further work on the items identified by our study could be undertaken to differentiate between the tasks and skills that educational supervisors want the house officer to experience within the preregistration year and those the house officer must perform competently to achieve registration.

For the self evaluation instrument, those items scoring below 50% acceptance will be rejected as tasks for house officers and those with over 90% acceptance will be accepted. Items ranging from 50% to 90% will be further modified in the light of the comments in round 2. The practical and laboratory investigations are undoubtedly part of some house officer jobs and not others. The instrument will record whether the house officers perform these tasks, and educational supervisors will be left to decide whether this is acceptable within their discipline.

The information gained from the self evaluation instrument will be used to encourage discussion between educational supervisors and house officers about the appropriateness, completeness, and quality of the educational programmes set for them. It could also form a potential feedback loop for assessing the effectiveness of the programmes.

Acknowledgments

We thank the Northern region educational supervisors, clinical tutors, and postgraduate centre managers.

Contributors: JS designed the study protocol, collected and analysed the data, and participated in writing the paper. CO'H, PH, and JAS conceived the study, developed the study design, and participated in writing the paper. JRB and SS guided the study design and participated in writing the paper. JS and CO'Halloran will act as guarantors for the paper.

Footnotes

  • Funding Medical and Dental Education Levy and the Northumberland Health Authority.

  • Competing interests SJS was chairman of the Local Medical Workforce Advisory Group while the study was being conducted.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
View Abstract