Published 11 July 2008, doi:10.1136/bmj.39541.449306.AD
Cite this as: BMJ 2008;337:a514

Head to Head

Are learning portfolios worth the effort? No

¹ MDCL 3519, McMaster University, 1200 Main St W, Hamilton, ON, Canada 8N 3Z5

Portfolios were introduced with the aim of improving the learning and assessment of doctors. Erik Driessen (doi: 10.1136/bmj.39540.415822.AD) believes that they work well when used correctly, but Geoff Norman remains unconvinced

Learning portfolios, defined by one group as "a collection of a learner’s work that gives evidence of learning and may be used for the purposes of assessment"¹ have received increased attention of educators over the past few years. According to a systematic review by Driessen and colleagues, they have the potential to improve formative and summative assessment at all levels from undergraduate to post-professional education.² The learning portfolio is not just another learning or assessment method. It is really a Jack of all trades, appearing in multiple guises to do almost anything we demand of it for learners at any level from novice to expert. It is an opportunity for students to report on "work done, feedback received, progress made and plans for improving competence."²

How can it adopt so many roles? Simply because the title really is an accurate description—it is a portfolio with individually specified content. Although, this makes for maximum versatility, it also creates difficulties for anyone attempting to decide whether portfolios are good or bad. Unless you can specify what lies inside the folder, it’s difficult to make generalisations about the value of the method. Asking whether learning portfolios are good for learning or assessment is a bit like asking whether drugs are good for treating disease. The answer must be, "It depends." However, fools rush in, so I will attempt some general observations.

Lack of evidence

Although the review found that, consistent with the rhetoric, most people regarded learning portfolios as useful for reflective learning, professional development, etc, this conclusion was based almost entirely on self perception. One study was conducted with practising physicians and examined self reported intention to change their practices.³ The other used interviews of students’ perceptions of their personal and professional development.⁴ It is doubtful whether these self assessments can be treated as legitimate performance measures, particularly in view of the considerable literature indicating that people, including health professionals, have enormous difficulty assessing themselves.⁵ Furthermore, it takes a lot of work to write portfolios and to mark them. Use of portfolios in the final examination at Dundee required at least 3-4 examiner hours for each student,⁶ a total of 400-500 hours of examiner time.

And it’s not clear that the effort is rewarded. In one study, only 35% of trainees thought it provided an opportunity to analyse critical clinical incidents and only 15% thought it analysed critical incidents of professional behaviour⁷; another study found that only about half of trainee doctors and supervisors thought portfolios were "a good idea," and trainees described a "sense of burden" associated with their use.⁸

Although portfolios have been used in summative assessment, occasionally in very high stakes situations,⁶ the evidence of reliability and validity is quite sparse. Several studies of inter-rater reliability showed an average reliability of 0.63, which is only marginally acceptable. Even this seems optimistic to me; portfolios are usually unstructured essays, and reliability of essay rating is notoriously poor.⁹ Further, although many studies show poor reliability, not much is known about why.

What does it show?

Evidence of validity is based on one study,¹⁰ which claimed that that the portfolio was a valid test of reflective ability, since "quality of reflection" assessed by review of the portfolio predicted the final grade. However, the final grade was based in the same portfolio, using different raters and criteria. This might be seen more as evidence of inter-rater, inter-form reliability than of validity. But none of this addresses content validity. The portfolio is likely to be a highly unrepresentative sample of performance because the student generally decides what is included. It remains to be seen how much selective content sampling is affecting validity since no studies have looked at this.¹¹

Does this then imply that there is no role for learning portfolios? Well, it may be that their real value is a side effect of the drug. It may not have much use in summative assessment; there is really no evidence, despite the rhetoric, that it assesses unique aspects of competence validly, and it certainly is more labour intensive than alternatives like multiple choice questions that have proved reliability and validity. It may not be possible to show that the learning portfolio stimulates learning. And if it is applied mechanically, participants clearly view it as nothing more than a labour intensive add-on.² But when it is integrated into the curriculum, it will force students to write something and teachers to spend some time individually with students to review their work. To the extent that it brings teachers and students together, it may have value as a counterpoint to the anonymity of the culture of lecture learning and sit down final examinations that is regrettably far too typical. And that’s not an entirely bad thing.

Cite this as: BMJ 2008;337:a514

---

Competing interests: None declared.

References

1. Grant A, Dornan TL. What is a learning portfolio? Diabet Med 2001;18(suppl 1):1-4.[Web of Science][Medline]
2. Driessen E, van Tartwijk K, van der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ 2007;41:1224-33.[Web of Science][Medline]
3. Campbell CM, Parboosingh JT, Gondocz ST, Babitskaya G, Lindsay E, De Guzman RC, et al. Study of physicians’ use of a software program to create a portfolio of their self directed learning. Acad Med 1996;71:S49-50.[Web of Science][Medline]
4. Gordon J. Assessing students’ personal and professional development using portfolios and interviews. Med Educ 2003;37:335-40.[CrossRef][Web of Science][Medline]
5. Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med 2005;80:46-54.[CrossRef]
6. Davis MH, Ben-David M, Harden RM, Howie P, Ker J, McGhee C, et al. Portfolio assessment in medical students’ final examinations. Med Teach 2001;23:357-66.[CrossRef][Web of Science][Medline]
7. Ryland I, Brown J, O’Brien M, Graham D, Gillies R, Chapman T, et al. The portfolio: how was it for you? Views of F2 doctors form the Mersey Deanery Foundation Project. Clin Med 2006;6:378-80.[Web of Science][Medline]
8. Hrisos S, Illing JC, Burford BC. Portfolio learning for foundation doctors: early feedback on its use in the clinical workplace. Med Educ 2008;42:214-23.[Web of Science][Medline]
9. Norcini JJ, Diserens D, Day SC, Cebul RD, Schwartz JS, Beck LH, et al. The scoring and reproducibility of an essay test of clinical judgment. Acad Med 1990;65:S41-2.[Web of Science][Medline]
10. Driessen EW, Overeem K, van Tartwijk J, van der Vleuten CP, Muijtjens AM. Validity of portfolio assessment: which qualities determine ratings? Med Educ 2006;40:862-6.[CrossRef][Web of Science][Medline]
11. Wimmers PF, Splinter TA, Hancock GR, Schmidt HG. Clinical competence: general ability or case-specific? Adv Health Sci Educ Theory Pract 2007;12:299-314.[CrossRef][Medline]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Article

Are learning portfolios worth the effort? Yes

Erik Driessen
BMJ 2008 337: a513. [Extract] [Full Text]

Online poll

Find out more>>