Intended for healthcare professionals


Developing a quality criteria framework for patient decision aids: online international Delphi consensus process

BMJ 2006; 333 doi: (Published 24 August 2006) Cite this as: BMJ 2006;333:417
  1. Glyn Elwyn (elwyng{at}, research professor, primary medical care1,
  2. Annette O'Connor, professor2,
  3. Dawn Stacey, assistant professor3,
  4. Robert Volk, associate professor4,
  5. Adrian Edwards, research professor, primary medical care1,
  6. Angela Coulter, chief executive5,
  7. Richard Thomson, professor of epidemiology and public health6,
  8. Alexandra Barratt, associate professor, epidemiology7,
  9. Michael Barry, chief, general medicine unit9,
  10. Steven Bernstein, research scientist10,
  11. Phyllis Butow, professor8,
  12. Aileen Clarke, consultant in public health11,
  13. Vikki Entwistle, reader12,
  14. Deb Feldman-Stewart, associate professor13,
  15. Margaret Holmes-Rovner, professor14,
  16. Hilary Llewellyn-Thomas, professor15,
  17. Nora Moumjid, health economist16,
  18. Al Mulley, chief, general medicine division9,
  19. Cornelia Ruland, professor17,
  20. Karen Sepucha, senior scientist9,
  21. Alan Sykes, statistician18,
  22. Tim Whelan, professor, The International Patient Decision Aids Standards (IPDAS) Collaboration19
  1. 1 Department of General Practice, Centre for Health Sciences Research, Cardiff University, Cardiff CF14 4YS,
  2. 2 University of Ottawa and Ottawa Health Research Institute, Clinical Epidemiology Program, Ottawa, ON, Canada K1Y 4E9,
  3. 3 School of Nursing, University of Ottawa, Ottawa, ON, Canada K1H 8M5,
  4. 4 Department of Family and Community Medicine, Baylor College of Medicine, Houston, TX 77098-3915, USA,
  5. 5 Picker Institute Europe, King's Mead House, Oxford OX1 1RX,
  6. 6 Newcastle upon Tyne Medical School, School of Population and Health Sciences, Newcastle upon Tyne NE2 4HH,
  7. 7 Screening and Test Evaluation Program, School of Public Health, University of Sydney, NSW 2006, Australia,
  8. 8 Medical Psychology Research Unit, School of Psychology, University of Sydney,
  9. 9 Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114, USA,
  10. 10 HSR&D Center of Excellence, VA Ann Arbor Healthcare System, USA,
  11. 11 Public Health Resource Unit, Oxford OX4 2GX,
  12. 12 Social Dimensions of Health Institute, University of Dundee, Dundee DD1 4HJ,
  13. 13 Division of Cancer Care and Epidemiology, Cancer Research Institute, Queen's University, Kingston, ON, Canada,
  14. 14 Center for Ethics, Michigan State University, East Lansing, MI 48824, USA,
  15. 15 Center for the Evaluative Clinical Sciences, Department of Community and Family Medicine, Dartmouth Medical School, Hanover, NH, USA,
  16. 16 GRESAC, Centre Léon Bérard, 69008 Lyon, France,
  17. 17 Rikshospitalet Radiumhospitalet, Oslo, Norway,
  18. 18 Acadvent Ltd, 171 Gower Road, Swansea,
  19. 19 Department of Medicine, McMaster University and Juravinski Cancer Centre, Hamilton, ON, Canada
  1. Correspondence to: Glyn Elwyn
  • Accepted 13 July 2006


Objective To develop a set of quality criteria for patient decision support technologies (decision aids).

Design and setting Two stage web based Delphi process using online rating process to enable international collaboration.

Participants Individuals from four stakeholder groups (researchers, practitioners, patients, policy makers) representing 14 countries reviewed evidence summaries and rated the importance of 80 criteria in 12 quality domains ona1to9 scale. Second round participants received feedback from the first round and repeated their assessment of the 80 criteria plus three new ones.

Main outcome measure Aggregate ratings for each criterion calculated using medians weighted to compensate for different numbers in stakeholder groups; criteria rated between 7 and 9 were retained.

Results 212 nominated people were invited to participate. Of those invited, 122 participated in the first round (77 researchers, 21 patients, 10 practitioners, 14 policy makers); 104/122 (85%) participated in the second round. 74 of 83 criteria were retained in the following domains: systematic development process (9/9 criteria); providing information about options (13/13); presenting probabilities (11/13); clarifying and expressing values (3/3); using patient stories (2/5); guiding/coaching (3/5); disclosing conflicts of interest (5/5); providing internet access (6/6); balanced presentation of options (3/3); using plain language (4/6); basing information on up to date evidence (7/7); and establishing effectiveness (8/8).

Conclusions Criteria were given the highest ratings where evidence existed, and these were retained. Gaps in research were highlighted. Developers, users, and purchasers of patient decision aids now have a checklist for appraising quality. An instrument for measuring quality of decision aids is being developed.


  • Embedded ImageAn extra table, a statistical appendix, and a checklist are on

    This is Version 2 of the article, which includes the full list of authors.

    The following people contributed to background evidence documents: J Austoker (UK), H Bekker (UK), J Belkora (USA), C Braddock (USA), P Butow (AU), E Chan (USA), A Charvet (Switzerland), J Davison (Canada), J Dolan (USA), A Fagerlin (USA), J Fowler (USA), D Frosch (USA), P Hewitson (UK), T Hope (UK), M Jacobsen (Canada), M O'Kane (USA), A Kennedy (Switzerland), S Knight (USA), M Kupperman (USA), B Ling (USA), T Marteau (UK), K McCaffery (Australia), M O'Connor (USA), E Ozanne (USA), M Pignone (USA), A Raffle (UK), L Schwartz (USA), S Sheridan (USA), S Stableford (USA), D Stilwell (USA), V Tait (Canada), D Timmermans (Netherlands), L Trevena (Australia), C Wills (USA), S Woloshin (USA), S Ziebland (UK). We also thank the following for their assistance: P Shekelle (USA), J Muir Gray (UK), P Tugwell (Canada), J Wennberg (USA), and the other participants in the consensus process. We acknowledge Raymond Ramirez and J Scott Smith for development and management of the Delphi process website.

  • Contributors GE, AO'C, DS, Michael Barry, Steven Bernstein, Richard Thomson, AC, and AE were responsible for the study concept, design, and management. AO'C, Hilary Llewellyn-Thomas, DS, and the chairs of the evidence groups were responsible for producing background documents. DS and RV were responsible for data acquisition, and GE, AO'C, and DS were responsible for the analysis and interpretation. GE and AO'C drafted the manuscript, and all the authors revised it for important intellectual content. Alan Sykes did the statistical analysis. AO'C and GE obtained funding for the work. GE is the guarantor.

  • Funding Canadian Institutes of Health Research Group grant; Cardiff University internal funding.

  • Competing interests AO'C receives financial support from the not for profit Foundation for Informed Medical Decision Making, Boston. This foundation receives royalties from Health Dialog, a commercial producer and promoter of patient decision aids. Other authors: none declared.

View Full Text