Research Methods & Reporting

Publication guidelines for quality improvement studies in health care: evolution of the SQUIRE project

BMJ 2009; 338 doi: https://doi.org/10.1136/bmj.a3152 (Published 19 January 2009) Cite this as: BMJ 2009;338:a3152
  1. Frank Davidoff, executive editor1,
  2. Paul Batalden, director2,
  3. David Stevens, director of the quality literature programme 2,
  4. Greg Ogrinc, associate director of the quality literature programme32,
  5. Susan E Mooney, medical director for quality improvement 42
  6. for the SQUIRE development group
  1. 1Institute for Healthcare Improvement, 143 Garden Street, Wethersfield, CT 06109, USA
  2. 2Center for Leadership and Improvement, Dartmouth Institute for Health Policy and Clinical Practice Lebanon, NH 03766, USA
  3. 3White River Junction VA Hospital, White River Junction, VT 05009-0001, USA
  4. 4Alice Peck Day Memorial Hospital, Lebanon, NH 03766, USA
  1. Correspondence to: F Davidoff fdavidoff{at}cox.net
  • Accepted 20 October 2008

Abstract

In 2005 we published draft guidelines for reporting studies of quality improvement, as the initial step in a consensus process for development of a more definitive version. The current article contains the revised version, which we refer to as standards for quality improvement reporting excellence (SQUIRE). This narrative progress report summarises the special features of improvement that are reflected in SQUIRE, and describes major differences between SQUIRE and the initial draft guidelines. It also briefly describes the guideline development process; considers the limitations of and unresolved questions about SQUIRE; describes ancillary supporting documents and alternative versions under development; and discusses plans for dissemination, testing, and further development of SQUIRE.

Introduction

A great deal of meaningful and effective work is now done in clinical settings to improve the quality and safety of care. Unfortunately, relatively little of that work is reported in the biomedical literature, and much of what is published could be described more effectively. Failure to publish is potentially a serious barrier to the development of improvement science, because public sharing of concepts, methods, and findings is essential to the progress of all scientific work, both theoretical and applied. To help strengthen the evidence base for improvement in health care, we proposed draft guidelines for reporting planned original studies of improvement interventions in 2005.1 Our aims were to stimulate the publication of high calibre improvement studies and to increase the completeness, accuracy, and transparency of published reports of that work.

Our initial draft guidelines were based largely on personal experience with improvement work, and were intended only as an initial step toward creation of recognised publication standards. We have now refined and extended that draft, and present here the resulting revised version, which we refer to as the standards for quality improvement reporting excellence or SQUIRE (table). In this narrative progress report, we describe the special features of quality improvement that are reflected in SQUIRE and examine the major differences between SQUIRE and the initial draft guidelines. We also briefly outline the consensus process used to develop SQUIRE, including our responses to critical feedback obtained during that process. Finally, we consider the limitations of and questions about the SQUIRE guidelines, describe ancillary supporting documents and various versions currently under development, and explain plans for their dissemination, testing, and further development.

SQUIRE guidelines (Standards for QUality Improvement Reporting Excellence)

View this table:

Special features of quality improvement

Unlike conceptually neat and procedurally unambiguous interventions such as drugs, tests, and procedures that directly affect the biology of disease, and are the objects of study in most clinical research, improvement is essentially a social process. Improvement is an applied science rather than an academic discipline2; its immediate purpose is to change human performance, rather than generate new, generalisable knowledge,3 and it is driven primarily by experiential learning.4 5 Like other social processes, improvement is inherently context dependent; it is reflexive, meaning that improvement interventions are repeatedly modified in response to outcome feedback, with the result that both its interventions and outcomes are relatively unstable; and it generally involves complex, multicomponent interventions. Although traditional experimental and quasi-experimental methods are important for learning whether improvement interventions change behaviour, they do not provide appropriate and effective methods for addressing the crucial pragmatic (or “realist”) questions about improvement that are derived from its complex social nature: what is it about the mechanism of a particular intervention that works, for whom, and under what circumstances?2 3 6

Using combinations of methods that answer both the experimental and pragmatic questions is not an easy task, because those two contrasting methodologies can sometimes work at cross purposes. For example, true experimental studies are designed to minimise the confounding effects of context, such as the impact of the heterogeneity of local settings, staff and other study participants, resources, and culture, on measured outcomes. But trying to control context out of improvement interventions is both inappropriate and counterproductive because improvement interventions are inherently and strongly context dependent.2 3 Similarly, true experimental studies require strict adherence to study protocols because it reduces the impact of many potential confounders. But rigid adherence to initial improvement plans is incompatible with an essential element of improvement, which is continued modification of those plans in response to outcome feedback (reflexiveness). We have attempted to maintain a balance between experimental and pragmatic (or realist) methodologies in the SQUIRE guidelines; both are important and necessary, and they are mutually complementary.

Differences between SQUIRE and draft guidelines

The SQUIRE guidelines differ in several important ways from the initial draft guidelines. Firstly, as noted, SQUIRE highlights more explicitly the essential and unique properties of improvement interventions, particularly their social nature, focus on changing performance, context dependence, complexity, nonlinearity, adaptation, and iterative modification based on outcome feedback (reflexiveness). Secondly, SQUIRE distinguishes more clearly between improvement practice (planning and implementing improvement interventions) and the evaluation of improvement projects (designing and carrying out studies to assess whether those interventions work and why they do or do not work). Thirdly, SQUIRE now explicitly specifies elements of study design that make it possible to assess both whether improvement interventions work (by minimising bias and confounding) and why interventions are or are not effective (by identifying the effects of context and identifying mechanisms of change). And finally, SQUIRE explicitly addresses the often confusing ethical dimensions of improvement projects and improvement studies.7 8 Other differences between SQUIRE and the draft guidelines are available on the SQUIRE website (www.squire-statement.org).

The development process

The SQUIRE development process was designed to produce consensus among a broad constituency of experts and users on both the content and format of guideline items. It proceeded along the following six lines. We first obtained informal feedback on the utility, strengths, and limitations of the draft guidelines from potential authors in a series of seminars at national and international meetings, as well as from experienced publication guideline developers at the organisational meeting of the EQUATOR network.9 Authors, peer reviewers, and journal editors then “road tested” the draft guidelines as a working tool for editing and revising submitted manuscripts.10 11 Next, we solicited and published written commentaries on the initial version of the guidelines.12 13 14 15 16 We also did a literature review on epistemology, methodology, and the evaluation of complex interventions, particularly in social sciences. In April 2007, we subjected the draft guidelines to intensive analysis, comment, and recommendations for change at a two day meeting of 30 stakeholders. After that meeting, we obtained further critical appraisal of the guidelines through three cycles of a Delphi process with an international group of more than 50 consultants.

Informal feedback

Informal input about the draft guidelines from authors and peer reviewers raised four particularly relevant issues: uncertainty as to which studies the guidelines apply; the possibility that their use might force quality improvement reports into a rigid, narrow format; the concern that their slavish application might result in lengthy and unreadable reports that are indiscriminately laden with detail; and difficulty knowing if, when, and how other publication guidelines should be used in conjunction with guidelines for reporting quality improvement studies.

Deciding when to use the guidelines

Publications on improvement in health care are emerging in four general categories: empirical studies on the effectiveness of quality improvement interventions; stories, theories, and frameworks; literature reviews and syntheses; and the development and testing of improvement related tools and methods (L Rubenstein et al, unpublished data). Our guideline development process has made it clear that the SQUIRE guidelines can and should apply to reports in the first category: original, planned studies of interventions that are designed to improve clinical outcomes by delivering clinically proved care measures more appropriately, effectively, and efficiently.

Forcing articles into a rigid format

Publication guidelines are often referred to as checklists because, like other such documents, they serve as aide-mémoires, which have proved increasingly valuable in managing information in complex systems.17 Rigid or mechanical application of checklists can prevent users from making sense of complex information.18 19 At the same time, however (and paradoxically), checklists, like all constraints and reminders, can serve as important drivers for creativity. The SQUIRE guidelines must therefore always be understood and used as signposts, not shackles.20

Creating longer articles

Improvement is a complex undertaking, and its evaluation can produce substantial amounts of qualitative and quantitative information. Adding irrelevant information simply to “cover” guideline items would be counterproductive; on the other hand, added length that makes reports of improvement studies more complete, coherent, usable, and systematic helps the guidelines meet a principal aim of SQUIRE. Publishing portions of improvement studies only in electronic form can make the content of long articles publicly available while conserving space in print publication.

Conjoint use with other publication guidelines

Most other biomedical publication guidelines are designed to improve the reporting of studies that use specific experimental designs. The SQUIRE guidelines, in contrast, are concerned with the reporting of studies in a defined content area—improvement and safety. These two guideline types are therefore complementary, rather than redundant or conflicting. When appropriate, other specific design related guidelines can and should be used in conjunction with SQUIRE.

Formal commentaries

The written commentaries provided both supportive and critical input on the draft guidelines.12 13 14 15 16 One suggested that the guidelines’ “pragmatic” focus was an important complement to guidelines for reporting traditional experimental clinical science.12 The guidelines were also seen as a potentially valuable instrument for strengthening the design and conduct of improvement research, resulting in greater synergy with improvement practice15 and increasing the feasibility of combining improvement studies in systematic reviews. However, other commentaries on the draft guidelines raised concerns: that they were inattentive to racial and ethnic disparities in care14; that their proposed introduction, methods, results, and discussion (IMRaD) structure might be incompatible with the reality that improvement interventions are designed to change over time13; and that their use could result in a “dumbing down” of improvement science.16 Our responses to these concerns are as follows.

Health disparities

We do not believe it would be useful, even if it were possible, to address every relevant content issue in a concise set of quality improvement reporting guidelines. We do agree, however, that disparities in care are not considered often enough in improvement work, and that improvement initiatives should address this important issue whenever possible. We have therefore highlighted this issue in the SQUIRE guidelines (table, item 13.b.1).

IMRaD structure

The study protocols traditionally described in the methods section of clinical trials are rigidly fixed, as required by the dictates of experimental design.21 In contrast, improvement is a reflexive learning process—that is, improvement interventions are most effective when they are modified in response to outcome feedback. On these grounds, it has been suggested that reporting improvement interventions in the IMRaD format logically requires multiple, sequential pairs of methods and results sections, one pair for each iteration of the evolving intervention.13 We maintain, however, that the changing, reflexive nature of improvement does not exempt improvement studies from answering the four fundamental questions required in all scholarly inquiry: Why did you start? What did you do? What did you find? What does it mean? These same questions define the four elements of the IMRaD framework.22 23

Although some authors and editors might understandably choose to use a modified IMRaD format that involves a series of small sequential methods and results sections, we believe that approach is often both unnecessary and confusing. We therefore continue to support describing the initial improvement plan, and the theory (mechanism) on which it is based, in a single methods section. Because the changes in interventions over time and the learning that comes from making those changes are themselves important outcomes in improvement projects, in our view they belong collectively in a single results section.1

Dumbing down improvement reports

The declared purpose of all publication guidelines is to improve the completeness and transparency of reporting. Because it is precisely these characteristics of reporting that make it possible to detect weak, sloppy, or poorly designed studies, it is difficult to understand how use of the draft guidelines might lead to a dumbing down of improvement science. The underlying concern here therefore seems to have less to do with transparency than with the inference that the draft guidelines failed to require sufficiently rigorous standards of evidence.16 21 We recognise that those traditional experimental standards are powerful instruments for protecting the integrity of outcome measurements, largely by minimising selection bias.21 24 Although those standards are necessary in improvement studies, they are not sufficient because they fail to take into account the particular epistemology of improvement that derives from its applied purpose and social nature. As noted, the SQUIRE guidelines specify methodologies that are appropriate for both experimental and pragmatic (or realist) evaluation of improvement programmes.

Consensus meeting of editors and research scholars

With support from the Robert Wood Johnson Foundation, we undertook an intensive critical appraisal of the draft guidelines at a two day meeting in April 2007. Thirty participants attended, including clinicians, improvement professionals, epidemiologists, clinical researchers, and journal editors, several from outside the United States. Before the meeting, we sent participants a reading list and a concept paper on the epistemology of improvement. In plenary and small group sessions, participants critically discussed and debated the content and wording of every item in the draft guidelines and recommended changes. They also provided input on plans for dissemination, adoption, and future uses of the guidelines. Working from transcribed audiorecordings of all meeting sessions and flip charts listing the key discussion points, a coordinating group (the authors of this paper) then revised, refined, and expanded the draft guidelines.

Delphi process

Following the consensus meeting, we circulated sequential revisions of the guidelines for further comment and suggestions in three cycles of a Delphi process. The group involved in that process included the meeting participants and roughly 20 additional expert consultants. We then surveyed all participants as to their willingness to endorse the final consensus version (SQUIRE).

Limitations and questions

The SQUIRE guidelines have been characterised as providing both too little and too much information: too little, because they fail to represent adequately the many unique and nuanced issues in the practice and evaluation of improvement2 3 4 12 13 14 15 16 21 24 25; too much, because the detail and density of the item descriptions might seem intimidating to authors. We recognise that the SQUIRE item descriptions are much more detailed than those of some other publication guidelines. In our view, however, the complexity of the improvement process, plus the relative unfamiliarity of improvement interventions and of the methods for evaluating them, justify that level of detail, particularly in light of the diverse backgrounds of people working to improve health care. Moreover, the level of detail in the SQUIRE guidelines is quite similar to that of recently published guidelines for reporting observational studies, which also involve considerable complexities of study design.26 To increase the usability of SQUIRE, we are making available a shortened electronic version on the SQUIRE website, accompanied by a glossary of terms used in the item descriptions that may be unfamiliar to users.

Applying SQUIRE

Authors’ interest in using publication guidelines increases when journals make them part of the peer review and editorial process. We therefore encourage the widest possible use of the SQUIRE guidelines by editors. Unfortunately, little is known about the most effective ways to apply publication guidelines in practice. Therefore, editors have been forced to learn from experience how to use other publication guidelines, and the specifics of their use vary widely from journal to journal. We also lack systematic knowledge of how authors can use publication guidelines most productively. Our experience suggests, however, that SQUIRE is most helpful if authors simply keep the general content of the guideline items in mind as they write their initial drafts, then refer to the details of individual items as they critically appraise what they have written during the revision process. The most effective way to use publication guidelines in practice seems to us to be an empirical question; we therefore strongly encourage editors and authors to collect, analyse, and report their experiences in using SQUIRE and other publication guidelines.

Current and future directions

A SQUIRE explanation and elaboration document has been published elsewhere.27 Like other such documents,28 29 30 31 this document provides much of the necessary depth and detail that cannot be included in a set of concise guideline items. It presents the rationale for including each guideline item in SQUIRE, along with published examples of reporting for each item, and commentary on the strengths and weaknesses of those examples.

The SQUIRE website (www.squire-statement.org) will provide an authentic electronic home for the guidelines and a medium for their progressive refinement. We also intend the site to serve as an interactive electronic community for authors, students, teachers, reviewers, and editors who are interested in the emerging body of scholarly and practical knowledge on improvement.

Although the primary purpose of SQUIRE is to enhance the reporting of improvement studies, we believe the guidelines can also be useful for educational purposes, particularly for understanding and exploring further the epistemology of improvement and the methods for evaluating improvement work. We believe, similarly, that SQUIRE can help in planning and executing improvement interventions, carrying out studies of those interventions, and developing skill in writing about improvement. We encourage these uses, as well as efforts to assess SQUIRE’s impact on the completeness and transparency of published improvement studies32 33 and to obtain empirical evidence that individual guideline items contribute materially to the value of published information in improvement science.

Notes

Cite this as: BMJ 2009;338:a3152

Footnotes

  • We thank Rosemary Gibson and Laura Leviton for support of this project; the Institute for Healthcare Improvement for help in hosting the review meeting in Cambridge; and Joy McAvoy for administrative work in coordinating the entire development process.

  • The following people contributed critical input to the guidelines during their development: Kay Dickersin, Donald Goldmann, Peter Goetzsche, Gordon Guyatt, Hal Luft, Kathryn McPherson, Victor Montori, Dale Needham, Duncan Neuhauser, Kaveh Shojania, Vincenza Snow, Ed Wagner, Val Weber. The following participants in the consensus process also provided critical input on the guidelines, and endorsed the final version. Their endorsements are personal and do not imply endorsement by any group, organisation, or agency: David Aron, Virginia Barbour, Jesse Berlin, Steven Berman, Donald Berwick, Maureen Bisognano, Andrew Booth, Isabelle Boutron, Peter Buerhaus, Marshall Chin, Benjamin Crabtree, Linda Cronenwett, Mary Dixon-Woods, Brad Doebbling, Denise Dougherty, Martin Eccles, Susan Ellenberg, William Garrity, Lawrence Green, Trisha Greenhalgh, Linda Headrick, Susan Horn, Julie Johnson, Kate Koplan, David Korn, Uma Kotegal, Seth Landefield, Elizabeth Loder, Joanne Lynn, Susan Mallett, Peter Margolis, Diana Mason, Don Minckler, Brian Mittman, Cynthia Mulrow, Eugene Nelson, Paul Plsek, Peter Pronovost, Lloyd Provost, Philippe Ravaud, Roger Resar, Jane Roessner, John-Arne Røttingen, Lisa Rubenstein, Harold Sox, Ted Speroff, Richard Thomson, Erik von Elm, Elizabeth Wager, Doug Wakefield, Bill Weeks, Hywel Williams, Sankey Williams. All authors contributed substantively to the ideas developed in the SQUIRE project, and provided critical input on the guidelines. All authors reviewed, commented on, and approved the final version of the paper. FD drafted the paper and is the guarantor.

  • Funding: The SQUIRE project was supported in part by a grant from the Robert Wood Johnson Foundation (RWJF grant number 58073).

  • Competing interests: None declared.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References

View Abstract