Jump to: Page Content, Site Navigation, Site Search,
You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.
BMJ 2008;336:1491-1494 (28 June), doi:10.1136/bmj.39570.749884.BE (published 24 June 2008)
Loes M T Schouten, senior consultant1, Marlies E J L Hulscher, senior researcher2, Jannes J E van Everdingen, senior consultant1, Robbert Huijsman, professor3, Richard P T M Grol, professor and director2
1 Dutch Institute for Healthcare Improvement, PO Box 20064, 3502 LB Utrecht, Netherlands, 2 Centre for Quality of Care Research, University Medical Centre St Radboud, Nijmegen, 3 Institute of Health Policy and Management, Erasmus MC, University Medical Centre Rotterdam
Correspondence to: L M T Schouten l.schouten{at}cbo.nl
Data sources Relevant studies through Medline, Embase, PsycINFO, CINAHL, and Cochrane databases.
Study selection Two reviewers independently extracted data on topics, participants, setting, study design, and outcomes.
Data synthesis Of 1104 articles identified, 72 were included in the study. Twelve reports representing nine studies (including two randomised controlled trials) used a controlled design to measure the effects of the quality improvement collaborative intervention on care processes or outcomes of care. Systematic review of these nine studies showed moderate positive results. Seven studies (including one randomised controlled trial) reported an effect on some of the selected outcome measures. Two studies (including one randomised controlled trial) did not show any significant effect.
Conclusions The evidence underlying quality improvement collaboratives is positive but limited and the effects cannot be predicted with great certainty. Considering that quality improvement collaboratives seem to play a key part in current strategies focused on accelerating improvement, but may have only modest effects on outcomes at best, further knowledge of the basic components effectiveness, cost effectiveness, and success factors is crucial to determine the value of quality improvement collaboratives.
Different types of multiorganisational collaboratives exist, the purpose of which are to improve care.1 2 3 The term quality improvement collaborative seems to be used for different multifaceted packages that focus on accelerating better outcomes.4 Quality improvement collaboratives are used in different clinical areas and organisational contexts and have been adopted by numerous large and small healthcare systems and individual clinics. These initiatives represent substantial investments of time, effort, and funding in the delivery of health care, although estimates of the total investment and applications of the collaborative are not available.5 The strength of the quality improvement collaborative seems to be the relatively efficient use of experts and peers and the exchange of best practices to facilitate and guide improvement.
The earliest well documented activities of quality improvement collaboratives are those of the Northern New England Cardiovascular Disease Study Group, established in 1986, and the Vermont Oxford Network, established in 1988. Another well known approach is the Breakthrough Series developed by the Institute of Healthcare Improvement in 1995. Many of the present approaches of the quality improvement collaborative are based on those of the Breakthrough Series.
A recent non-systematic review6 concluded that the collaborative methodology has important potential to improve outcomes for patients and to facilitate sustainability of quality improvement. Unfortunately this introduction to the collaborative methodology neither considers whether the evaluation of effectiveness is based on a controlled or an uncontrolled study design nor makes clear on what types of quality improvement collaborative the conclusions are based.
Clear evidence of the effectiveness of the methods is lacking, despite the ongoing initiatives of the quality improvement collaborative reflecting different multifaceted intervention packages, the growing number of published papers, good face validity of the model, and facilitators claiming that many professionals appreciate taking part in a collaborative for both professional and organisational development.4 Little is known about the effectiveness of quality improvement collaboratives or specific components that enhance the effectiveness of such collaboratives, and there is hardly any information about their cost effectiveness and sustainability. We therefore assessed the effectiveness of the quality improvement collaborative by systematically reviewing empirical studies.
|
Study selection
We included studies that were written in English, contained data on the effectiveness of care processes or outcomes, were in a healthcare setting, and met the criteria for a quality improvement collaborative derived from a rigorous analysis of the theoretical literature on quality improvement collaboratives (box 2).1 4 7 8 9 To include studies we used the following definition and criteria. A quality improvement collaborative is an organised, multifaceted approach to quality improvement that involves five essential features: there is a specified topic—a subject exists with large variations in care or gaps between best and current practice; clinical experts and experts in quality improvement provide ideas and support for improvement—they identify, consolidate, clarify, and share scientific knowledge and best practice as well as knowledge in quality improvement; a critical mass of multiprofessional teams from multiple sites is willing to improve and share care; a model for improvement focuses on setting clear and measurable targets, collecting data, and testing changes on a small scale to advance reinvention and learning by doing; and the collaborative process involves a series of structured activities (meetings, an active email list, visits to facilitators) in a given time frame to advance improvement, exchange ideas, and share experiences of the participating teams.
|
Data extraction
Each potentially eligible study was independently assessed by two reviewers (LMTS and MEJLH) for inclusion and quality. We assessed the methodological quality of the studies by evaluating the design, method of randomisation, characteristics of control sites, protection against bias, reliable outcome measures, and how sites and patients lost to follow-up had been handled in the analysis (see checklist of the Cochrane Effective Practice and Organisation of Care Review Group at www.epoc.cochrane.org). We used a standardised extraction checklist to obtain data on topics, study design, setting, numbers of participants, characteristics of the collaborative strategy, and relevant results. Two of the authors (LMTS and MEJLH) independently completed this checklist for each study. Disagreements on data extraction and classification of study results were resolved by consensus. We categorised the studies in groups on the basis of their study designs and characteristics of the collaborative strategy used. We could not use formal meta-analytical techniques for pooling results because the studies used many different effect measures.
|
Eleven reports (eight studies)w61-w71 used a "comparison group" study design. One studyw72 used an interrupted time series design (table 1
). Most studies were published in 2004 and 2005. The characteristics of the collaborative strategy used in these nine studies varied. Seven studiesw61- w68 w72 were explicitly based on the Breakthrough Series. Four of these combined the Breakthrough Series with elements of the chronic care model (www.improvingchroniccare.org).w62 w64-w68 Two studiesw69-w71 were based on the Vermont Oxford Network. Box 3 gives an overview of the different collaborative strategies. Table 1
shows the methodological quality of the nine studies. Most studies had important flaws. Limitations included possible differences in baseline measurement, limited data on characteristics of control sites, no specification of blinded assessment, and possible contamination.
|
|
|
All three controlled before and after studies combining the Breakthrough Series with the chronic care modelw62 w65-w68 showed significant improvement of some of the selected process and outcome measures of care. Benedetti et alw62 compared participating providers with non-participating providers and reported significant improvements in, for example, rates of annual diabetes examinations for eyes and feet, and better outcomes for haemoglobin A1c and blood pressure. Mangione-Smith et alw65 and Schonlau et alw66 showed significant improvements for specific items of patient self management (for example, peak flow monitoring in 70% versus 43% of patients) and education (such as instructions for the use of metered dose inhalers in 30% versus 9% of patients) in asthma care. Mangione-Smith et alw65 also reported higher scores on quality of life measures for the intervention group in a survey carried out after measurement. The levels of asthma severity between the intervention group and the control group, however, differed. In the same study Schonlau et alw66 showed significant improvement in satisfaction with communication (overall score 62% v 39%). Asch et alw67 reported significant improvement in specific items on counselling and education (related to diet, drugs, exercise, weight loss, disease management, water weight, and goal setting) in the care of patients with chronic heart failure. The process measures with greater improvement were those with initially low performance rates, and the rates remained below 50% for most educational processes. Significant improvement in two of four measures for the appropriate use of drugs—where baseline rates were good—was less dramatic (appropriate angiotensin converting enzyme inhibitors, 93% v 87% and lipid lowering therapy, 66% v 64%). The changes in the rates of counselling were confirmed in a cross sectional survey among patients.w68
The controlled before and after study based on the Vermont Oxford Networkw69 w70 reported a significant decrease in the rate of infection at six intervention neonatal intensive care units (12.3% v 16.5%) and a decrease in the rate of supplemental oxygen at four neonatal intensive care units (34% v 38.7%). Although the effect among the units on both arms was heterogeneous, the changes show that the intervention units improved at a significantly faster rate in a four year period than did the 66 comparison units. In the same study, Rogowski et alw70 presented data showing that costs may be reduced as a result of participation in the quality improvement collaborative, although the cost savings across hospitals were heterogeneous. Table 3
gives a summary of effectiveness found in the nine controlled studies.
|
Twelve reports representing nine studies (including two recent randomised controlled trials) used a controlled design to measure the effects of the quality improvement collaboratives intervention on processes of care or outcomes of care. The studies were based on different collaborative strategies. Seven studies evaluated the Breakthrough Series, four of these were studies on the Breakthrough Series combined with chronic care model and two were based on the Vermont Oxford Network method. A systematic review of the studies produced moderate positive results. Seven studies (including one randomised controlled trial) showed at least a positive effect of a specific selection of processes of care studied. Two studies (including one randomised controlled trial) did not show any significant effect.
As a result of flaws in the methodological quality of the studies and the heterogeneity of the intervention itself, there is no certainty that the quality improvement collaborative was responsible for an effect. Six studies reported possible differences in baseline measurement. One of the controlled studies was a Breakthrough Series embedded in a seven year quality improvement programme. Four of the studies contained elements of the chronic care model in the intervention. Two of the controlled studies were based on the Vermont Oxford Network. This type of quality improvement collaborative differs from the Breakthrough Series in that it is long term: efforts are led and supported by ongoing data collection of individual member organisations and an ongoing infrastructure of communication and meetings exists that goes beyond a particular limited time frame of a quality improvement collaborative initiative. We were unable to disentangle the different components of an intervention or to assess interactions between longitudinal activities for quality improvement or elements of the chronic care model and collaborative components.
Fifty three (88%) of the 60 uncontrolled reports highlighted specific improvements in patient care and organisational performance that resulted from participating in a quality improvement collaborative. Several reports showed dramatic improvements of 30% to 80%. Almost all of the uncontrolled reports, however, had design limitations, were methodologically weak, and were probably biased in favour of positive findings in successful teams.
The evidence of the impact of quality improvement collaboratives is positive but limited. The apparent inconsistency between the widespread belief in and use of quality improvement collaboratives and the available evidence heightens the importance of a deeper understanding of the relative strength of this intervention. Quality improvement collaboratives are, by their nature, complex and applied in many different ways. Considering that quality improvement collaboratives seem to play a key part in current strategies focused on accelerating improvement, represent substantial investments of time and funding, but may have only modest effects on outcomes at best, then further knowledge of the effectiveness of the basic components, cost effectiveness, variability within collaboratives, and success factors is crucial for determining their value. What mechanisms are responsible for the results and their variations: for example, does effectiveness depend on the topic chosen and are there specific components, supportive contextual factors, or site characteristics that enhance the effectiveness of quality improvement collaboratives? It is possible that a quality improvement collaborative works for some organisations but not for others because of inherent differences in the history and culture of organisations. The data collected in the included studies did not provide the information needed to understand and explain the findings. To understand how and why quality improvement collaboratives work it is necessary to look into the "black box" of the intervention and to study the determinants of success or failure. A detailed formative evaluation of the projects might provide additional insight into these problems. The studies needed balance between uncontrolled process oriented reports and rigorously controlled designs,10 11 12 and a sound integration of process and effect data is needed to understand in more detail why some quality improvement collaboratives and some organisations participating in such a collaborative are successful while others fail to change practice. We look forward to studies adding to this body of knowledge.13 14 15
Limitations should be considered in interpreting the results of this review. Firstly, as in any systematic review we may have missed relevant studies. We searched multiple databases, however, and checked our search with free text words with a strategy that included MeSH terms (box 1) based on key words in the relevant studies. These searches did not add new studies. Secondly, our search was limited to quality improvement collaboratives involving the five essential features described in our inclusion criteria and to English language journals. This might have introduced bias if the effectiveness described in these studies differed systematically from those involving other features and appearing in other languages. Thirdly, the key components of some quality improvement collaboratives could have been misclassified, although our abstraction process showed good inter-rater reliability.
Despite these limitations, this review shows that the evidence underlying quality improvement collaboratives is positive but still limited and that the effects cannot be predicted with great certainty.
|
Contributors: LMTS and MEJLH analysed the data. JJEvE, RH, and RPTMG interpreted the data. All authors conceived and designed the study, drafted and revised the manuscript, and approved the final version. LMTS is guarantor.
Competing interests: LMTS and JJEvE work at the Dutch Institute for Quality Improvement. The views and opinions expressed are those of the authors and do not necessarily reflect those of the institute.
Ethical approval: Not required.
Provenance and peer review: Not commissioned; externally peer reviewed.
![]()
CiteULike
Complore
Connotea
Del.icio.us
Digg
Reddit
StumbleUpon
Technorati What's this?