Jump to: Page Content, Site Navigation, Site Search,
You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.
BMJ 2004;328:388 (14 February), doi:10.1136/bmj.38009.706319.47 (published 6 February 2004)
Peter A Margolis, professor of pediatrics and epidemiology1, Carole M Lannon, associate professor of pediatrics and internal medicine1, Jayne M Stuart, assistant professor of pediatrics1, Bruce J Fried, associate professor of health policy and administration2, Lynette Keyes-Elstein, assistant director of biostatistics3, Donald E Moore, Jr, director, division of continuing medical education4
1 University of North Carolina at Chapel Hill, North Carolina Center for Children's Healthcare Improvement, 730 Airport Rd, Ste 104, CB#7226, Chapel Hill, NC 27599, USA, 2 University of North Carolina at Chapel Hill, School of Public Health, Department of Health Policy and Administration, Chapel Hill, 3 Rho Inc, Chapel Hill, NC 27514, USA, 4 Vanderbilt University School of Medicine, Nashville, TN 37232, USA
Correspondence to: P A Margolis Peter_Margolis{at}med.unc.edu
Design Randomised trial in primary care practices.
Setting Private paediatric and family practices in two areas of North Carolina.
Participants Random sample of 44 practices allocated to intervention and control groups.
Intervention Practice based continuing medical education in which project staff coached practice staff in reviewing performance and identifying, testing, and implementing new care processes (such as chart screening) to improve delivery of preventive care.
Main outcome measure Change over time in the proportion of children aged 24-30 months who received age appropriate care for four preventive services (immunisations, and screening for tuberculosis, anaemia, and lead).
Results The proportion of children per practice with age appropriate delivery of all four preventive services changed, after a one year period of implementation, from 7% to 34% in intervention practices and from 9% to 10% in control practices. After adjustment for baseline differences in the groups, the change in the prevalence of all four services between the beginning and the end of the study was 4.6-fold greater (95% confidence interval 1.6 to 13.2) in intervention practices. Thirty months after baseline, the proportion of children who were up to date with preventive services was higher in intervention than in control practices; results for screening for tuberculosis (54% v 32%), lead (68% v 30%), and anaemia (79% v 71%) were statistically significant (P < 0.05).
Conclusion Continuing education combined with process improvement methods is effective in increasing rates of delivery of preventive care to children.
We randomly selected practices from those meeting the eligibility criteria (see bmj.com) and stratified by factors potentially predictive of the success of the intervention: type of practice (paediatric/family practice), number of newborns enrolled each month, annual Medicaid billing. Within each stratum, we randomised practices to either an intervention group, in which practices received assistance to establish office systems for prevention, or a control group.
Intervention
We have described an intervention that used practice based CME and process improvement methods to support the implementation of "office systems" for delivery of preventive care.3 The intervention was based on the plan-do-study-act (PDSA) cycle of process improvement.4
In the first step, practices formed an improvement team of clerical, nursing, and physician staff members and discussed the results of chart abstractions.
In the second step, project staff provided education about preventive care and effective delivery strategies for preventives services (for example, developing a preventive services summary, establishing a tracking or recall system5). Practices selected performance improvement goals and identified strategies that might improve care.6 We provided an organised set of tools (for example, preventive services flow sheets) to test the changes that were made, and project staff helped practices to customise these tools.
During the third step, project staff helped practices use repeated PDSA cycles in small samples of patients to understand how to adapt new approaches to current office routines. In the fourth step, changes that had the desired effect on the process of preventive care after testing were spread throughout the practice by training staff in new roles.
Two teams consisting of a trained nurse and doctor helped to carry out the intervention. These teams met with practices monthly over a year, using a defined curriculum. During the subsequent year, we checked in on each practice by telephone every two to three months to discuss problems with the logistical aspects of implementation, and to offer advice and support to overcome them.
Outcome measures
Before the study, we selected four preventive services (immunisations, and screening for anaemia, lead, and tuberculosis) that are generally recommended by authoritative organisations for children in the first two years of life The services we chose tend to be well documented in patients' records. Tuberculosis screening consisted of testing for intradermal purified protein derivative or giving the Mantoux test, or risk assessment by 24 months of age; anaemia screening consisted of a complete cell count, packed cell volume, or haemoglobin concentration, or a risk assessment by 18 months of age; screening for environmental lead exposure consisted of a blood test or risk assessment by 24 months of age. A complete immunisation schedule comprised four injections of diphtheria-pertussistetanus vaccine; three oral polio vaccines; one measles, mumps, and rubella immunisation; three Haeomphylus influenzae type B vaccines; and three hepatitis B vaccines by 24 months of age.
The primary outcome measure was the change over time in the proportion of children in each practice who received all four of these services.
Data collection
We abstracted data from medical records, using repeated, random samples of 30 charts of children between 24 and 30 months of age in each practice, and from surveys of doctors and office staff administered at baseline and at the end of the study period.3 Patients who had been seen at least three times and for whom there was no evidence of having transferred out of the practice were eligible. Abstractors were not informed of the study arm to which each practice was assigned.
We used the data from medical records to give feedback to intervention practices about their performance, including comparison with all other practices, every six months. Control group practices received feedback at baseline and annually for two years without comparison with other practices. Follow up of intervention practices was planned for 15-18 months after the 12 month intervention.
Statistical analysis
All our analyses were intention to treat. To compare the magnitude and pattern of change over time in delivery of preventive service between intervention and control practices, we fitted a logistic random regression model.7
Feedback of results from abstraction from charts at baseline typically took place in a meeting three months after baseline data collection from charts. We selected nine months after the feedback meeting as the earliest point at which changes in performance could be detected. All chart abstractions before this point (12 months after baseline data collection) were designated "implementation" points; all points after this mark were designated follow up. "Implementation" and follow up points in control practices were not considered separately.
We compared the change in the proportion of children per practice with age appropriate preventive services over four time intervals: after the implementation period (at 12 months), and 18, 24, and 30 months after the baseline chart abstraction.
|
|
Implementation of intervention
Practices in the intervention group were followed for an average of 24 months after the beginning of the intervention. During the implementation period, all intervention practices developed improvement teams. Project teams met with practice improvement teams a median 8.5 times (range 5-14 times). Of the 22 intervention practices, 18 (82%) implemented preventive services summaries in patient charts, 17 (77%) used tools to support risk assessments, 15 (68%) used prompting by clinicians, and 7 (32%) instituted new health maintenance records for well child visits.
Effectiveness of intervention
Figure 2 shows the pattern of change over time in the proportion of children per practice with all four preventive services in intervention and control groups. During the implementation period the slopes of the lines did not differ significantly between the intervention and control groups (P = 0.11). During the follow up period, the proportion of children with all services in control practices remained relatively constant, changing from 0.09 at the start of the follow up period to only 0.10 after 30 months of follow up. In contrast, the proportion of children with all services in intervention practices increased from 0.07 to 0.34 over the same time period. The change in the prevalence of all four services between the beginning and the end of the study was 4.6-fold greater (95% confidence interval 1.6 to 13.2) in intervention practices than in control practices (table 2).
|
|
The intervention had the greatest impact on lead, tuberculosis, and anaemia screening (fig 3). At the end of the follow up period, the proportion of children per practice who had a record of receiving each of the four individual preventive services was higher in intervention than in control practices; differences for tuberculosis (54% v 32%), lead (68% v 30%), and anaemia (79% v 71%) screening were significant (P < 0.05). For immunisation rates, the improvement over 30 months was about the same in intervention practices and control practices.
|
A growing body of literature suggests that continuing medical education based on the way care is delivered in the practice setting can affect the outcomes of care delivery.8 However, efforts to help practices implement office systems have produced mixed results. In a randomised trial, Dietrich et al found that in primary care practices that implemented office systems, the performance of mammography and clinical breast examinations improved significantly.9 Subsequent attempts to introduce office systems to improve preventive care for adults have been unsuccessful.10 11 Reasons for the limited impact of such interventions include the size and complexity of practices involved, staff turnover, using a suboptimal quality improvement model that placed too much emphasis on planning rather than testing changes, insufficient emphasis on measurement to determine if changes were resulting in improvement, lack of motivation to change, inadequately developed content materials, inexperienced improvement team leaders, and insufficient time for improvement activities.12 Using a formal practice assessment, a practice-wide meeting, and prevention tools, and giving feedback on performance every six months, Goodwin et al reported a small but statistically significant increase in rates of preventive services.13
We used a somewhat different approach, working side by side with the office team, providing information and coaching to each practice to develop improvement expertise within the practice. This avoided the loss of performance associated with "train the trainer" models14 because it did not depend on novice leaders during what was often their first application of improvement methods. We emphasised frequent, small scale tests to enable practices to "try out" changes without risking disruption of practice routines. The provision of tools and materials allowed practices to concentrate on improving care, and the emphasis on measurement encouraged practices to learn from their data, thereby engendering trust in the process.
Limitations
This study has several limitations. We selected practices that provided care for relatively large numbers of children in order to be able to detect an intervention effect. Small paediatric practices and most family practices were excluded. Although this may limit the generalis-ability of the study to multi-physician settings, such environments tend to be more complex and thus stand to benefit more from quality improvement efforts.
The primary outcome measure was the proportion of children within each practice who received all four age appropriate services. Some clinicians may not have agreed that all procedures were necessary; some children may not have been exposed to practice changes; and some services may have been provided without being documented. The increased use of blood and skin testing, as well as risk factor screening, suggests that improvements did not represent improved documentation alone.
|
Immunisation rates did not improve significantly, but at the time the study was conducted North Carolina implemented a universal vaccine purchase programme. It became the state with the highest rates of immunisations in the United States.15
Conclusion
This study shows that continuing education oriented to improving primary care practices' systems for delivery of care is associated with important improvements in preventive care. Our findings are potentially important for current efforts to incorporate performance improvement and systems thinking as a core competency for physicians and other health professionals. The next step will be to reduce the costs of assistance and to disseminate new approaches to more practices more rapidly. Future studies should also explore how to further increase the reliability of care, magnify the rate of improvement, and sustain improvements.
The trial was a collaboration involving the University of North Carolina at Chapel Hill (UNC); the North Carolina Division of Medical Assistance; North Carolina Office of Research, Demonstrations, and Rural Health; Carolinas Medical Center; and the NC Area Health Education Centers. We wish to recognise the leadership of the North Carolina Office of Rural Health, Research and Demonstrations (Jeffrey Simms, Burnie Patterson), the North Carolina Area Health Education Centers (Harry Gallis, Thomas Bacon), and the North Carolina Division of Medical Assistance (Dennis Williams) without whose vision and persistence this study would not have been possible. We are indebted to the practice assistance teams at the Charlotte Area Health Education Center (Laura Noonan, Patricia Nesbit, Dawn Carpenter) and the North Carolina Office of Rural Health, Research and Demonstrations (Carol Powell, Deborah Shah), who provided assistance to the participating practices. We also thank the staff of the North Carolina Center for Children's Healthcare Improvement at the University of North Carolina at Chapel Hill (Sarah McGovern, Jennifer Pearce, Jenny Bowes) who participated in curriculum development, training, and data collection and Laura Peterson, who helped with preparation of the manuscript. Most importantly, we wish to recognise the dedication of the primary care practices without whose participation this study would not have been possible.
This is the abridged version of an article that was posted on bmj.com on 6 February 2004: http://bmj.com/cgi/doi/10.1136/bmj.38009.706319.47 Funding: US Agency for Healthcare Research and Quality (RO1-HS08509), US Bureau of Maternal and Child Health (RO1-HS08509), the North Carolina Division of Medical Assistance, the North Carolina Area Health Education Centers, and the Robert Wood Johnson Foundation Generalist Faculty Scholars Program.
Competing interests: None declared.
Ethical approval: University of North Carolina at Chapel Hill ethics committee.
Read all Rapid Responses