Learning In Practice

Effect of a community oriented problem based learning curriculum on quality of primary care delivered by graduates: historical cohort comparison study

BMJ 2005; 331 doi: https://doi.org/10.1136/bmj.38636.582546.7C (Published 27 October 2005) Cite this as: BMJ 2005;331:1002
  1. Robyn Tamblyn, professor (robyn.tamblyn{at}mcgill.ca)1,
  2. Michal Abrahamowicz, professor2,
  3. Dale Dauphinee, executive director3,
  4. Nadyne Girard, project manager1,
  5. Gillian Bartlett, assistant professor4,
  6. Paul Grand'Maison, assistant dean premedical studies5,
  7. Carlos Brailovsky, professor6
  1. 1 McGill University, Montreal, QC, Canada H3A 1A3
  2. 2McGill University Health Centre-Montreal General Hospital, L10.520, Montreal, QC, Canada H3G 1A4
  3. 3 Medical Council of Canada, Ottawa, ON, Canada K1G 3H7
  4. 4 Clinical and Health Informatics Research Group, McGill University
  5. 5 Faculty of Medicine, University of Sherbrooke, Sherbrooke, QC, Canada J1K 2R0
  6. 6 Centre d'évaluation des sciences de la santé, Université Laval, QC, Canada G1K 7P4
  1. Correspondence to: R Tamblyn
  • Accepted 26 September 2005

Abstract

Objective To assess whether the transition from a traditional curriculum to a community oriented problem based learning curriculum at Sherbrooke University is associated with the expected improvements in preventive care and continuity of care without a decline in diagnosis and management of disease.

Design Historical cohort comparison study.

Setting Sherbrooke University and three traditional medical schools in Quebec, Canada.

Participants 751 doctors from four graduation cohorts (1988-91); three before the transition to community based problem based learning (n = 600) and one after the transition (n = 151).

Outcome measures Annual performance in preventive care (mammography screening rate), continuity of care, diagnosis (difference in prescribing rates for specific diseases and relief of symptoms), and management (prescribing rate for contraindicated drugs) assessed using provincial health databases for the first 4-7 years of practice.

Results After transition to a community oriented problem based learning curriculum, graduates of Sherbrooke University showed a statistically significant improvement in mammography screening rates (55 more women screened per 1000, 95% confidence interval 10.6 to 99.3) and continuity of care (3.3% more visits coordinated by the doctor, 0.9% to 5.8%) compared with graduates of a traditional medical curriculum. Indicators of diagnostic and management performance did not show the hypothesised decline. Sherbrooke graduates showed a significant fourfold increase in disease specific prescribing rates compared with prescribing for symptom relief after the transition.

Conclusion Transition to a community oriented problem based learning curriculum was associated with significant improvements in preventive care and continuity of care and an improvement in indicators of diagnostic performance.

Introduction

Periodic reforms are made in medical education on the assumption that basic training affects how doctors will practice.1 Although considerable investment is made in such reforms, systematic assessment of expected benefits in actual practice is lacking, particularly for quality and outcome of care.2 3w1 w2 The lack of progress in investigating the effects on practice outcomes of the hotly debated reform to community oriented problem based learning.4 58w2-w9 has been the subject of recent editorials and reviews.2 3w1 w2 In problem based learning, fundamental changes are made in the way in which medical knowledge and skills are acquired (through problem based education rather than by discipline specific education), the method of learning professional development skills (self directed, small group), and the environment within which experience of practice is acquired (greater emphasis on community based practice compared with hospital based practice). Community oriented problem based learning was expected to better prepare graduates to provide comprehensive preventive, acute, and chronic disease care through community oriented patient centred learning. Yet concerns have been expressed that such a curriculum might reduce core competencies in medical diagnosis and treatment through gaps in relevant medical knowledge and diluted experience of acute care.4 8 9w5 These concerns have been buttressed by evidence of lower licensing examination scores and suboptimal diagnostic reasoning in some studies of graduates of a problem based learning curriculum, but the effects have not been assessed in practice.9w5 w6 w10 w11

In 1987 the University of Sherbrooke in Quebec, Canada, was one of the first medical schools to undergo a reform from a traditional curriculum (system based or disciplinary based) to a problem based learning curriculum.10 We tested the hypothesis that graduates of a problem based learning curriculum would provide better continuity of care and preventive care in primary care than graduates of a traditional curriculum. We also assessed whether the Sherbrooke graduates exhibited a decline in performance in selected indicators of diagnosis and disease management.

Methods

To assess the effect of transition to a community oriented problem based learning curriculum on practice performance, we carried out a longitudinal follow-up study of four classes of Sherbrooke graduates; three before the transition (graduates of 1988-90) and one after the transition (graduates of 1991). We compared the differences in the performance of Sherbrooke graduates before and after the transition with a control group of graduate doctors trained in the same period in medical schools with a traditional curriculum. Eligible doctors were those who graduated from one of the four medical schools in Quebec (Sherbrooke, Laval, Montreal, and McGill), completed postgraduate training in family medicine, and were licensed to practise as a family doctor in Quebec between 1990 and 1993 (see bmj.com for details of examination). The follow-up period for each graduation cohort was between 1 July of the certification examination year and 31 December 1996, providing an assessment of practice for the first four to seven years. Eligible doctors were identified by the Quebec College of Physicians. We excluded salaried doctors because their patients could not be identified and doctors who subsequently trained in another specialty. The provincial insurance agency la Régie de l'assurance maladie du Québec provided the data on all patients seen by each doctor, for each year of practice.

The Sherbrooke reform

In 1987 Sherbrooke University implemented a community oriented problem based learning curriculum to enhance the ability of medical graduates to meet the needs of the population.10 11 Educational reforms were aimed at fostering humanistic behaviour, self directed learning abilities, a comprehensive understanding of the biological, psychological, and social aspects of health problems, and skills in preventive, curative, and rehabilitative care. Didactic lectures were replaced by biweekly problem oriented tutorials in groups of 5-8 students, aimed at integrating knowledge on basic, clinical, bio-psychosocial, and preventive care through the analysis of problems. The students also undertook a 12 week preclerkship in clinical reasoning skills. Evaluation was modified to include assessment by the tutor, an objective structured clinical examination, and open ended exercises in problem analysis, reducing the weight of summative examinations to 25% of the academic grade. Community based clinical experience was increased by five weeks in the preclinical years and by 11 weeks in the 74 weeks of the clerkship.

Medical school training and certification

The College of Physicians provided data on medical school attended, year of graduation, and sex. The doctors had graduated from the four medical schools in Quebec. We determined the location of postgraduate training by linking with the Canadian post-MD education registry, a longitudinal database on postgraduate training that is updated annually by Canadian faculties of medicine.w12 We used the family medicine certification examination score12 to adjust for differences in academic ability between graduates of the different schools. To protect confidentiality, an encrypted doctor identifier linked data on doctors to patient information.

Practice activity

Data sources

We assessed practice using four, previously validated, health administrative databases.13 14w13 w14 The registrant database provided details on the patients' age, sex, post code, and date of death. The medical services database provided data on all services delivered on a fee for service basis (type, location, diagnosis, treating and referring doctor). The prescription database provided details on drugs, doses, duration, prescribing doctor, and date of each prescription dispensed for patients aged 65 or older. The hospitalisation database provided discharge diagnoses and admission and discharge dates for all hospital admissions. We approximated education and income from the 1991 census value of mean family income and educational achievement for all residents within the same post code area (average of 366 households).w15 From the medical services claims files we identified all patients seen by each doctor from the licensure date to 31 December 1996. La Régie de l'assurance maladie du Québec provided personal data for each patient as well as data on all medical services, prescriptions, and admissions to hospital during the follow-up period. Similar information was retrieved for the year before the first contact with the study doctor to provide an assessment of patient characteristics that were not influenced by the doctor.15

Practice environment and performance indicators

For descriptive purposes we used the medical services claims files to obtain the number and type of practice settings, geographical distribution of the practice population (rural-remote, intermediate, urban; see bmj.com for regions), and workload (practice size, days worked, daily patient volume) for study doctors for each follow-up year.

To assess the hypothesised benefits of the transition to problem based learning in providing more comprehensive care, we selected two performance indicators with an established link to health outcome.15w16 Mammography screening rate was used to assess preventive care, as doctors who screen for breast cancer are also more likely to carry out other preventive services.16 Continuity of care was chosen because of its importance in prevention and chronic disease management.17w17 We selected two additional performance indicators to test hypothesised negative effects of the transition on the quality of diagnosis and treatment. We selected the difference between disease specific prescribing rates and prescribing rates for symptom relief because higher rates of disease specific prescribing compared with symptom relief prescribing18 have been linked to diagnostic competence15w16 and because excess use of drugs for symptom relief is associated with potentially avoidable injuries (benzodiazepines)w18 w19 and gastrointestinal bleeding (non-steroidal anti-inflammatory drugs).w20 w21 We selected the prescribing rate for contraindicated drugs because prescribing errors account for 20% of drug related adverse events and these problems are partly related to deficiencies in doctors'knowledge.19w22

Practice populations

We assessed mammography screening and continuity of care only in the primary care practice population; defined as all patients seen in their doctor's surgery or in a clinic in the respective year who had had a major assessment by the study doctor. We assessed prescribing rates in ambulatory patients aged 65 years or more (older patients). Ambulatory patients were those for whom the study doctor had billed for at least one visit in an outpatient clinic, surgery, or emergency room setting. We identified each practice population for each doctor in each calendar year. We used only patients meeting eligibility criteria in a respective calendar year to assess annual performance indicators.

Mammography screening rate

Annual mammography screening rate was defined as the proportion of eligible women in the primary care practice who were referred for a bilateral mammogram by the doctor.15w16 Eligible women were aged between 50 and 69, had no previous diagnosis of breast disease or diagnostic mammogram, were due for screening (24 months or more had elapsed since their last mammogram), and were not receiving primary care from a gynaecologist or obstetrician.

Continuity of care

We defined annual continuity of care as the proportion of all visits that were made to the study doctor or generated by his or her referral among all visits made by his or her patients during the year.15 We first calculated for each patient the proportion of all visits in the year that were made either to the study doctor or to other doctors on the basis of referral by the study doctor. We then calculated the doctor level continuity of care as the weighted mean of the proportions for all of the study doctor's patients, with individual patient's proportions assigned weights equal to the square root of the total number of all visits made by the patient during the year.

Prescribing rates

Disease specific and symptom relief prescribing rates—We used the drugs prescribed by the study doctor to all older patients to determine the annual difference between disease specific and symptom relief prescribing rates. Disease specific drugs were those that would rarely be prescribed without a disease being confirmed by investigation (for example, anticoagulants) and symptom relief drugs were those that relieve symptoms but have little effect on the disease process (for example, benzodiazepines), using the McGavock classification.18

Contraindicated prescribing rate—We defined the annual contraindicated prescribing rate as the proportion of ambulatory older patients prescribed a relatively contraindicated drug by the study doctor. Relatively contraindicated drugs were defined by an updated expert review as 30 drugs that should be avoided in older people because of possible toxic effects, and the availability of safer alternatives.20w23

Practice covariates

To adjust for differences in the case mix of the doctors' practice populations, we retrieved the characteristics for each eligible patient in the 12 months before the first contact with the study doctor in each year. We then aggregated patient specific values for a given year to characterise each doctor's practice population by age and sex distribution, mean family income, educational achievement, geographical access to health care,15 propensity for the use of healthcare services (ambulatory care groups) groupings, number of doctors seen),w24 comorbidity (Charlson index),w25 and rate of admissions to hospital. We modelled annual measures of practice population characteristics as time dependent covariates. For relatively contraindicated prescribing, we did not adjust for differences in practice population because the prescription of relatively contraindicated drugs is rarely justified by patient characteristics.w16 To adjust for potential differences in practice experience, we used the cumulative number of preceding months of active experience in practice as a continuous time dependent covariate in the analysis.15

Analysis

To test hypothesised changes in performance related to the transition from a traditional to a problem based learning curriculum, we used repeated measures generalised estimated equations extensionw26 of multiple linear regression analysis. Annual values of each outcome represented the repeated measures for each doctor. We assumed an autoregressive order 1 structure of the covariance matrix of residuals to account for the correlation between annual measures for the same doctor. To minimise the effect of imprecise observations, we excluded from the analysis those doctors with fewer than five eligible patients, and we weighted the data for other doctors by the logarithm of their practice size.15w16 We adjusted each model for doctors' sex, certification examination score, cumulative months of experience in practice, and annual measures of case mix.

For each outcome we estimated three multivariate generalised estimated equations models. In the first two models we estimated the adjusted differences between medical schools for graduates trained before the transition (graduates of 1988-90) and those trained after the transition (graduates of 1991). Sherbrooke was the reference category, and three dummy variables represented graduates from the other three medical schools. The third model tested separately for graduates from each of the four schools whether there was a significant within school difference in performance of graduates in the period before the transition compared with performance after the transition. To estimate the differences between before and after the transition, we analysed the data from all schools and cohorts in the same model. The model included a series of binary variables, each assigning the value of 1 to the 1991 graduation cohort (after transition) from a given school. In additional analyses we assessed if the change after transition in a given outcome at a particular school was different from the change at Sherbrooke after the transition by testing a series of transition by school interactions in a multivariable generalised estimated equations model that included the main effects of schools (before transition) and the effect of the transition among Sherbrooke graduates. In sensitivity analysis we assessed whether undergraduate and postgraduate training in the same university confounded associations. We used SAS procedure Genmod, version 8.0 in all generalised estimated equations analyses.

Results

Between 1988 and 1991, 2013 doctors completed undergraduate training in Quebec medical schools and 930 (46.2%) of these doctors entered postgraduate training in family medicine. Among these doctors, 813 (87.4%) were licensed to practise in Quebec (table 1). Doctors entering primary care practice were predominantly women, particularly in the period after the transition to a community oriented problem based learning curriculum, except for McGill graduates. Most family doctors completed undergraduate and postgraduate training in the same medical school. The mean age at the completion of postgraduate training was between 26 and 28 years. Family medicine certification examination scores for doctors from the four schools varied by slightly more than one quarter of a standard deviation. Although the mean score for Montreal graduates dropped by more than one quarter of a standard deviation in the period after the transition, no differences in scores between the periods before and after transition achieved statistical significance (table 1). Most doctors entered fee for service practice, the lowest proportion being from Laval.

Table 1

Characteristics of 813 study doctors who completed Quebec medical school between 1988 and 1991 and entered primary care in Quebec between 1990 and 1996 by medical school of undergraduate training. Values are numbers (percentages) unless stated otherwise

View this table:

Most of the doctors worked in two or more practice settings annually (table 2). McGill graduates had a distinctive profile, as they saw a higher proportion of patients in intensive care and inpatient settings, and most of their patients were from urban regions. In contrast, graduates from Laval and Sherbrooke saw the greatest proportion of patients from rural and remote areas. Montreal graduates saw the greatest number of patients per year and had the highest mean volume of patients per day.

Table 2

Annual practice setting and workload characteristics of 751 Canadian family doctors in fee for service practice in first 4-7 years of practice by undergraduate medical school (1990-6)

View this table:

Before the transition, Sherbrooke graduates had the highest mammography screening rate (mean 158 per 1000 v 94.6 to 114.0 per 1000 for the other schools), significantly higher than Laval graduates (table 3). After the transition, Sherbrooke graduates showed a significant and substantial increase in mammography screening rate (table 3). Increases in screening rates for the three other schools in the period after the transition were substantially smaller than for Sherbrooke and statistically non-significant.

Table 3

Association between medical school attended and quality of preventive care (mammography screening of women aged 50-69 per 1000 in primary care population) for doctors graduating before and after transition to a problem based learning curriculum in first 4-7 years of primary care practice (1990-6)

View this table:

Continuity of care was similar among graduates from the different medical schools in the period before the transition, with the exception of significantly lower continuity of care for McGill graduates (table 4). After the transition, Sherbrooke graduates showed a statistically significant improvement in continuity of care (table 4). Although McGill graduates also showed substantive improvement in the period after the transition, the change in continuity of care was statistically non-significant, possibly due to fewer graduates.

Table 4

Association between medical school attended and continuity of care for doctors graduating before and after transition to a problem based learning curriculum in first 4-7 years of primary care practice (1990-6)

View this table:

In the period before the transition, McGill graduates prescribed disease specific drugs to 10.3 more patients per 1000 than symptom relief drugs, whereas Sherbrooke graduates prescribed fewer disease specific drugs than symptom relief drugs (table 5). After the transition, however, Sherbrooke showed a fourfold greater improvement in the difference between the two rates than did McGill (32.2 v 8.0, table 5). This allowed Sherbrooke graduates to surpass the differences in rates for McGill graduates in the period after the transition (16.1 v11.8, table 5). The increase for Montreal graduates was also statistically significant although smaller than for Sherbrooke graduates, even if the difference in the magnitude of the changes before transition between the schools was statistically non-significant.

Table 5

Association between medical school attended and quality of disease management (proportion of patients aged 65 years seen in ambulatory setting and prescribed drugs by study doctor) for doctors graduating before and after transition to problem based learning curriculum in first 4-7 years of primary care practice (1990-6)

View this table:

Sherbrooke graduates had significantly higher rates of contraindicated prescribing in the period after the transition than did graduates from some of the other schools (table 5). Contraindicated prescribing rates declined in the period after the transition, with the largest declines in Laval (11.8 per 1000). We found no evidence of any increase at Sherbrooke, where a non-significant decrease of 5.2 per 1000 was observed.

Results of additional analyses that tested interactions between the effect of the individual schools and cohorts after the transition were generally consistent with the results shown in tables 3, 4, and 5. The point estimates of the interaction effect suggested that changes in most outcomes in the other schools were two to three times smaller than in Sherbrooke. Although many interactions were statistically significant in conventional regression analysis, they became marginally statistically nonsignificant in the more conservative generalised estimated equations analysis that accounted for correlation of repeated outcomes of the same doctor. The magnitude of changes in practice outcomes in the period before and after the transition was not materially affected by the location of postgraduate training (same versus different medical school).

Discussion

Transition to a community oriented problem based learning curriculum at Sherbrooke University, Quebec, was associated with substantial and statistically significant improvements in preventive care and continuity of care, without the hypothesised decline in performance indicators associated with management and diagnosis. Indeed, the transition was associated with a significant improvement in the difference between disease specific and symptom relief prescribing rates which has been shown to be a marker of diagnostic ability.15w16 Although significant improvements were seen in three of four indicators, we were not able to show that these changes were significantly larger than simultaneous changes in other schools, owing to a small number of graduates after the transition and low power. Future studies would need to provide several cohorts of graduates in the post-transitional period to provide conclusive evidence of such differences.

We were able to compare the quality of care delivered by primary care doctors who graduated from community oriented problem based and traditional curriculums. By assembling cohorts of graduates who entered primary care both before and after the transition to a problem based learning curriculum at Sherbrooke University, we were able to control for differences between graduates of different medical schools before the transition as well as for temporal trends in the quality of care. The use of administrative databases avoided participation and self report biases, as outcomes were measured among all graduates who entered primary care, and among all patients seen by these graduates in the first 4-7 years of practice.

Although requirements for admission were not altered as part of the reform,10 the particular attributes of the reform that may have generated the positive changes in practice performance can not be readily disentangled for Sherbrooke, or for most other schools that have implemented innovations to their curriculum.w4 That significant differences existed in outcomes of graduates of different medical schools in the period before the transition, even after adjustment for certification examination scores, suggests that school level factors beyond the formal curriculum influence subsequent practice activity. The greater emphasis on community based practice experience in the problem based learning curriculum may have contributed to improvements in continuity and preventive care of Sherbrooke graduates. To inform future policy for medical education, the effect of the clinical training environment on subsequent medical practice needs to be systematically investigated.21

In our study the performance of new graduates fell short of desirable outcomes. In particular, mammography screening was below the 70% coverage expected to reduce mortality from breast cancer,22 a universal problem that has led to the introduction of screening centres for self referral for mammography.w27 w28 The prevalence of contraindicated prescribing for older patients is considered a systemic problem that contributes to avoidable morbidity.23 The variability in contraindicated rates between graduates from different medical schools suggests that basic training may be an additional factor that needs to be examined to improve future prescribing behaviour.

Our study has limitations. Using administrative databases we could assess only a subset of outcomes that would be of importance in judging the quality of primary care. We excluded salaried doctors from the evaluation. Although the proportion of salaried doctors varied by medical school, investigation has shown that salaried practice is predominantly influenced by the doctor's sex.w29 Our evaluation of the effect of a community oriented problem based learning curriculum was limited to one school's experience and the first graduation class of that curriculum. To enhance the generalisability of our findings, similar evaluation should be carried out in other schools with a problem based learning curriculum. Future research should assess whether transition to a problem based learning curriculum is associated with other expected changes in the quality of care, particularly in populations of patients where important gaps exist between optimal and current therapy, such as those with asthma or diabetes and those needing management for cardiovascular risk.24w30

What is already known on this topic

Community oriented problem based learning has been implemented in many medical schools to improve the ability of graduates to deliver more comprehensive medical care

Concerns are that problem based learning may lead to gaps in knowledge and reduced core competencies in diagnosis and treatment

The impact of problem based learning on subsequent medical practice has not been assessed

What this study adds

Transition to community based problem based learning in one medical school was associated with improvement in comprehensive care without a decline in diagnosis and management

Doctors graduating from traditional medical schools within the same period did not exhibit the same improvement in performance

Footnotes

  • Embedded Image Web references w1-w30 and additional information are on bmj.com

  • Contributors RT, MA, and DD were responsible for the conception and design of the study. PG, CB, RT, and NG were responsible for the assembly of the study population and database linkages. MA was responsible for the statistical approach. GB and NG were responsible for the analysis. All authors contributed to the interpretation of the results and the preparation of the manuscript. RT and MA are the guarantors.

  • Funding Canadian Institutes of Health Research and Fonds de Recherche en Santé du Québec.

  • Competing interests None declared.

  • Ethical approval This study was approved by McGill University institutional review board and Quebec Access to Information.

References

View Abstract