Intended for healthcare professionals

Information In Practice

Effect of electronic health records in ambulatory care: retrospective, serial, cross sectional study

BMJ 2005; 330 doi: (Published 10 March 2005) Cite this as: BMJ 2005;330:581
  1. Terhilda Garrido, senior director (terhilda.garrido{at},
  2. Laura Jamieson, senior business consultant1,
  3. Yvonne Zhou, project manager1,
  4. Andrew Wiesenthal, associate executive director, Permanente Federation1,
  5. Louise Liang, senior vice president for quality and clinical systems support, Kaiser Foundation Health Plan1
  1. 1 Clinical Systems Planning and Consulting, Kaiser Permanente, 2101 Webster Street, Oakland, CA 94612, USA
  1. Correspondence to: T Garrido
  • Accepted 31 January 2005


Objective To evaluate the effect of implementing comprehensive, integrated electronic health record systems on use and quality of ambulatory care

Design Retrospective, serial, cross sectional study.

Setting Colorado and Northwest regions of Kaiser Permanente, a US integrated healthcare delivery system.

Population 367 795 members in the Colorado region and 449 728 members in the Northwest region.

Intervention Implementation of electronic health record systems.

Main outcome measures Total number of office visits and use of primary care, specialty care, clinical laboratory, radiology services, and telephone contact. Health Plan Employer Data and Information Set to assess quality.

Results Two years after electronic health records were fully implemented, age adjusted rates of office visits fell by 9% in both regions. Age adjusted primary care visits decreased by 11% in both regions and specialty care visits decreased by 5% in Colorado and 6% in the Northwest. All these decreases were significant (P < 0.0001). The percentage of members making ≥ 3 visits a year decreased by 10% in Colorado and 11% in the Northwest, and the percentage of members with ≤ 2 visits a year increased. In the Northwest, scheduled telephone contact increased from a baseline of 1.26 per member per year to 2.09 after two years. Use of clinical laboratory and radiology services did not change conclusively. Intermediate measures of quality of health care remained unchanged or improved slightly.

Conclusions Readily available, comprehensive, integrated clinical information reduced use of ambulatory care while maintaining quality and allowed doctors to replace some office visits with telephone contacts. Shifting patterns of use suggest reduced numbers of ambulatory care visits that are inappropriate or marginally productive.


The quality of health care in the United States warrants concern.1 The problems have been categorised as underuse, overuse, or misuse of healthcare services,2 and uncertainty in clinical decision making about individual patients plays a part in inappropriate use.3

Uncertainty in clinical decision making can arise from unavailable or poor quality data on the patient. Medical records are still predominantly paper based despite well documented shortcomings in terms of accuracy, completeness, availability, and legibility.48 Indeed, although widescale development of electronic health record systems has been repeatedly recommended in the United States,4 7 9only about 5% of US primary care providers use an electronic health record.10 Incomplete, illegible, or unavailable patient information may necessitate conservative management strategies and result in redundant or marginally productive visits, diagnostic and screening tests, and interventions. Preventive care and patient education may also be overlooked if consultations have to focus on rebuilding clinical data.

Electronic health records reduce uncertainty by providing greater accessibility, accuracy, and completeness of clinical information than their paper counterparts.9 Two Kaiser Permanente regions separately implemented comprehensive electronic health record systems. We examined their effect on selected measures of use and quality of ambulatory care.


Kaiser Permanente is the largest, not for profit, integrated healthcare delivery system in the United States. It comprises eight operating regions and serves more than 8.2 million members in nine states and the District of Columbia.11 Doctors join regional exclusive partnerships or professional corporations that contract with the Kaiser Foundation Health Plan and assume full responsibility for providing and arranging necessary medical care for members. Kaiser Permanente's integrated system provides all healthcare needs for adults and children, including preventive, routine, specialty, emergency, and inpatient care, ancillary testing, pharmacy and rehabilitative services, and home care. Two Kaiser Permanente regions, Colorado and the Northwest, implemented electronic health record systems (table 1).

Table 1

Characteristics of Colorado and Northwest healthcare regions

View this table:

Electronic health record systems

Before implementation, individual paper based medical records were manually delivered to multiple sites throughout the integrated Kaiser Permanente system. These sites were often several miles distant from each other and from the medical records warehouse; availability of records for same day and unscheduled care was unreliable.

Paper medical records were essentially eliminated after implementation. Patients' electronic charts became available for unscheduled phone contacts, same day ambulatory care, and emergency department visits at Kaiser Permanente facilities. Although the regions implemented different systems, an internally developed clinical information system in Colorado and a externally supplied system (EpicCare) in the Northwest region, they shared important characteristics:

  • Integrated documentation and reporting of clinical results reporting, including comprehensive recording of use of primary and specialty care, telephone contact, urgent care, and emergency department

  • Computerised entry of physician orders for tests and prescriptions

  • 24 hour availability of medical records at the point of care

  • Immediate availability for all potential users—for example, staff in telephone advice centres, pharmacists, and staff reporting clinical results

  • Easy searching.

Decision support functions varied between systems in terms of scope and function.


To evaluate the effect of electronic health records, we conducted retrospective serial cross sectional studies for each region. We used administrative data to assess usage and the Health Plan Employer Data and Information Set to assess quality of care before and after implementation. This dataset is collated by the National Committee for Quality Assurance, which is recognised as the sole authority on the comparative performance of managed healthcare plans in the United States. The Health Plan Employer and Data Information Set is a series of standardised performance measures covering all population needs and pressing public health problems, such as appropriate treatment for patients with chronic diseases and care delivered to elderly people. Managed healthcare plans participate voluntarily. Results are based on statistically valid random samples of members and are rigorously audited.

The electronic health records were introduced progressively over more than a year and in different calendar years in the two regions. We reviewed data for entire calendar years, delimiting the implementation period by the calendar year in which implementation began (year I1) and the calendar year in which it was completed (year I2). Given natural lags in implementation and impact, the baseline period includes year I1 and the intervention period begins in year I2.

To determine meaningful utilisation measures, we conducted interviews with roughly 100 clinical and operations leaders, managers, frontline clinicians, and staff from medicine, nursing, pharmacy, clinical laboratory and radiology services, risk management, medical records, and other areas after implementation of electronic health records. The interviews suggested the electronic system increased the efficiency of ambulatory care visits and reduced redundant or marginally productive healthcare services by allowing patient issues to be resolved during the first contact, enabling more services to be offered per visit and reducing the need for separate health maintenance visits. Informants also reported reduced use of clinical laboratory, radiology, and emergency services and increased effectiveness of scheduled and unscheduled telephone contact with patients.

To evaluate these impressions, we examined total office visits and separated use into primary and specialty care. We included visits to doctors and other health professionals. We also examined the frequency distribution of primary care visits. Data on total office visits and use of emergency departments, radiology departments, clinical laboratories, and telephone call centres were obtained from regional data warehouses.

Most informants perceived that the electronic health record improved quality of care. To assess quality of care, we extracted data from the Health Plan Employer and Data Information Set. From the 30 measures available in the dataset, we selected three that represented activities in primary and specialty care, care of routine and chronic conditions, and care requiring a cross departmental referral. These were percentages of relevant members receiving advice on stopping smoking, screening for cervical cancer, and retinal examinations in diabetes. These three measures were among the few that remained consistently defined over the eight year study period; the definitions of many measures have evolved over time.

Statistical analysis

We stratified the annual rates of total office visits per region by primary care and specialty care and adjusted to a fixed age distribution over the study period using two age categories (0-64 years and ≥ 65). Adjusting for smaller age increments and sex did not affect the results. We calculated the difference between the age adjusted visit rates in year I1 and the observation period.

Year I1 was selected as the baseline year for several reasons. In keeping with national trends, use of ambulatory care had been rising at Kaiser Permanente before electronic health records were introduced (fig 1). The reasons for the rise include changing reimbursement mechanisms, policies to review usage, and new technologies.12 Year I1 represents maximum capture of this upward trend. It also represents the last point of substantially steady state operational practices in ambulatory care. Although implementation began in year I1, it began slowly in isolated segments of ambulatory care, and it was some time before electronic records affected health care; most of the implementation occurred in year I2.

Fig 1
Fig 1

Rates of office visits adjusted for age before and after introduction of electronic health records in primary care

We selected year 2 as the primary comparison year because it was the last point for which consistent data from both regions were available. Regional measures of use of ambulatory care subsequently changed. In the Northwest region, data for two additional years were available because of the earlier implementation dates.

Because of the large sample size (around 400 000 members and 2 million visits a year in each region), the difference between the two rates approximated the normal distribution; we therefore used a z test to assess the significance of the difference. We also examined use of emergency departments, radiology, laboratory services, and call centres by this method.

Frequency distribution

We examined the frequency of primary care visits in the years before and after implementation. Frequencies were grouped into categories—0 visits, 1 visit, 2 visits, 3-5 visits, 6-10 visits, and ≥ 11 visits per member per year by region. We tested the changes for significance with a χ2 test.


At the time of assessment, the Colorado and Northwest regions had two and four years' experience, respectively, with electronic health records. Changes in definitions of measures in the Colorado region precluded meaningful comparisons over a longer time.

Use of ambulatory care

Both regions had significant decreases in use of services. The age adjusted number of total office visits per member in year 2 decreased by 9% compared with year I1 (P < 0.0001, in both regions), and age adjusted primary care visits decreased by 11%. Age adjusted specialty care visits decreased by 5% in Colorado and 6% in the Northwest (both P < 0.0001). In year 4, the total office visit rate in the Northwest region was 8% lower than before electronic records became available (fig 1). Partial implementation had minimal effect during year I1.

The frequency pattern of ambulatory primary care visits suggested a general decrease in use across all patients, with larger reductions among patients making three or more visits (fig 2). The percentage of members making three or more visits a year decreased by 10% in the Colorado region and 11% in the Northwest region between year I1 and year 2. In year 4, the rate in the Northwest region had decreased by an additional 2%. Moreover, the percentage of members with ≤ 2 visits a year increased. This finding is particularly striking in light of the ageing Colorado membership (a disproportionate number of people aged over 65 were enrolled during the study period) and is consistent with the effects of electronic health records described by clinical and operational leaders.

Fig 2

We reviewed data relating to other factors that could potentially explain decreased use of ambulatory care. Rates of visits to emergency departments (internal and external to Kaiser Permanente) did not rise over the study. To rule out inadvertent reductions in access to services or shifting of care to other providers, we reviewed the ratio of all primary care providers (physicians, nurse practitioners, and physician assistants) to members and the ratio of referrals to outside providers in both regions throughout the study. Both ratios remained stable.

To rule out other global influences, we compared the changes in the Colorado and Northwest regions with trends in Kaiser Permanente regions without electronic health records and with national trends. Inconsistent definitions of office visits precluded direct comparisons between regions. We examined the rate of change in office visits, as independently defined by three other Kaiser Permanente regions, for which trend data were available for the same period. The data did not show comparable decreases. The rate of ambulatory care visits by people aged 45 or older increased by 14% across the United States between 1992 and 2002, which encompasses our study period.13

Telephone contact

The electronic health record enabled more effective telephone contacts. In the Northwest region, telephone encounters scheduled at the discretion of physicians increased from a baseline of 1.26 per member per year to 2.09 after two years. In the Colorado region, staffing of call centres briefly shifted from primarily nursing staff to include doctors with access to electronic health records. Appointments needed by patients after telephone contact decreased by 7% when contact was with a doctor with access to electronic health records. Doctors reported being able to resolve health issues by phone more readily with the electronic health record. Rates of appointments after telephone contact rose when staffing reverted to nurses. Comparable data on telephone contact from other Kaiser Permanente regions were not available.

The effect of telephone contact on use of ambulatory care is shown through a contemporaneous operational evaluation of the telephone treatment of uncomplicated urinary tract infections. In the Colorado region, a previously available nursing protocol was built into an electronic health record template in year I1, increasing ease of access for nursing staff. Between year I1 and year 2, the age adjusted visit rate for urinary tract infection among women fell by 31%. The records of women treated with antibiotics for a urinary tract infection during three months in year I1 were randomly audited. Of 262 women whose records were audited, 73 were prescribed antibiotics by a nurse; 67 of these did not require a return visit within eight weeks of treatment, indicating appropriate resolution.

Radiology and clinical laboratory services

Age adjusted rates of use of radiology services decreased by 14% in the first two years after introduction of electronic health records in the Northwest region. Despite more recent increases in general use of imaging inside and outside Kaiser Permanente, the age adjusted rate remained 4% lower than before implementation. The chief of radiology in the Colorado region believed strongly that availability of electronic records to all carers improved interpretation of films.

Laboratory usage in the Northwest region had decreased by 18% four years after electronic health record were introduced; rates subsequently increased 5-7% annually. Rates of laboratory usage in the Colorado region remained generally stable, rising 14% before electronic health records were introduced and falling 2.9% in the two subsequent years.

Quality of care process measures

Intermediate measures of quality of health care remained unchanged or slightly improved after electronic health record were introduced (table 2). This allays any fleeting concerns that decreased usage compromised quality of care.

Table 2

Percentage of eligible members receiving intervention in Colorado and Northwest regions before and after implementation of electronic health records (years I1 and I2). Data from Health Plan Employer and Data Information Set quality of care indicators

View this table:


We compared use of ambulatory care before and after the introduction of electronic health records and found decreased use of both primary and specialty services. In addition, the percentage of members with more than three visits a year fell. Telephone contacts, which were rendered more effective by the immediate availability of patient information, supplanted some outpatient visits. We also found stable or improved quality of care measures after electronic health records were implemented.

Possible explanations

The observed changes in use of services have many potential explanations. Efficiency of outpatient care may have been increased by the readily available comprehensive clinical information. This conclusion is supported by the shifting frequency distribution of visits, the increased use of phone contact in lieu of a visit, and doctors' consistent observations across the two regions that electronic health records enabled them to identify and resolve patients' health issues in the first contact or with fewer contacts.

We were able to eliminate several confounding factors as potential causes of reduced use of ambulatory care. These include changes in age distribution, reduced practitioner availability (as measured by the ratio of all primary care providers to members), and possible shifting of primary care to specialty, emergency, or outside care (as measured by rates of visits and outside referral rates).

Organisational pressure to reduce use of ambulatory care could have caused similar effects. Despite differing operating systems and organisational goals, neither region had an implicit or explicit goal to reduce outpatient visits during the study. No meaningful changes occurred in copayments for primary care or specialty care, and neither the health plan products nor the products offered by employer purchaser groups changed substantially during the study period.

Decision support embedded in the electronic health records also potentially influenced usage. The role of decision support systems in guiding clinical decision making has been documented.14 15 Decision supports were more developed and widespread in the Northwest system than in the Colorado system, but their contribution to the reduced rates of use was not a focus of this study.


A non-representative population could potentially limit the applicability of our findings. However, neither region represents an unusual patient population. The members in both regions have an average or above average illness burden, as measured by the nation's Medicare hierarchical category classification scores.

Administrative level data leave some questions about the effect of electronic health records unanswered. For instance, keyinformants expected that improved availability of complete laboratory data would eliminate redundant testing, yet we found no conclusive evidence that this was the case. Decreases in use of laboratory services were neither consistent in the two regions nor sustained over time. However, inappropriate use of clinical laboratory services before the availability of electronic health records may also include underuse of some tests (such as glycated haemoglobin and lipid screening). Examination of ordering patterns for specific tests may better reflect the effect of electronic health records than overall trends.

What is already known on this topic

Electronic medical records have been shown to improve the quality of health care in specific areas

Their effect on overall use and quality of ambulatory care is unknown

What this study adds

Introduction of electronic records reduced visits to primary and specialist outpatient care in two regions of a US health maintenance organisation

No concomitant increase in use of other services was found

Selected measures of quality of care were unaffected

Effect on doctors

Introduction of electronic health records represents a substantial change in doctors' workflow. Some research indicates that electronic health record systems impose a greater burden on clinicians.1618 This effect may be temporary19 or situation dependent.20 However, internal Kaiser Permanente work indicates that the effect of electronic health records on clinician workload is highly variable but time neutral on average (unpublished data). Additionally, Geisinger Health System noted a significant improvement in productivity after introducing electronic records, as measured by McGladrey relative value units (Joseph Bisordi, personal communication, 2003).

Interviews with clinical and operations leaders indicated that strong leadership support for realising potential efficiency gains was key to successful implementation. Organisational structure supporting free flow of information and efficiency gains is key to realising the benefits of electronic health record implementation.21 However, the organisational dynamics affecting implementation require further study.

Effect on quality

Despite perceptions of improved quality of care after electronic health records were introduced, we found generally stable and only occasionally improved performance on selected measures. Both regions are high performing, which may make it more difficult to identify marginal quality improvements.22 Health Plan Employer Data and Information Set data for reporting year 2003 placed Colorado among the 10 top performing health plans in the country for effectiveness of care measures and the Northwest among the Pacific region's five top performing plans.23 At a minimum, we can assert that electronic health records and the resulting effects on usage do not reduce the quality of care and may in fact increase appropriate use of healthcare services.


We thank the many physicians, operations leaders, and analysts in the Kaiser Permanente Colorado and Northwest regions for their contribution to this study. In particular, we thank Homer Chin, Nan Robertson, and Marianne Gapinski for their sponsorship and insights and Jenni Green for advice and help in writing this article.


  • Contributors AW sponsored and facilitated the work. TG directed the study and drafting of the paper. LJ and YZ analysed the data. TG contributed to the conception and design of the study. LJ was involved in interviewing clinical and operations leaders. AW, TG, LJ, and YZ took part in planning the study, interpreting the data, and commented on the manuscript. AW and LL have sponsored the continuation, dissemination, and application of this work. TG is the guarantor.

  • Funding Kaiser Foundation Health Plan and Hospitals.

  • Competing interests None declared.

  • Ethical approval Not required.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.