BMJ  2003;326:1070 (17 May), doi:10.1136/bmj.326.7398.1070

Primary care

Systematic review of scope and quality of electronic patient record data in primary care

Krish Thiru, research fellow1, Alan Hassey, general practitioner, Frank Sullivan, general practitioner2

1 Fisher Medical Centre Research Unit, Skipton, North Yorkshire BD23 1EU, 2 Tayside Centre for General Practice, University of Dundee, Dundee DD2 4AD

Correspondence to: K Thiru krish.thiru{at}st-marys.nhs.uk

Abstract

Objective To systematically review measures of data quality in electronic patient records (EPRs) in primary care.

Design Systematic review of English language publications, 1980-2001.

Data sources Bibliographic searches of medical databases, specialist medical informatics databases, conference proceedings, and institutional contacts.

Study selection Studies selected according to a predefined framework for categorising review papers.

Data extraction Reference standards and measurements used to judge quality.

Results Bibliographic searches identified 4589 publications. After primary exclusions 174 articles were classified, 52 of which met the inclusion criteria for review. Selected studies were primarily descriptive surveys. Variability in methods prevented meta-analysis of results. Forty eight publications were concerned with diagnostic data, 37 studies measured data quality, and 15 scoped EPR quality. Reliability of data was assessed with rate comparison. Measures of sensitivity were highly dependent on the element of EPR data being investigated, while the positive predictive value was consistently high, indicating good validity. Prescribing data were generally of better quality than diagnostic or lifestyle data.

Conclusion The lack of standardised methods for assessment of quality of data in electronic patient records makes it difficult to compare results between studies. Studies should present data quality measures with clear numerators, denominators, and confidence intervals. Ambiguous terms such as "accuracy" should be avoided unless precisely defined.

Introduction

Accountability in the NHS is crucially dependent on the availability of high quality clinical information. This relies on the data collected.1 A clear message emerging from government policy initiatives is the need for high quality data on health collected through electronic patient record (EPR) systems. The assessment of quality and improvement of primary care datasets has been repeatedly emphasised.2 However, the criteria against which quality should be judged remain unclear.

We identified one review of mainly secondary care studies that described system and organisational factors that affect quality of the data in EPR.3 We carried out a similar review but in primary care.

Methods

We searched all major bibliographic databases and several specialist datasets during the last quarter of 2001 (see bmj.com for databases and sources and web table A for search criteria).

We established a framework for categorising and selecting review papers (box). Eligible papers had to satisfy at least one aspect (numbered) of each category (A-C) within the box.


Framework for assessing eligibility of publications for review

All three categories (A-C) needed to be satisfied for a paper to be selected

A Reference standard

A modification of the "distance from patient" concept, which classified the reference standard used to judge quality3

  1. Studies that used objective "close to patient" standards by using techniques such as video recording or direct examination
  2. Studies that used interviews or questionnaire surveys of patient, next of kin, or their immediate carers as reference standard
  3. Studies that used routine consultation data (databases, EPRs, paper records, discharge letter, etc) as standard reference
  4. Studies that used national statistics or equivalent survey results as their reference standard

B Study objectives

  1. Studies that measured change in EPR data quality or those that measured EPR data quality were classified as measuring data quality
  2. Studies that used EPRs and commented on their quality were classified as scoping data quality

C Data types

Publications that investigate:

  1. Diagnostic or symptom state of the patient
  2. Patient management data—for example, health promotion, drug treatment, referrals, tests
  3. Wider aspects of patient and practice management—for example, family history, ethnicity, socioeconomic status, immunisation, hospital episodes, consultation rates


Results

We identified 4589 abstracts and categorised 174 documents after primary exclusions. Of these, we included 47 journal publications, four reports, and one thesis from 1980-2001. Thirty seven studies measured data quality, and 15 used electronic patient records and commented on quality in the presence of a reference standard (scoping). These were analysed separately (table). Forty eight studies assessed diagnostic data, 20 assessed management information, and 13 examined wider aspects of routine data.


View this table:
[in this window]
[in a new window]
  		Proportions of data type being investigated, reference standards used to assess quality, and commonest measures of quality. Figures are numbers (percentage) of studies

 

Measuring data quality
Thirty seven studies measured data quality, and 31 were from the United Kingdom. A similar proportion had been published since 1995. Table B on bmj.com gives full details of categorisation (according to that shown in the box) and characteristics. Eight studies were prospective, although the data extraction was primarily cross sectional. The remaining articles were cross sectional or retrospective surveys. Two studies were interventional: one a case-control study involving onsite training and the other a study carried out before and after a software update. Both showed substantial improvements in recording levels after the intervention. A retrospective cohort study also showed an increase in completeness and accuracy of EPRs over five years.

Structured data (codes, classifications, and nomen-clatures) were most commonly investigated. Although textual data were mentioned, only one study considered it in any detail. Twelve documents did not present their data structure (that is, coding system name) while most did not present the precise codes being investigated. UK publications generally used Read and OXMIS (Oxford medical information systems) codes. In other countries the ICPC (international classification of primary care) codes were more widely used. ICD (international classification of diseases) codes act as a referencing standard for these primary care coding systems. Where necessary, subsidiary codes (for example, chapter headings from British National Formulary; Prescription Pricing Authority) were used.

Quality of data (reliability) was usually measured with rate comparisons. Data validity was expressed under a range of terms (completeness, correctness, accuracy, consistency, and appropriateness), which were rarely defined. Sensitivity (completeness) was the commonest such index (table, webextra table B on bmj.com).

Seven studies carried out questionnaire and telephone surveying for a reference standard, one study used video recording, and 24 used clinical information gathered during the consultation (table). Seventeen publications used triangulation within the EPR to test internal consistency of data. Medication data was the most common internal reference standard. EPR diagnostic status was appraised through electronic prescribing information and subsequently validated against the paper notes. Hospital discharge details have also been used to evaluate EPR diagnostic status. The presence of hospital diagnosis and procedural data have been found to improve the quality of data in primary care. Eighteen studies used national statistics or survey data as a reference standard for data reliability. A third of UK studies used the fourth national study of morbidity in general practice (MSGP4).

Scoping data
Fifteen studies used EPR data for research or practice management. Although the intention of these studies was not to measure data quality, they gave insight into issues of data validation. These studies relied more on positive predictive value as a quality measure than sensitivity (table). Fourteen studies considered the diagnostic status of the patient, with 10 publications dealing primarily with information on patient identification and case validation. Three used survey techniques to establish diagnostic status. Of the 12 retrospective investigations, seven used centralised datasets. These "scoping" studies were more than twice as likely to present confidence intervals than studies that measured data quality (10/15 (67%) v 11/37 (30%)).

Levels of data recording
Prescribing data are generally the most sensitive, and the ability to link prescriptions with diagnosis was the favoured means of identifying patients and establishing the predictive validity of diagnostic codes. The sensitivity of other EPR elements was wide ranging, while positive predictive value was consistently high. Diseases with clear diagnostic criteria were generally better recorded, as were data on specific procedures. Lifestyle and socioeconomic data were rarely studied and then only in terms of sensitivity.

Discussion

We believe this is the first systematic review to investigate the measurement of quality of data in primary care. Most research has been published since 1995, reflecting the increasing importance and use of EPRs. Publications were mostly descriptive. This is indicative of a topic in which the direction of change is externally imposed (that is, controlled by the pace of technological development). The appraisal of data quality has favoured practices that embrace technology and so will be an overestimate of the general picture.

The dominance of UK publications is unsurprising given the scope of this review. This also suggests an understanding of the importance of the quality of EPR data in terms of health policy and validated research databases. within the United Kingdom. Publications from non-English speaking countries were disadvantaged under our selection criteria. Those that were identified used similar techniques to measure data quality.

Measuring quality
The element of the EPR being investigated (numerator) and the components of the reference standard used to appraise its quality (denominator) were often not clearly defined within the literature (for instance, diagnostic code/diagnostic criteria). When they were defined there was inconsistency between studies. This makes comparisons risky and meta-analytical interpretation of results impossible.

Measurement theory requires that both the concepts of validity and reliability be addressed. Reliability (a precursor to validity) is a measure of stability and is appraised through the subjective comparison of rates and prevalence. Sensitivity and positive predictive value, the most widespread measures of data validity, presuppose that the selected denominator is an adequate representation of the true dimension being measured. To identify the real health status of the patient subjective (perceived), objective, and diagnostic dimensions need to be measured by different techniques and their appropriateness for EPR validation considered. To aid interpretation and make comparisons between populations, confidence intervals should be provided.

When the opportunity to record clinical data in different forms (paper and computer) exists, this decreases validity of any one to act as a true reference standard. The use of paper notes to assess EPR validity will become increasingly inaccurate as clinicians migrate to electronic systems. In the medium term it is best to consider several independent markers of quality, and those studies that used several explicit reference standards (triangulation) were more likely to reflect the true quality of electronic data (see table B on bmj.com).

To facilitate comparisons of data quality across sites and systems, it is essential to have a reference standard. In the longer term we recommend the establishment of internal reference standards based on those objective and diagnostic EPR elements recognised as having high positive predictive value (that is, diagnostic codes, prescriptions, test results, referral outcomes, procedural codes). Such reference standards can then be used to explore measures of sensitivity.


What is already known on this topic

The demonstration of quality is central to the NHS strategic agenda

Data from electronic records are expected to have a central role within healthcare commissioning, quality control, clinical governance, and the new GP contract

No standard methods of measuring data quality have been described

What this study adds

A framework for categorising and selecting papers which report data quality in primary care

Reliability of data was measured through rate comparison in 73% of studies, while validity was calculated mostly through measures of sensitivity

Markers of quality should comprise internal reference standards based on objective and diagnostic EPR elements that have high positive predictive value


This is an abridged version; the full version is on bmj.com

We thank E Mitchell (Tayside Centre for General Practice) and N Booth (Sowerby Centre for Health Informatics at Newcastle) for access to their specialist medical informatics databases.

Contributors: See bmj.com

Funding: Fisher Medical Centre Research Unit is funded by the NHS Executive Northern and Yorkshire Region. The guarantor accepts full responsibility for the conduct of the study, had access to the data, and controlled the decision to publish.

Competing interests: None declared.

References

  1. Moss F. Spreading the word: information for quality. Quality Health Care 1994; 46-50.
  2. House of Lords. Select committee on science and technology: fourth report. 2001. http://www.parliament.the-stationery-office.co.uk/pa/ld200001/ldselect/ldsctech/57/5701.htm (accessed 1 Jan 2003).
  3. Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997;4: 342-55.[Abstract/Free Full Text]
(Accepted February 6, 2003)


Add to CiteULike CiteULike   Add to Complore Complore   Add to Connotea Connotea   Add to Del.icio.us Del.icio.us   Add to Digg Digg   Add to Reddit Reddit   Add to StumbleUpon StumbleUpon   Add to Technorati Technorati    What's this?

This article has been cited by other articles:

  • Dean, B. B., Lam, J., Natoli, J. L., Butler, Q., Aguilar, D., Nordyke, R. J. (2009). Review: Use of Electronic Medical Records for Health Outcomes Research: A Literature Review. Med Care Res Rev 66: 611-638 [Abstract]  
  • Esteban-Vasallo, M., Dominguez-Berjon, M., Astray-Mochales, J, Genova-Maleras, R, Perez-Sania, A, Sanchez-Perruca, L, Aguilera-Guzman, M, Gonzalez-Sanz, F. (2009). Epidemiological usefulness of population-based electronic clinical records in primary care: estimation of the prevalence of chronic diseases. Fam Pract 26: 445-454 [Abstract] [Full text]  
  • Floor-Schreudering, A., De Smet, P. A., Buurma, H., Egberts, A. C., Bouvy, M. L (2009). Documentation Quality in Community Pharmacy: Completeness of Electronic Patient Records After Patients' First Visits. The Annals of Pharmacotherapy 43: 1787-1794 [Abstract] [Full text]  
  • Ong, D. S. Y., Kuyvenhoven, M. M., van Dijk, L., Verheij, T. J. M. (2008). Antibiotics for respiratory, ear and urinary tract disorders and consistency among GPs. J Antimicrob Chemother 62: 587-592 [Abstract] [Full text]  
  • Soler, J.-K., Okkes, I., Wood, M., Lamberts, H. (2008). The coming of age of ICPC: celebrating the 21st birthday of the International Classification of Primary Care. Fam Pract 25: 312-317 [Abstract] [Full text]  
  • Majeed, A., Car, J., Sheikh, A. (2008). Accuracy and completeness of electronic patient records in primary care. Fam Pract 25: 213-214 [Full text]  
  • Gormley, G., Connolly, D., Catney, D., Freeman, L., Murray, L. J, Gavin, A. (2008). Reporting of research data by GPs: a cautionary tale for primary care researchers. Fam Pract 25: 209-212 [Abstract] [Full text]  
  • Voorham, J., Haaijer-Ruskamp, F. M., Stolk, R. P., Wolffenbuttel, B. H.R., Denig, P., the Groningen Initiative to Analyze Type 2 Diabete, (2008). Influence of Elevated Cardiometabolic Risk Factor Levels on Treatment Changes in Type 2 Diabetes. Diabetes Care 31: 501-503 [Abstract] [Full text]  
  • Lam, K., Parkin, T., Riggs, C., Morgan, K. (2007). Use of free text clinical records in identifying syndromes and analysing health data. Vet Rec. 161: 547-551 [Abstract] [Full text]  
  • Gadzhanova, S., Iankov, I. I., Warren, J. R., Stanek, J., Misan, G. M., Baig, Z., Ponte, L. (2007). Developing High-specificity Anti-hypertensive Alerts by Therapeutic State Analysis of Electronic Prescribing Records. J. Am. Med. Inform. Assoc. 14: 100-109 [Abstract] [Full text]  
  • de Lusignan, S., Belsey, J., Hague, N., Dhoul, N., van Vlymen, J. (2006). Audit-based education to reduce suboptimal management of cholesterol in primary care: a before and after study. J Public Health (Oxf) 28: 361-369 [Abstract] [Full text]  
  • Brouwer, H., Bindels, P., Weert, H. (2006). Data quality improvement in general practice. Fam Pract 23: 529-536 [Abstract] [Full text]  
  • Jordan, K., Porcheret, M., Kadam, U. T., Croft, P. (2006). The use of general practice consultation databases in rheumatology research. Rheumatology (Oxford) 45: 126-128 [Full text]  
  • Linder, J. A., Bates, D. W., Williams, D. H., Connolly, M. A., Middleton, B. (2006). Acute Infections in Primary Care: Accuracy of Electronic Diagnoses and Electronic Antibiotic Prescribing. J. Am. Med. Inform. Assoc. 13: 61-66 [Abstract] [Full text]  
  • Hamilton, W, Peters, T J, Round, A, Sharp, D (2005). What are the clinical features of lung cancer before the diagnosis is made? A population based case-control study. Thorax 60: 1059-1065 [Abstract] [Full text]  
  • Buurma, H., De Smet, P. A., Kruijtbosch, M., Egberts, A. C. (2005). Disease and Intolerability Documentation in Electronic Patient Records. The Annals of Pharmacotherapy 39: 1640-1646 [Abstract] [Full text]  
  • Feron, J M, Paulus, D, Tonglet, R, Lorant, V, Pestiaux, D (2005). Substantial use of primary health care by prisoners: epidemiological description and possible explanations. J. Epidemiol. Community Health 59: 651-655 [Abstract] [Full text]  
  • Han, Y., Huh, S. J., Ju, S. G., Ahn, Y. C., Lim, D. H., Lee, J. E., Park, W. (2005). Impact of an Electronic Chart on the Staff Workload in a Radiation Oncology Department. Jpn J Clin Oncol 35: 470-474 [Abstract] [Full text]  
  • Delpierre, C., Cuzin, L., Fillaux, J., Alvarez, M., Massip, P., Lang, T. (2004). A systematic review of computer-based patient record systems and quality of care: more randomized clinical trials or a broader approach?. Int J Qual Health Care 16: 407-416 [Abstract] [Full text]  
  • Jordan, K., Porcheret, M., Croft, P. (2004). Quality of morbidity coding in general practice computerized medical records: a systematic review. Fam Pract 21: 396-412 [Abstract] [Full text]  
  • Morris, C J, Rodgers, S, Hammersley, V S, Avery, A J, Cantrill, J A (2004). Indicators for preventable drug related morbidity: application in primary care. Qual Saf Health Care 13: 181-185 [Abstract] [Full text]  
  • Winkelman, W. J, Leonard, K. J (2004). Overcoming Structural Constraints to Patient Utilization of Electronic Medical Records: A Critical Review and Proposal for an Evaluation Framework. J Am Med Inform Assoc 11: 151-161 [Abstract] [Full text]  
  • Stengel, D., Bauwens, K., Walter, M., Kopfer, T., Ekkernkamp, A. (2004). Comparison of Handheld Computer-Assisted and Conventional Paper Chart Documentation of Medical Records. A Randomized, Controlled Trial. JBJS 86: 553-560 [Abstract] [Full text]  
  • Winkelman, W. J., Leonard, K. J. (2004). Overcoming Structural Constraints to Patient Utilization of Electronic Medical Records: A Critical Review and Proposal for an Evaluation Framework. J. Am. Med. Inform. Assoc. 11: 151-161 [Abstract] [Full text]  
  • Porcheret, M., Hughes, R., Evans, D., Jordan, K., Whitehurst, T., Ogden, H., Croft, P., on behalf of the North Staffordshire General Pract, (2004). Data Quality of General Practice Electronic Health Records: The Impact of a Program of Assessments, Feedback, and Training. J Am Med Inform Assoc 11: 78-86 [Abstract] [Full text]  
  • Porcheret, M., Hughes, R., Evans, D., Jordan, K., Whitehurst, T., Ogden, H., Croft, P. (2004). Data Quality of General Practice Electronic Health Records: The Impact of a Program of Assessments, Feedback, and Training. J. Am. Med. Inform. Assoc. 11: 78-86 [Abstract] [Full text]  

Rapid Responses:

Read all Rapid Responses

EUNICE should sort EPR's!
Alex G Manning
bmj.com, 17 May 2003 [Full text]
Electronic medical writing: A prescription for clarity?
Javier Rodríguez-Vera, et al.
bmj.com, 21 May 2003 [Full text]
Time to move from diagnostic codes to disease complexes
K Thiru, et al.
bmj.com, 29 May 2003 [Full text]
Access jobs at BMJ Careers
Whats new online at Student BMJ