What happened to the valid POEMs? A survey of review articles on the treatment of type 2 diabetesBMJ 2003; 327 doi: http://dx.doi.org/10.1136/bmj.327.7409.266 (Published 31 July 2003) Cite this as: BMJ 2003;327:266
- 1Harrisburg Family Practice Residency, Harrisburg, Pennsylvania, USA
- 2University of Virginia Department of Family Medicine, Charlottesville, Virginia, USA
- Correspondence to: A F Shaughnessy, PinnacleHealth System, PO Box 8700, Harrisburg, PA 17105, USA
- Accepted 1 April 2003
Objective To evaluate systematically the review literature on type 2 diabetes to assess transmission of the findings of the United Kingdom prospective diabetes study (UKPDS), an important source of recent valid patient oriented evidence that matters (POEMs).
Design Inception cohort analysis of the recent medical literature.
Studies reviewed Thirty five reviews on treatment of type 2 diabetes.
Main outcome measures Presentation of three types of information from UKPDS in review articles: recommendations based on patient oriented outcomes of study; recommendations contradicted by patient oriented outcomes of study; and recommendations based on disease oriented outcomes for which no patient oriented evidence exists.
Results Only six of the reviews included the POEM that tight blood glucose control had no effect on diabetes related or overall mortality. Just seven mentioned that metformin treatment was associated with decreased mortality. Most (30) of the reviews did not report that diabetic patients with hypertension benefit more from good blood pressure control than good blood glucose control. No review pointed out that treatment of overweight patients with type 2 diabetes with insulin or sulphonylurea drugs had no effect on microvascular or macrovascular outcomes. Thirteen reviews recommended drugs as first line treatment for which we do not have patient oriented outcomes data. The average validity assessment score was 1.3 out of a possible score of 15 (95% confidence interval 0.9 to 1.8).
Conclusions Review articles on the treatment of type 2 diabetes have not accurately transmitted the valid POEM results of the UKPDS to clinicians. Clinicians relying on review articles written by experts as a source of valid POEMs may be misled.
“Information mastery” involves the ability to identify, evaluate, and apply valid and relevant information quickly.1 It is based on the concept that information has different degrees of usefulness, and that the best information is highly valid, highly relevant, and takes little work to locate, evaluate, and understand. Validity is a matter of satisfying the criteria developed by the Evidence-Based Medicine Working Group.2 Relevant information is called “patient oriented evidence that matters” (POEMs).3 Patient oriented evidence tells clinicians, directly and without the need for extrapolation, that a diagnostic, therapeutic, or preventive procedure helps patients live longer or live better. This information matters when it requires a change in practice of a clinician.
Patient oriented evidence is contrasted with “disease oriented evidence” (DOE), which is research focusing on either intermediate or surrogate outcomes.4 Many practices in medicine are currently based on disease oriented evidence, which may later be shown to be either correct or incorrect when the patient oriented outcomes are studied. Numerous examples exist of medical practice based on disease oriented evidence that have been shown, after the publication of truly patient oriented evidence, to be not only ineffective but even harmful (table 1).5 On the other hand, new, valid POEMs often are rejected, especially when they don't “make sense” or conflict with disease oriented evidence.
Although valid POEMs are usually found in research articles, most clinicians rely on expert recommendations on which to base their clinical care.6 These recommendations are transmitted via review articles, book chapters, continuing medical education presentations, consensus conferences, and consensus guideline development.
These recommendations can be supported either by patient oriented evidence, disease oriented evidence, or some combination of preliminary research findings augmented with expert opinion. We sought to measure the accuracy of one mode of information dissemination by evaluating how experts represented the results of the United Kingdom prospective diabetes study (UKPDS).7–10 We choose this study and the transmission of its results for several reasons. Other than the university group diabetes project, it is the only large study of patients with type 2 diabetes of new onset to evaluate the effect of intensive blood glucose control on long term mortality. As such, it presents vital information that should be used to guide patient care. Also, it is a useful study for our purposes because it contains both patient oriented outcomes that are valuable to clinicians as well as several outcomes that are disease oriented and thus have less immediate clinical application.
The United Kingdom prospective diabetes study (UKPDS)
Started in 1977, the UKPDS was designed to determine whether tight glycaemic control decreases diabetes related complications and increases life expectancy. A sub-study within the main study investigated whether tighter control of blood pressure in patients with hypertension decreased complications.
The investigators enrolled about 4000 patients with type 2 diabetes of new onset. These patients were assigned to receive either conventional or more intensive treatment and were monitored for a median of 10.7 years for long term effects. Conventional treatment aimed to maintain fasting plasma glucose readings below 15 mmol/l (270 mg/dl), whereas intensive treatment aimed for “tight” control of less than 6 mmol/l (110 mg/dl). Half of the patients receiving intensive treatment, mostly non-overweight patients, reached this goal. Table 2 lists the major outcomes of the UKPDS.7–11
The UKPDS provided several outcomes that can be categorised as POEMs (table 2). Attempting to achieve tight blood glucose control did not prevent premature mortality. However, regardless of their level of blood glucose control, overweight patients receiving metformin had significantly fewer diabetes related outcomes, and fewer died because of diabetes or other causes. The effect on outcomes of tight blood pressure control (<150/<85 mm Hg) were more impressive than tight blood glucose control: in addition to lowering the risk of aggregate complications, good blood pressure control also decreased mortality.
The study also evaluated the effect of blood glucose control on disease oriented outcomes. Tight control of blood glucose decreased the aggregate risk of 21 different complications, although most of this benefit was due to changes in intermediate outcomes. For example, the need for photocoagulation was diminished, although rates of vision loss were not affected. Changes in serum creatinine levels were less, though the likelihood of developing end stage renal disease was not affected.
To evaluate the selection and transmission of the results of the UKPDS, we performed a literature search to identify review articles written two years or more after publication of the major findings of the study. We were very broad in our definition of a review article, including reviews indexed by the National Library of Medicine (Medline), review articles published in controlled circulation journals, and textbooks, including electronic texts.
To assemble a convenience sample of review articles we searched Medline using the medical subject heading “diabetes mellitus, non-insulin dependent (EXPLODE),” subheading “therapy, drug therapy,” and limits “review” and English language publications. We also searched the Cumulative Index to Nursing and Allied Health Literature (CINAHL) using the search terms “diabetes (EXPLODE, FOCUS),” subheading “drug therapy,” and limit “review.” Other databases searched were the Cochrane database (completed reviews and the Database of Abstracts of Reviews of Effectiveness), the Turning Research into Practice (TRIP) database, Bandolier on the Web, and MD Consult. We also searched four controlled circulation journals with online contents. The search was not designed to be exhaustive but to be a realistic representation of expert written reviews available to many practising clinicians.
We included articles if they were reviews (including systematic reviews and meta-analyses), commentaries, structured reviews, or book chapters published in English focusing on the treatment of patients with type 2 diabetes. We excluded editorials; drug monographs; articles dealing primarily with the pathophysiology, screening, or diagnosis of diabetes; reviews focusing on non-drug treatments; and reviews of treatment of type 2 diabetes restricted to children or adolescents.
Evaluation of reviews
After obtaining the articles, the identifying information (author, institution, journal source) was masked, and copies of articles were evaluated separately by both of us. We evaluated each article for validity using a scoring system of strict12 and looser13 criteria of validity (see box), weighting the scoring system to focus on the most crucial elements. We piloted these criteria on review articles on other topics and were able to use them to separate reviews into three categories: systematic reviews (score ≥ 10); evidence based reviews (score 5-9), and expertise based reviews (score < 5). Each of us separately reviewed all of the papers and was unaware of the scores of the other reviewer. Inter-rater reliability was measured through use of the ϕ statistic, which offers several advantages over the κ statistic.14 Discrepancies were resolved by discussion.
Table 2 summarises the main results of the four major papers reporting on the UKPDS published in 1998. To determine the flow of information, we evaluated the review articles to document the presentation of three types of information: recommendations based on the patient oriented outcomes of the study, recommendations contradicted by the patient oriented outcomes of the study, and, recommendations based on disease oriented outcomes for which no patient oriented evidence exists.
From our literature search we identified 436 reviews published between November 2000 and May 2002 (see figure). Because of an overlap among sources, these numbers do not represent separate reviews. We found six online book chapters in MD Consult and 257 page hits to diabetes in four non-indexed controlled circulation journals with online contents. After removing reviews that did not meet our inclusion criteria and five reviews that did not present any results from the UKPDS, we analysed the content of 35 review articles of type 2 diabetes.
Table 3 outlines the characteristics of the review articles. Based on academic affiliation, most (30/35) of the reviews were “academic summary reviews”3 written by endocrinologists for non-endocrinologists.
In our evaluation of articles' validity, we agreed 96.3% of the time (ϕ = 0.649, P < 0.0001). Most of our disagreements centred on the question of whether patient oriented evidence was used to support key recommendations. The mean validity score for the 35 papers was 1.3 out of possible score of 15 (95% confidence interval 0.9 to 1.8). None of the reviews would have met the criteria set forth by the Evidence-Based Medicine Working Group to be considered a valid systematic review.12 With the less rigorous criteria set forth by Siwek,13 the reviews, on average, fulfilled only 13% of the requirements (average score 1.2 out of a possible 9 (95% confidence interval 0.8 to 1.6). In our independent analysis of these review articles, we agreed with each other 85.4% of the time (ϕ = 0.632, P < 0.0001).
Criteria used to evaluate the validity of the reviews
Strict criteria (adapted from Oxman12)
The search methodology was detailed and exhaustive (weight = 2)
The inclusion or exclusion criteria were reported (weight = 2)
Validity criteria were reported (or considered but not used) (weight = 2)
The method used to combine studies was reported (weight = 1)
Looser criteria (adapted from Siwek13)
At least two relevant, evidence based sources were searched (weight = 2)
Patient oriented outcomes were presented to support key recommendations (weight = 2)
The authors addressed clinical versus statistical significance (weight = 1)
Levels of evidence were presented to support key recommendations (weight = 3)
Presentation of patient oriented evidence that matters
The finding that tight glucose control does not change overall mortality or diabetes related mortality was mentioned in only six of the 35 reviews. One reviewer explained that the lack of effect could have been due to an insufficient effect on haemoglobin A1c, too short a study period, or that, despite discussing mortality, “the end-points measured were not sufficiently sensitive.”15
Only 14 of the reviews mentioned the effect of metformin on diabetes related outcomes in overweight patients. Twenty eight reviews failed to inform readers that metformin was the only drug associated with decreased mortality.
Almost half (17) of the reviews did not mention the need for blood pressure control in patients with diabetes. Only five reviewers pointed out that diabetic patients with hypertension benefit more from blood pressure control than blood glucose control. Only seven of the reviews pointed out the finding that angiotensin converting enzyme inhibitors and β blockers, as starting drugs for controlling hypertension, were equivalent.
Disease oriented evidence
Most of the reviews (28) mentioned that tight control of blood glucose decreased the aggregate outcome of 21 complications. However, only one reviewer pointed out that much of this aggregate decrease could be attributed to changes in the surrogate outcomes of less change in serum creatinine concentration and a decreased need for photocoagulation, without any difference between the groups in vision loss or need for dialysis. No reviewer pointed out that treatment with insulin or sulphonylurea drugs had no significant effect on microvascular or macrovascular outcomes in overweight patients, the group most commonly encountered in clinical practice.
Several reviews (7) reported the weaker epidemiological analysis relating decreased haemoglobin A1c to decreased complications,16 using this analysis to support a conclusion that HbA1c should be as low as possible for all patients regardless of the methods used to do so. No reviewer mentioned that lowering HbA1c with drug treatment was not associated with a significant benefit in the intention to treat analysis of the randomised study.
Issues for which no patient oriented evidence exists
Several hypoglycaemic agents are marketed that were not studied in the UKPDS. Currently there are no patient oriented outcomes data available to support their use in treating patients with type 2 diabetes. However, 13 of the reviews recommended drugs without POEM data as first line treatment, and seven stated that drugs that have an equivalent effect on HbA1c were interchangeable.
We found that the results of the most important research in diabetes in the past 25 years were incompletely and often inaccurately transmitted to readers. The extent to which these mainly non-systematic reviews have changed clinical practice is unknown, but reviews are one of the typical ways new information is transmitted to clinicians. Thus, our results have far reaching implications for how the current medical information system transmits new research results from academia to practitioners.
The lack of effect of tight blood glucose control on mortality was not reported by 83% of the reviewers, and none of the reviewers reported that neither treatment with insulin or sulphonylureas affected microvascular or macrovascular complications of diabetes in overweight patients.11 17 Instead, more optimistic results were usually reported, focusing on morbidity: “This study… has reinforced the belief that improved control of blood glucose levels can substantially lower the overall morbidity associated with this disease, underscoring the urgency to obtain better glucose control in these patients.”18 Only five of the reviewers mentioned the greater impact of blood pressure control in patients with diabetes who also have hypertension. A subsequent analysis of preventive measures in patients with diabetes found that control of cholesterol and blood pressure have a significantly greater effect on mortality and complications than tight glucose control.19
Many recommendations did not take into account the result, shown in the randomised aspect of the UKPDS, that changes in haemoglobin A1c, a marker of long term blood glucose control, had no impact on the morbidity or mortality associated with diabetes. Instead, several reviews, including that by the American Diabetes Association, chose to focus on the observational data provided in a subsequent analysis by Stratton.16 Treating this controlled trial as an epidemiological study may be illusory20 and reflective of the preconceptions of the reviewers.11 The newly found neutral or negative effect of hormone replacement therapy to decrease cardiovascular disease, which was thought to be beneficial based on population studies, is the latest example of the hazards of this type of research. The intention to treat (that is, randomised) results of the trial did not show a substantial benefit from reducing HbA1c levels and is a better indicator of the true impact of drug treatment. No reviewer discussed the methodological problems with the non-randomised analysis.
We were surprised by the low methodological rigour of these review articles. Others have found similar results.21 22 Initially, we intended to stratify the results of our analysis according to the quality scoring system we developed (table 2). However, the uniformly low scores of the reviews prevented this type of analysis.
The current system of transmitting new research to clinicians by means of reviews is less than optimal, at least for new important research in type 2 diabetes. Many of the reviews in our analysis did not provide the valid POEMs or identify the level of evidence supporting the reviewers' recommendations. Instead, the recommended first line treatments for type 2 diabetes seem to be based on pathophysiological reasoning rather than patient oriented outcomes data.23 Much of the advice given by reviewers is either based on valid disease oriented evidence or on non-valid POEMs. In contrast to POEMs, this advice can be called “prescriptive recommendations based on substandard evidence” (PROSE).
What is already known on this topic
Review articles, especially if they are not systematic in their approach, can present incomplete or incorrect information, with few clues given to unsuspecting readers
Previous research has indicated that the quality of a review article varies inversely with the expertise of the author
What this study adds
The results of the United Kingdom prospective diabetes study have not been transmitted accurately via review articles
Much of the “patient oriented evidence that matters” from the study is missing from these reviews
Clinicians relying on these information sources for accurate clinical information may be misled
Basing clinical practice on PROSE is not necessarily harmful unless it is contradicted by evidence from valid POEMs. Useful tools for clinicians wanting to find the best information to answer their clinical questions should, instead of providing PROSE, present recommendations with the level of evidence clearly identified for readers who must know whether the information is based on POEMs, disease oriented evidence, or the experience of the writer. Efforts are under way to include strict definitions for relevance and “levels of evidence” ratings to conclusions stated in non-systematic clinical reviews.13 In a subsequent publication, we will provide further information on the availability and assessment of other information tools. Well informed and open minded clinicians can then improve their patient management and the lives of their patients as valid POEMs replace lower level evidence.
The review articles evaluated in this study (w1-w40) are listed on bmj.com
We thank Laurie Schwing for her help with the literature search.
Contributors Both authors contributed equally to the study, and both are guarantors for it.
Funding Partial funding to support this project was obtained from the PinnacleHealth System Foundation.
Competing interests Both authors hold stocks in InfoPOEMs and receive royalties from the sale of InfoRetriever and InfoPOEMs.