A third of recommendations in Scottish guidelines are based on poorest evidence, finds study

Expert groups that produce the longest clinical guidelines make the biggest proportion of recommendations that are based on poor evidence, a study has concluded.1

To investigate the levels of evidence used in guidelines, the researchers examined 42 guidelines of the Scottish Intercollegiate Guideline Network (SIGN). They chose these guidelines because they are internationally respected and they contribute to national policies, such as the Quality and Outcomes Framework (QOF), the incentive payment scheme for GPs in the United Kingdom.

SIGN guidelines have four grades of recommendation. A grade recommendations are the best and are based on at least one high quality meta-analysis, systematic review, or randomised trial applicable to the target population, while D grade recommendations are based on case reports, case series, or expert opinion.

The researchers found that over half (51%) of the guidelines’ 1999 recommendations were based on poor evidence (level C and D) and over a third (35%) were almost entirely based on expert opinion (level D).

They also found that the proportion of level D evidence increased with the number of recommendations made (Kendall’s τ=0.22 (approximate 95% confidence interval 0.008 to 0.45); P=0.04; and Spearman’s ρ=0.22 (0.02 to 0.57); P=0.04).

The authors said that they have identified a trend that has been overlooked in the past: that some groups emphasise poorly evidenced recommendations.

They did not examine why the trend occurred, although it supported the observation that experts who indulged in “‘group think’ may view their own opinion as more authoritative then science can support.”

The authors, from the research and development support unit at Dumfries and Galloway Royal Infirmary in Dumfries, pointed out that relying on expert opinion had a poor track record and could cause harm. For example, they said, in the past expert groups had for too long recommended the use of radical mastectomy over conservative surgery for breast cancer, pulmonary artery catheters in heart failure, and electronic fetal monitoring in low risk pregnancies.

The longest SIGN guideline was on diabetes, which at 161 pages was 61 pages longer than the next largest. It used only level D evidence for its recommendations. One reason for this may be the fact that SIGN guidelines are used to inform QOF. The requirement to evaluate performance drives the use of surrogate outcomes, said the researchers. Previously the guideline recommended reducing glycosylated haemoglobin, which resulted in the increased use of rosiglitazone. Both the practice and the drug are now known to be potentially harmful. The authors ask: “Is it possible that the repeated use of surrogate outcomes rises from group dynamics driven by a powerful external agenda?”

They concluded that “guidelines should be brief and based on scientific evidence” and that their findings support calls for a review of how evidence is used in guidelines.

• Download figure
• Open in new tab
• Download powerpoint