Intended for healthcare professionals

Analysis

Content area experts as authors: helpful or harmful for systematic reviews and meta-analyses?

BMJ 2012; 345 doi: https://doi.org/10.1136/bmj.e7031 (Published 01 November 2012) Cite this as: BMJ 2012;345:e7031
  1. Peter C Gøtzsche, professor1,
  2. John P A Ioannidis, professor 2
  1. 1Nordic Cochrane Centre, Rigshospitalet, Blegdamsvej 9, DK-2100 Copenhagen, Denmark
  2. 2Stanford Prevention Research Center, Stanford University School of Medicine, Stanford, California, USA
  1. Correspondence to: P C Gøtzsche pcg{at}cochrane.dk
  • Accepted 17 September 2012

Peter Gøtzsche and John Ioannidis argue that it is not always sensible to include subject experts as authors of systematic reviews and meta-analyses

Systematic reviews commonly include experts in the topic area as authors, as advised by the Cochrane Collaboration and the US Institute of Medicine (box 1).1 2 The Agency for Healthcare Quality and Research in the United States and Canada3 expects content experts to become authors of its evidence based practice reports and guidelines (Stephanie Chang, personal communication), as does the UK National Institute for Health and Clinical Excellence.4 In many countries, content area experts and their professional societies have a primary role in initiating health technology assessments, setting the agenda, specifying the questions and their boundaries, deciding what data should be included, and how the results should be interpreted and applied. Such experts are therefore probably the most influential contributors to these reports, even if none of them are authors.

Box 1: Recommendations of major groups regarding involvement of content area experts as authors of systematic reviews

Cochrane Collaboration

“Review teams must include expertise in the topic area being reviewed and include, or have access to, expertise in systematic review methodology (including statistical expertise).”1

Institute of Medicine, National Academy of Sciences, US

“Include expertise in the pertinent clinical content areas.”2

Agency for Healthcare Quality and Research Evidence-Practice Centers, US and Canada

“All EPCs collaborate with other medical and research organizations so that a broad range of experts is included in the development process.”3

National Institute for Health and Clinical Excellence, UK

“Between six and eight members of the GDG [Guideline Development Group] should be healthcare professionals (‘healthcare professional members’) who either treat people with the condition directly or manage services.”4

However, as evidence based medicine has gained momentum it has become clear that expert advice is often unreliable,5 raising questions about the validity of including content area experts as authors. We also wonder whether the pervasive incorporation of content area experts in systematic reviews and health technology assessments signifies the taming of evidence based medicine by the centuries old, expert based power system in healthcare.

No studies have examined whether it is helpful or harmful for systematic reviews to have content area experts as authors. Although it could be studied using pairs of review teams with randomisation of content area experts to only one of the pairs that examine the same research question, experts may behave differently if they know they are being observed. Another option is to study areas where several systematic reviews have been published and compare the results and conclusions between those with and without expert authors. However, areas with many reviews of the same question are likely to be particularly controversial and therefore atypical. In the absence of relevant trials, we discuss the potential benefits and harms of including content area experts as authors and provide examples from our projects.

Who is a content area expert?

For this paper, we pragmatically define a content area expert as it is most often done—namely, a specialist clinician in the area under review or an author of an included study. Depending on the topic, different specialists may qualify as experts. For example, in a review of cancer screening, experts could be oncologists, radiotherapists, surgeons, radiologists, pathologists, or general practitioners. In can be hard to agree what constitutes a content area expert in some types of reviews, such as studies of general health checks, which involve testing for many different diseases and risk factors,6 and the use of biomarkers for diverse diseases.7

Potential benefits of including content area experts

Content area experts may have inside knowledge of unpublished trials or data or about things that went wrong, such as unsuccessful blinding or under-reporting of harms, that are not apparent in trial reports or subsequent letters to the editor. They are familiar with what is of current interest in their field and what questions are considered most pressing to answer. They could also mould the review in a way that makes it more interesting and informative.

However, it does not follow that content area experts need to be authors. Moreover, when expert agreement on important issues is low, it may be unhelpful to rely on one or two experts as coauthors, as they may have personal prejudices and idiosyncrasies.8 And if many experts or professional groups and societies are involved, the methodologists’ contributions may be drowned in a sea of expert opinions. If the special knowledge of experts is needed, it can be obtained by inviting them to stakeholder meetings2 3 4 or by asking them to peer review the protocol or the review.

For reviews involving meta-analyses of raw, individual patient data, it is common practice that investigators from the primary studies coauthor the review. Involvement of these experts is considered essential in understanding how the studies were conducted, whether there are any peculiarities in the data, and to correct inconsistencies and errors that may arise from logical queries in the datasets. Their author role is expected given that they “control” the studies and the data. However, if raw data become available to the public, bodies such as the Organisation for Economic Cooperation and Development, the European Commission, and the US National Institutes of Health agree that coauthorship cannot be a condition for using the data.9

Potential harms of including content area experts

The strong opinions specialist authors often have can make it difficult to perform unbiased systematic reviews, and the conclusions reached from a certain set of results can be particularly unpredictable. Many content area experts write opinionated non-systematic reviews or commentaries, and the stronger the expertise, the stronger the prior opinion, the lower the quality of the reviews, and the less time is spent on preparing them.8

When convincing randomised trials or systematic reviews find results that invalidate expert based practice, there is always a flurry of reviews, editorials, and letters from content area experts that try to refute, or even denigrate, the evidence. This wholesale editorial assault has been studied empirically—for example, for percutaneous coronary intervention in stable coronary disease,10 hormone treatment in postmenopausal women,11 and mammography screening.12 In the case of hormone therapy, access to industry documents showed many of the reports were ghostwritten by industry.13 14

Furthermore, depending on what type of expert is involved, the review may go down different paths and come to different conclusions. A survey of urologists and radiation oncologists, for example, showed that, for prostate cancer, the experts were more likely to recommend the treatment in their own specialty than the other treatment, independently of the Gleason scores and prostate specific antigen levels, even though their estimates of the risk of impotence and incontinence after radical prostatectomy and radiotherapy, respectively, were very similar.15

Even experts in the same discipline vary greatly in their opinions. A group of 57 medical specialists met to develop a guideline for a common and important intervention and were asked to write down their prior beliefs about the probability of a particularly important outcome. The answers ranged from 0-100% with clustering around 5% and 80%.8 16

Experts also have highly varying opinions about the overall scientific quality of review articles. In a study involving 36 reviews, the intraclass correlation coefficient was only 0.23 (95% confidence interval 0.03 to 0.45) for content area experts, whereas it was 0.79 (0.65 to 0.87) for experts in research methodology.8 Opinions about the rigour of primary research also vary widely, and experts tend to praise the papers that provide the results that confirm their beliefs.12

A further point is that research studies are often biased by their design, analysis, reporting, and interpretation,1 12 and the bias may have been introduced by content area experts who share the same views as those experts who might be considered as authors for the systematic review. It therefore doesn’t make much sense to include such experts when trying to unravel the hidden bias in study reports, unless one is reasonably certain that these experts also aim at getting to the truth rather than protecting the prevailing beliefs or income in their specialty. We would not want to be guided by people who have an interest in concealing uncomfortable evidence, and clinicians, for example, find it particularly difficult to acknowledge the harms their interventions may cause.12 17

When reviews include authors of the primary studies—for example, in meta-analyses of individual patient data—the situation can be particularly difficult, as we have both experienced.18 19 Primary authors are likely to defend their results and see the meta-analysis as an opportunity to advance their views. When presented with the results of a heterogeneous meta-analysis of the association between high levels of insulin growth factor and risk of prostate cancer that included their data, primary authors of studies with significant results concluded that the meta-analysis showed a strong effect, whereas methodologists concluded the effect was null or tiny (odds ratio <1.20).18

Box 2 shows some examples from our experience.12 19 20 We use them to show some of the problems that can be encountered.

Box 2: Examples of problems of including content area experts in systematic reviews and meta-analyses

Genes influencing disease progression in people infected with HIV

In 1998, JPAI was part of an international consortium set up to perform a meta-analysis of individual patient data from studies of the effect of three genes on disease progression in HIV infected people. There was extreme interest in the topic, several studies had appeared in the most influential journals, and the data were conflicting. The protocol, data collection, and analysis plan were approved by all who contributed data to the meta-analysis.

After a lot of work, we presented the results to the collaborators who were also authors of the original papers on these genes. One of the genes (SDF-1) had no effect in any of the analyses, and the confidence intervals excluded a meaningful effect. One team had previously claimed a strong, and highly complex, effect of SDF-1 in a paper published in a high impact journal. They disagreed with the meta-analysis results, and after extensive debate withdrew themselves and their dataset from the collaboration. We therefore had to re-run all the analyses. The results remained largely similar, but now another investigator felt that he could not publish a paper that contradicted his extremely well cited paper. More months of discussion followed, and eventually he decided not to coauthor the meta-analysis, but to his credit he did not withdraw his datasets. The meta-analysis was published about 3.5 years after it started,20 and about half of this time was spent in debates with primary investigators who didn’t like the results.

Intravenous alpha-1 antitrypsin19

This treatment is used in some countries for patients with lung disease caused by inherited alpha-1 antitrypsin deficiency. We (PCG and H K Johansen) published the protocol for the review with a content area expert who had conducted the only two trials and who also had a financial conflict of interest in relation to one of the companies marketing alpha-1 antitrypsin. Content area experts had successfully lobbied a political majority in the Danish parliament to agree to reimburse the drug, which may cost up to $150 000 (£93 000); €116 000) annually per patient in the United States. When we showed the final draft review to the expert coauthor with whom we had written the protocol, he stepped down as author. There is no convincing evidence that the drug is effective, and an earlier, shorter version of the review had prompted the Danish parliament to decline to reimburse the drug. We then had a struggle within the Cochrane Collaboration to get the review published without an expert author.

Discussion

We have given reasons and examples that suggest that the general recommendation that content area experts should always be included as authors of systematic reviews and meta-analyses is questionable. There are additional issues to consider.

Some reviews are so simple that no such expertise is needed. For example, an expert about how vitamins work is not needed in a review of the effect of vitamins on overall mortality.21

When basic knowledge of the area being reviewed is needed, such knowledge can often be acquired by consulting textbooks and reading review articles. Review authors can also consult content area experts at any stage of the review, and they are usually very happy to help out in this way.

Despite our general concerns about experts, we acknowledge that methodologists can also be biased and, conversely, that some content area experts can be unbiased. We have had several positive experiences with experts who were coauthors of primary studies and leading investigators in their fields. Some have coauthored meta-analyses showing that popular interventions in their specialty didn’t work—for example, interventions against house dust mites22 and percutaneous coronary intervention for patients with stable coronary artery disease.23 We have also found it rewarding to work with young clinicians who know their specialty well but have not yet developed strong views on the merits of their treatments.

Most importantly, science should be judged by its merits, not by whom the authors are or what type of expertise they represent. The history of science shows that many breakthroughs have come from people who were not established content area experts and sometimes had not even received a formal education in the area they studied.24 We regard the theory of evolution as the most important discovery of all times. Charles Darwin studied medicine, law, and theology. He had no qualifications in biology.

Conclusions

The risk of introducing bias in a review may be increased when content area experts are included as authors. We therefore recommend that, by default, teams performing systematic reviews and meta-analyses should not include content area experts as authors. If this is considered indispensable, the inclusion of such people should be carefully justified. We agree with the US Institute of Medicine2 that experts with financial conflicts of interests should not be included as authors.

The Cochrane Collaboration states in its handbook that review teams must include expertise in the topic area being reviewed and include, or have access to, expertise in systematic review methodology.1 We suggest turning this recommendation around: review teams should include expertise in systematic review methodology and have access to expertise in the topic area. The importance of safeguarding the independence of researchers performing systematic reviews cannot be overstated.25 26 27

Notes

Cite this as: BMJ 2012;345:e7031

Footnotes

  • Contributors and sources: The authors have expertise in conducting systematic reviews and meta-analyses, contributed about equally to the paper, and are both guarantors.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; and no other relationships or activities that could appear to have influenced the submitted work.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References

View Abstract