The private life of systematic reviewsBMJ 1997; 315 doi: https://doi.org/10.1136/bmj.315.7109.686a (Published 13 September 1997) Cite this as: BMJ 1997;315:686
- Ian Roberts, director,
- Gill Schierhout, research fellow
Hardly a month goes by without a report in the medical literature on some aspect of systematic review methodology. The articles address the parts of a systematic review that come under the scientific spotlight in the methods section of a report; they represent the public face of a systematic review.
The conduct of a systematic review, however, depends also on enlisting the cooperation of the trialists who provide the data. In contrast to the public nature of the more technical aspects of a systematic review, details of these interpersonal interchanges are never mentioned in the published report. We argue that a closer look at these interpersonal interactions raises important questions for the evolving science of reviewing the medical literature.
Personal communication with trialists is an important way of ensuring that trials are not missed in a systematic review. Trialists can be sent a list of the trials found to date and asked if they know of any others. Most investigators are pleased to help: “Thank you for your letter …. I enclose a list of relevant studies, some of which are controlled.”
Some investigators, however, recognise that trial identification is the most arduous part of a systematic review and see this as freeloading: “You asked me in your letter whether there are other trials relevant to your review …. That is an easy way to let other persons do the work for you …. There are other publications on particular detailed issues, such as …. You have included some of them in your survey …. There are many more. You should check yourself.”
This trialist also took the opportunity to comment on the quality of one of the trials already on the list: “This paper was rejected by many journals because the study was not properly performed. Finally it got published in this rather obscure journal.” Another trialist offered similar comments on a trial that was not on the list: “There is indeed one study published in …. There are some problems with this investigation as there was ….”
Unpublished studies sometimes differ systematically from those which have been published, not because of methodological weaknesses, but because they show smaller treatment effects. Eliciting unpublished information from trialists is the social challenge of a systematic review.
Usually unpublished information is willingly provided: “As I told you I have regretted several times that I did not publish these results …. I spent several hours last night trying to find these old papers …. I am extremely happy that you are doing this important work—I hope it will soon be published.”
Some trialists ask for formal recognition for providing unpublished data: “If my data are as applicable as it seems they might be I would like to be an author on the paper …. I am not a meta-analysis person, so my input might be minimal beyond the data. None the less, my academic career is enhanced by inclusion as an author in such work.”
When unpublished data are not forthcoming the reviewer is forced to play the ethics card. Reviewer: “It is well recognised that unpublished data tend to be less promising about the effectiveness of an intervention than do published data. In the context of a small intervention effect, a large unpublished trial could have an important effect on the conclusions of a systematic review. Now that we know that there is an unpublished trial of [number] participants, we have an obligation to try and include it, since not to do so would jeopardise the efforts of all those who have helped in providing data for the review.”
And, as the review concerned a topic where prospective trials were still being conducted to resolve clinical uncertainty: “I am sure that you know how much effort this [a new trial] entails for both patients and doctors. To do this when the necessary data may already be available (albeit requiring time and effort to obtain) is a tragedy.”
Obtaining trialists' comments on the completed review is a courtesy and an opportunity to identify errors in data abstraction. Some trialists are suspicious of meta-analytic approaches: “You know and I do not know whether it is allowed to make such pooled analysis of studies including quite heterogeneous populations, who have been subjected to quite different dosage of medications and in whom the time point of outcome is so varying. I was informed that that was not allowed and therefore never done (but I am a simple clinician).”
Despite the use of explicit inclusion criteria, objections to the inclusion of certain trials are sometimes raised: “The inclusion of this trial as a class 3 category trial for those who have been involved in, or seriously confronted with, the various presentations of this trial is really hardly digestible: this trial had no protocol when it started, no written classification of categories, no advance written hypothesis to be tested …. The article has been refused by various journals for various reasons. This is confidential information and most probably you cannot and will not change this inclusion anymore, but nevertheless I wish to mention these aspects.”
The conclusions of a review can sometimes generate considerable controversy: “I strongly disagree with your conclusions. The study by [name] should not have been included—only studies assessed with a “3” should have been included.”
In other instances, investigators may be comfortable with the methods but interpret the overall results differently: “Overall, I feel that your review and evaluation are valuable in that they show a wide variation in the results of multiple studies. I, for one, would conclude from your study that there is no real benefit to be attributed.”
As the above examples illustrate, the private life of systematic reviews raises important questions for the science of reviewing research. In particular, it raises questions about the approach to trialists who have unpublished data. Do trialists own the results of a clinical trial, or are they merely custodians of information that was provided by patients for the public good? Should a request for unpublished data be a tentative petition to the benevolence of the trialist, or a demand that the trialists meet their ethical obligations? And if a trialist refuses to provide unpublished data, what sanction might reasonably be invoked?
Although this week's trial amnesty will undoubtedly help, making information about unpublished controlled trials publicly available will nevertheless remain a large part of the social challenge of systematic reviews.