Editor's Choice | This Week in BMJ | Press releases
BMJ No 7109 Volume 315 Views and reviews Saturday 13 September 1997 Personal viewsThe private life of systematic reviewsHardly a month goes by without a report in the medical literature on some aspect of systematic review methodology. The articles address the parts of a systematic review that come under the scientific spotlight in the methods section of a report; they represent the public face of a systematic review. The conduct of a systematic review, however, depends also on enlisting the cooperation of the trialists who provide the data. In contrast to the public nature of the more technical aspects of a systematic review, details of these interpersonal interchanges are never mentioned in the published report. We argue that a closer look at these interpersonal interactions raises important questions for the evolving science of reviewing the medical literature. Personal communication with trialists is an important way of ensuring that trials are not missed in a systematic review. Trialists can be sent a list of the trials found to date and asked if they know of any others. Most investigators are pleased to help: "Thank you for your letter .... I enclose a list of relevant studies, some of which are controlled." Some investigators, however, recognise that trial identification is the most arduous part of a systematic review and see this as freeloading: "You asked me in your letter whether there are other trials relevant to your review .... That is an easy way to let other persons do the work for you .... There are other publications on particular detailed issues, such as .... You have included some of them in your survey .... There are many more. You should check yourself." This trialist also took the opportunity to comment on the quality
of one of the trials already on the list: "This paper was rejected by
many journals because the study was not properly performed. Finally it
got published in this rather obscure journal." Another trialist
offered similar comments on a trial that was not on the list: "There
is indeed one study published in .... There are some p Unpublished studies sometimes differ systematically from those which
have been published, not because of methodological weaknesses, but
because they show smaller treatment effects. Eliciting unpublished
information from trialists is the social challenge of a systematic
review.
Usually unpublished information is willingly provided: "As I told you
I have regretted several times that I did not publish these results
.... I spent several hours last night trying to find these old
papers .... I am extremely happy that you are doing this important
work-I hope it will soon be published."
Some trialists ask for formal recognition for providing unpublished
data: "If my data are as applicable as it seems they might be I would
like to be an author on the paper .... I am not a meta-analysis person,
so my input might be minimal beyond the data. None the less, my
academic career is enhanced by inclusion as an author in such work."
When unpublished data are not forthcoming the reviewer is forced to
play the ethics card. Reviewer: "It is well recognised that
unpublished data tend to be less promising about the effectiveness of
an intervention than do published data. In the context of a small
intervention effect, a large unpublished trial could have an important
effect on the conclusions of a systematic review. Now that we know that
there is an unpublished trial of [number] participants, we have an
obligation to try and include it, since not to do so would jeopardise
the efforts of all those who have helped in providing data for the
review."
And, as the review concerned a topic where prospective trials were
still being conducted to resolve clinical uncertainty: "I am sure
that you know how much effort this [a new trial] entails for both
patients and doctors. To do this when the necessary data may already be
available (albeit requiring time and effort to obtain) is a tragedy."
Obtaining trialists' comments on the completed review is a courtesy
and an opportunity to identify errors in data abstraction. Some
trialists are suspicious of meta-analytic approaches: "You know and I
do not know whether it is allowed to make such pooled analysis of
studies including quite heterogeneous populations, who have been
subjected to quite different dosage of medications and in whom the time
point of outcome is so varying. I was informed that that was not
allowed and therefore never done (but I am a simple
clinician)." (Figure 0-2)
Despite the use of explicit inclusion criteria, objections to the
inclusion of certain trials are sometimes raised: "The inclusion of
this trial as a class 3 category trial for those who have been involved
in, or seriously confronted with, the various presentations of this
trial is really hardly digestible: this trial had no protocol when it
started, no written classification of categories, no advance
written hypothesis to be tested .... The article has been refused by
various journals for various reasons. This is confidential information
and most probably you cannot and will not change this inclusion
anymore, but nevertheless I wish to mention these aspects."
The conclusions of a review can sometimes generate considerable
controversy: "I strongly disagree with your conclusions. The study by
[name] should not have been included-only studies assessed with a
"3" should have been included."
In other instances, investigators may be comfortable with the methods
but interpret the overall results differently: "Overall, I feel that
your review and evaluation are valuable in that they show a wide
variation in the results of multiple studies. I, for one, would
conclude from your study that there is no real benefit to be
attributed."
As the above examples illustrate, the private life of systematic
reviews raises important questions for the science of reviewing
research. In particular, it raises questions about the approach to
trialists who have unpublished data. Do trialists own the results of a
clinical trial, or are they merely custodians of information that was
provided by patients for the public good? Should a request for
unpublished data be a tentative petition to the benevolence of the
trialist, or a demand that the trialists meet their ethical
obligations? And if a trialist refuses to provide unpublished data,
what sanction might reasonably be invoked?
Although this week's trial amnesty will undoubtedly help, making
information about unpublished controlled trials publicly available will
nevertheless remain a large part of the social challenge of systematic
reviews.
Ian Roberts,
Gill Schierhout,
|