Rapid responses are electronic comments to the editor. They enable our users
to debate issues raised in articles published on bmj.com. A rapid response
is first posted online. If you need the URL (web address) of an individual
response, simply click on the response headline and copy the URL from the
browser window. A proportion of responses will, after editing, be published
online and in the print journal as letters, which are indexed in PubMed.
Rapid responses are not indexed in PubMed and they are not journal articles.
The BMJ reserves the right to remove responses which are being
wilfully misrepresented as published articles or when it is brought to our
attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not
including references and author details. We will no longer post responses
that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Schmidt, Pittler and Ernst (1) share with the readers of this journal
an update on their findings concerning the extent to which publication
bias exists in 3 different journals dealing with Complementary and
Alternative Medicine (CAM). The authors results suggest that their is now
less publication bias in the 3 journals reviewed than was the case 6 years
ago. However, in reporting their findings they conclude that bias is still
rife and make no mention of the fact that publication bias is far from
unique to the CAM literature.
The following papers (2-8), which I briefly summarise, are only a few
of the many which suggest that the biomedical literature, beyond that
dealing with CAM, is similarly still plagued by publication bias and like
CAM has a long way to go in it's process of maturation.
A retrospective survey, of a cohort of 748 studies submitted to a
hospital ethics committee over 10 years, examined the extent to which
publication was influenced by study outcome. Response to the questionnaire
was received for 520 (70%) of the eligible studies. Of the 218 studies
analysed with tests of significance, those with positive results
(P<0.05) were much more likely to be published than those with negative
results (P0.10) (hazard ratio 2.32 (95% confidence interval 1.47 to 3.66),
P=0.0003). The authors concluded, "This study confirms the evidence of
publication bias found in other studies and identifies delay in
publication as an additional important factor". (2)
Another investigation attempted to assess whether authors were more
likely to report trials with statistically significant results in English
than in German. The authors studied pairs of RCT reports, matched for
first author and time of publication, with one report published in German
and the other in English. A total of 40 pairs of reports were analysed.
Only 35% of German-language articles, compared with 62% of English-
language articles, reported significant (p < 0.05) differences in the
main endpoint between study and control groups. Therefore the only
characteristic that predicted publication in an English-language journal
was a significant result. (3)
One group of researchers carried out a systematic search for
published reports of randomised controlled trials investigating
ondansetron's effect on postoperative emesis. Of 84 trials, published
between 1991 and 1996, 9 had been published in 14 further reports,
duplicating data from 3335 patients receiving ondansetron; none used a
clear cross reference. Overall the authors found 17% of published
randomised trials and 28% of the patient data were duplicated. Trials
reporting greater treatment effect were significantly more likely to be
duplicated. Inclusion of duplicated data in meta-analysis led to a 23%
overestimation of ondansetron's antiemetic efficacy. (4)
A recent investigation set out to assess the effect of publication
bias on systematic reviews and meta-analyses. The authors used two
different methods, the trim and fill methods, to analyse 48 reviews in the
Cochrane Database of Systematic Reviews. The trim method estimated that 26
(54%) of reviews had missing studies and in 10 the number missing was
significant. The corresponding figures with the fill method were 23 (48%)
and eight. In four cases, statistical inferences regarding intervention
effect were changed after adjusting for publication bias . It is
concluded, "Publication or related biases were common within the sample of
meta-analyses assessed". (5)
Free papers presented to the Annual Scientific Congress (ASC) of the
Royal Australasian College of Surgeons (RACS) were reviewed for the years
1994, 1995 and 1996. Reports were examined for evidence of publication
bias. Responses were obtained from 302 of 576 presentations considered
suitable. A total of 55% of responding authors reported publication of
their paper. There was a significant bias in favour of publication of
positive results (98 of 139 positive vs 76 of 159 inconclusive or negative
reports; P < 0.01). The authors concluded, "...there is bias in the
subsequent publication, which favours positive reports". (6)
A meta-analyses was performed which used data from 4 prospective
investigations which included a total of 997 initiated studies. The
studies were followed in order to determine study results, publication
status, and reasons for nonpublication. Subsequent analysis revealed a
positive association between "significant" study results and publication
(OR = 2.88; 95% [CI] 2.13 to 3.90). In every case, failure to publish was
investigator-based, and not due to editorial decisions. (7)
A recent investigation examined the extent to which publication bias
exists within the obesity treatment literature. The authors used a
quantitative literature synthesis of 4 published meta-analyses from the
peer reviewed literature. To assess publication bias, several regression
procedures were used. The authors found evidence of publication bias in 2
of the 4 published meta-analyses, such that reviews of published studies
were likely to overestimate clinical efficacy. The authors concluded, "As
in other disciplines, publication bias appears to exist in the obesity
treatment literature". (8)
Schmidt, Pittler and Ernst (1) share with us their findings, of
decreasing publication bias in 3 CAM related journals, with the
conclusion, "These findings imply that bias is still rife but is
diminishing." That the authors failed to put there finding in context is
unfortunate. They could have done so simply by making mention of the
extent to which publication bias is still rife in biomedical literature in
general. Attempting to improve the conduct and unbiased reporting of
research findings, prospective trial registration, and easier access to
data from published and unpublished studies is something that all health
care researchers and disciplines should be striving toward. (9)
References:
(1) Schmidt K, Pittler MH, Ernst E. Letters: Bias in alternative
medicine is still rife but is diminishing. BMJ 2001;323:1071.
(2) Stern JM, Simes RJ. Publication bias: evidence of delayed
publication in a cohort study of clinical research projects. BMJ
1997;315:640-645
(3) Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C,
Antes G. Language bias in randomised controlled trials published in
English and German. Lancet 1997;350:326-9
(4) Tramèr MR, Reynolds DJM , Moore RA, McQuay HJ. Impact of covert
duplicate publication on meta-analysis: a case study BMJ 1997;315:635-40.
(5) Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR. Empirical
assessment of effect of publication bias on meta-analyses. BMJ
2000;320:1574-7.
(6) Kiroff GK. Publication bias in presentations to the Annual
Scientific Congress. ANZ J Surg 2001;71:167-71.
(7) Dickersin K, Min YI. Publication bias: the problem that won't go
away. Ann N Y Acad Sci 1993;703:135-46.
(8) Allison DB, Faith MS, Gorman BS. Publication bias in obesity
treatment trials? Int J Obes Relat Metab Disord 1996;20:931-7.
(9) Sterne JAC, Egger M, Davey Smith G. Investigating and dealing
with publication and other biases in meta-analysis. BMJ 2001;323:101-5.
Competing interests:
No competing interests
06 November 2001
Adrian Wenban
Associate Governor Australian Spinal Research Foundation
Schmidt,Pittler, and Ernst's interesting data are not necessarily
evidence of lack of maturity of complementary medicine. It depends on
what we mean by "bias". If the researchers whose articles are reviewed,
have ignored or improperly interpreted unfavourable data, this is one kind
of "bias". But if the researchers tended to study therapies for which
there is already strong, positive anecdotal evidence, this can also be
called "bias". But it would not be a sign of immaturity.
Competing interests:
No competing interests
05 November 2001
Frank Leavitt
Chairman, Centre for Asian and International Bioethics,
Faculty of Health Sciences, Ben Gurion University of the Negev, Beer Sheva, Israel
Publication bias: not isolated to CAM literature.
Schmidt, Pittler and Ernst (1) share with the readers of this journal
an update on their findings concerning the extent to which publication
bias exists in 3 different journals dealing with Complementary and
Alternative Medicine (CAM). The authors results suggest that their is now
less publication bias in the 3 journals reviewed than was the case 6 years
ago. However, in reporting their findings they conclude that bias is still
rife and make no mention of the fact that publication bias is far from
unique to the CAM literature.
The following papers (2-8), which I briefly summarise, are only a few
of the many which suggest that the biomedical literature, beyond that
dealing with CAM, is similarly still plagued by publication bias and like
CAM has a long way to go in it's process of maturation.
A retrospective survey, of a cohort of 748 studies submitted to a
hospital ethics committee over 10 years, examined the extent to which
publication was influenced by study outcome. Response to the questionnaire
was received for 520 (70%) of the eligible studies. Of the 218 studies
analysed with tests of significance, those with positive results
(P<0.05) were much more likely to be published than those with negative
results (P0.10) (hazard ratio 2.32 (95% confidence interval 1.47 to 3.66),
P=0.0003). The authors concluded, "This study confirms the evidence of
publication bias found in other studies and identifies delay in
publication as an additional important factor". (2)
Another investigation attempted to assess whether authors were more
likely to report trials with statistically significant results in English
than in German. The authors studied pairs of RCT reports, matched for
first author and time of publication, with one report published in German
and the other in English. A total of 40 pairs of reports were analysed.
Only 35% of German-language articles, compared with 62% of English-
language articles, reported significant (p < 0.05) differences in the
main endpoint between study and control groups. Therefore the only
characteristic that predicted publication in an English-language journal
was a significant result. (3)
One group of researchers carried out a systematic search for
published reports of randomised controlled trials investigating
ondansetron's effect on postoperative emesis. Of 84 trials, published
between 1991 and 1996, 9 had been published in 14 further reports,
duplicating data from 3335 patients receiving ondansetron; none used a
clear cross reference. Overall the authors found 17% of published
randomised trials and 28% of the patient data were duplicated. Trials
reporting greater treatment effect were significantly more likely to be
duplicated. Inclusion of duplicated data in meta-analysis led to a 23%
overestimation of ondansetron's antiemetic efficacy. (4)
A recent investigation set out to assess the effect of publication
bias on systematic reviews and meta-analyses. The authors used two
different methods, the trim and fill methods, to analyse 48 reviews in the
Cochrane Database of Systematic Reviews. The trim method estimated that 26
(54%) of reviews had missing studies and in 10 the number missing was
significant. The corresponding figures with the fill method were 23 (48%)
and eight. In four cases, statistical inferences regarding intervention
effect were changed after adjusting for publication bias . It is
concluded, "Publication or related biases were common within the sample of
meta-analyses assessed". (5)
Free papers presented to the Annual Scientific Congress (ASC) of the
Royal Australasian College of Surgeons (RACS) were reviewed for the years
1994, 1995 and 1996. Reports were examined for evidence of publication
bias. Responses were obtained from 302 of 576 presentations considered
suitable. A total of 55% of responding authors reported publication of
their paper. There was a significant bias in favour of publication of
positive results (98 of 139 positive vs 76 of 159 inconclusive or negative
reports; P < 0.01). The authors concluded, "...there is bias in the
subsequent publication, which favours positive reports". (6)
A meta-analyses was performed which used data from 4 prospective
investigations which included a total of 997 initiated studies. The
studies were followed in order to determine study results, publication
status, and reasons for nonpublication. Subsequent analysis revealed a
positive association between "significant" study results and publication
(OR = 2.88; 95% [CI] 2.13 to 3.90). In every case, failure to publish was
investigator-based, and not due to editorial decisions. (7)
A recent investigation examined the extent to which publication bias
exists within the obesity treatment literature. The authors used a
quantitative literature synthesis of 4 published meta-analyses from the
peer reviewed literature. To assess publication bias, several regression
procedures were used. The authors found evidence of publication bias in 2
of the 4 published meta-analyses, such that reviews of published studies
were likely to overestimate clinical efficacy. The authors concluded, "As
in other disciplines, publication bias appears to exist in the obesity
treatment literature". (8)
Schmidt, Pittler and Ernst (1) share with us their findings, of
decreasing publication bias in 3 CAM related journals, with the
conclusion, "These findings imply that bias is still rife but is
diminishing." That the authors failed to put there finding in context is
unfortunate. They could have done so simply by making mention of the
extent to which publication bias is still rife in biomedical literature in
general. Attempting to improve the conduct and unbiased reporting of
research findings, prospective trial registration, and easier access to
data from published and unpublished studies is something that all health
care researchers and disciplines should be striving toward. (9)
References:
(1) Schmidt K, Pittler MH, Ernst E. Letters: Bias in alternative
medicine is still rife but is diminishing. BMJ 2001;323:1071.
(2) Stern JM, Simes RJ. Publication bias: evidence of delayed
publication in a cohort study of clinical research projects. BMJ
1997;315:640-645
(3) Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C,
Antes G. Language bias in randomised controlled trials published in
English and German. Lancet 1997;350:326-9
(4) Tramèr MR, Reynolds DJM , Moore RA, McQuay HJ. Impact of covert
duplicate publication on meta-analysis: a case study BMJ 1997;315:635-40.
(5) Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR. Empirical
assessment of effect of publication bias on meta-analyses. BMJ
2000;320:1574-7.
(6) Kiroff GK. Publication bias in presentations to the Annual
Scientific Congress. ANZ J Surg 2001;71:167-71.
(7) Dickersin K, Min YI. Publication bias: the problem that won't go
away. Ann N Y Acad Sci 1993;703:135-46.
(8) Allison DB, Faith MS, Gorman BS. Publication bias in obesity
treatment trials? Int J Obes Relat Metab Disord 1996;20:931-7.
(9) Sterne JAC, Egger M, Davey Smith G. Investigating and dealing
with publication and other biases in meta-analysis. BMJ 2001;323:101-5.
Competing interests: No competing interests