Papers

Publication bias: evidence of delayed publication in a cohort study of clinical research projects

^a National Health and Medical Research Council Clinical Trials Centre, Mallet Street Campus, University of Sydney, Sydney NSW 2006, Australia

Correspondence to: Professor Simes john@ctc.trials.su.oz.au

	Abstract

Top Abstract Introduction Methods Results Discussion References

Objectives: To determine the extent to which publication is influenced by study outcome.
Design: A cohort of studies submitted to a hospital ethics committee over 10 years were examined retrospectively by reviewing the protocols and by questionnaire. The primary method of analysis was Cox's proportional hazards model.
Setting: University hospital, Sydney, Australia.
Studies: 748 eligible studies submitted to Royal Prince Alfred Hospital Ethics Committee between 1979 and 1988.
Main outcome measures: Time to publication.
Results: Response to the questionnaire was received for 520 (70%) of the eligible studies. Of the 218 studies analysed with tests of significance, those with positive results (P<0.05) were much more likely to be published than those with negative results (P0.10) (hazard ratio 2.32 (95% confidence interval 1.47 to 3.66), P=0.0003), with a significantly shorter time to publication (median 4.8 v 8.0 years). This finding was even stronger for the group of 130 clinical trials (hazard ratio 3.13 (1.76 to 5.58), P=0.0001), with median times to publication of 4.7 and 8.0 years respectively. These results were not materially changed after adjusting for other significant predictors of publication. Studies with indefinite conclusions (0.05P<0.10) tended to have an even lower publication rate and longer time to publication than studies with negative results (hazard ratio 0.39 (0.13 to 1.12), P=0.08). For the 103 studies in which outcome was rated qualitatively, there was no clear cut evidence of publication bias, although the number of studies in this group was not large.
Conclusions: This study confirms the evidence of publication bias found in other studies and identifies delay in publication as an additional important factor. The study results support the need for prospective registration of trials to avoid publication bias and also support restricting the selection of trials to those started before a common date in undertaking systematic reviews.

	Introduction

Top Abstract Introduction Methods Results Discussion References

In evaluating the effectiveness of treatments, the highest level of evidence is believed to be obtained from a systematic review or meta-analysis of all randomised controlled trials. However, evidence of treatment effectiveness even from randomised trials can still be biased owing to a number of factors: bias within individual trials that are not properly randomised or not analysed according to intention to treat¹; bias in selecting trials for inclusion in a meta-analysis,^{2 3} particularly when only published trials are included⁴; and bias in selecting treatment questions after examining the data.⁵ In particular, if trials with a positive effect of treatment are more likely to be published, a review limited only to published trials would give a more positive effect of treatment than a review based on all trials (published and unpublished).

The selection of trials for inclusion in a systematic review may be biased if selection is restricted to published trials (publication bias),⁴ to trials published in English language journals (language bias),⁶ to trials published in prestigious journals,⁷ or to trials cited by other authors such as in review articles (reference bias).⁸ Furthermore, even if all relevant trials are eventually published, the selection of trials may still be biased if it is restricted to trials that are published early.

Evidence for publication bias has been shown in many studies,^{9 10 11 12 13 14 15 16 17 18 19} and particularly in three cohort studies of protocols submitted to institutional ethics committees.^{20 21 22} All three studies showed a significantly greater likelihood for trials with significant results to be published than for those with negative results. These studies did not, however, examine whether there was a delay in publication for trials that were eventually published.

We determined the extent of publication bias for studies submitted to an Australian ethics committee and whether publication was delayed for studies with negative results in comparison with those with positive results.

	Methods

Top Abstract Introduction Methods Results Discussion References

Between September 1979 and December 1988, 801 submissions were received by the Royal Prince Alfred Hospital Ethics Committee for proposed medical research. Each was accompanied by a protocol providing a detailed outline of the proposed research, from which was obtained information on approval, details of the research design, the planned sample size, and the nature of any intervention. Research design was classed primarily as observational study, clinical trial, or non-trial experiment according to the definitions of Easterbrook et al.²⁰

In July 1992 the principal investigator for each study was asked to complete a questionnaire providing information on the current status; starting date, closure of recruitment, and finishing date; sample size reached; the nature of funding (none, pharmaceutical, government, other (external), or other (internal)); the rating of scientific importance of the study; the status and date of the most recent analysis; the main research questions posed by the study at the outset; the results for the main research questions; and the publication status and date of initial publication as an article in a peer reviewed journal.

Eligible studies
Eligible studies were defined as single studies approved by the Royal Prince Alfred Hospital Ethics Committee between September 1979 and December 1988 with more than one patient and with protocol information available.

Classification of study outcome
For quantitative studies, in which the main study outcome was assessed by using statistical methods with tests of significance, outcome was classed as significant (P<0.05), as showing a non-significant trend (0.05P<0.10), or as non-significant or null (P0.10). Examples of such studies include those comparing treatments and epidemiological studies examining evidence of association for risk factors.

For qualitative studies, in which the main study outcome was assessed subjectively by the principal investigator, the study was classed as showing striking, important and definite, or unimportant and negative findings. Such studies included uncontrolled phase 2 clinical trials examining response rate to treatment and studies with descriptive statistics of a population.

Analysis
Potential factors predictive of time to publication were examined in a Cox regression analysis, in which time to publication was the time from the date that the study was approved by the ethics committee to the date of first publication. Unpublished studies were censored at the date the questionnaire was completed; studies that were in press were analysed as if published on the date of completion of the questionnaire. Studies for which no analysis had yet been undertaken and two studies whose early findings had first been published before approval from this ethics committee were excluded from the survival analysis. Funding sources were compared in two ways to allow comparison with earlier studies^{20 21}: pharmaceutical v non-pharmaceutical and external v internal or none. We made no allowance for multiple comparisons. With the exception of the investigator's rating of scientific importance of the study, which we judged to be largely influenced by study results, all other factors were examined in a multivariate Cox regression to determine the relative importance of study results on time to publication adjusted for any other significant factors. Quantitative and qualitative studies were examined in the same model by also including a factor for method of assessment of study outcome. A subsidiary logistic regression analysis was performed to permit a direct comparison of the results of this study with earlier cohort studies of clinical research projects, which were analysed using odds ratios for publication.

The association between the various study characteristics and study status was examined by calculating the ² statistic for an rx c contingency table.

	Results

Top Abstract Introduction Methods Results Discussion References

Inclusion criteria
A total of 801 protocols for 810 separate studies were submitted for approval between September 1979 and December 1988. Of these, 748 separate studies in 741 submissions were approved during this period and included in the study. We excluded studies that had not been approved during this period, 19 duplicate study submissions, four submissions that were not formal studies, five studies on one patient alone, and five studies whose submission protocols or relevant data were not available.

Study characteristics
Table 1 shows the characteristics of the eligible studies and the clinical trials subgroup.

Questionnaires were completed for 520 (70%) of the eligible studies, with a larger proportion of questionnaires completed for the more recently approved studies (P=0.0007). Apart from this, the returned questionnaires were representative of the total sample of eligible studies for all factors for which data were available from the study protocols.

Table 2 shows the status of the studies at the time of questionnaire completion. The main reasons for not starting in 100 studies were lack of funding (45), lack of support from colleagues (9), departure of staff (6), and impractical methods (6). Seventy studies (of which 50 were clinical trials) were abandoned after starting, mainly because of difficulties with patient accrual (23), funding problems (6), and technological problems (6) and because the early results that had been obtained were negative (6, all clinical trials).

Association between study characteristics and completion of study
Analysis of the association between the various study characteristics and completion of the 520 studies for which questionnaires were completed showed a higher rate of not starting for non-trial experiments (31/114; 27%) than for observational studies (23/129; 18%) or for clinical trials (43/277; 16%), and a higher completion rate for observational studies (85/129; 66%) than for clinical trials (156/277; 56%) or for non-trial experiments (46/114; 40%) (P<0.0001). There was also a far higher completion rate for studies that were undertaken as part of a degree compared with those that were not (85/107; 79% v 202/396; 51%) (P<0.0001). Single centre studies were more likely never to have started or to have been abandoned than were multicentre studies (123/337; 36% v 28/163; 17%) (P<0.0001). Studies with a small sample size reached had a higher rate of abandonment (<100 (40/231; 17%) v 100 (0/141)) (P<0.0001), which may simply reflect being abandoned because of poor accrual. Clinical trials with a more rigorous study design were far more likely to be completed (randomised (87/130; 67%) v non-randomised (69/147; 47%), P=0.002; placebo controlled (31/44; 70%) v non-placebo controlled (125/233; 54%), P=0.06; and blinded (43/58; 74%) v non-blinded (113/219; 52%), P=0.004).

Study outcome and method of analysis
Of the 520 studies with completed questionnaires, 321 had had analysis undertaken with results available and were included in further analysis of the association between study outcome and time to publication.

Table 3 summarises study outcome and method of analysis. 67% of quantitative studies, had a significant result for the primary research question, whereas only 26% of qualitative studies were in the most positive category. Accordingly, quantitative and qualitative studies were considered separately in subsequent analyses.

Study characteristics and publication
Table 4 shows the hazard ratios for publication for selected variables. Factors predictive of publication were significant results; research design using non-trial experiments; a high scientific importance rating of the study by the investigator; external funding; and studies with non-comparative study groups and clinical trials that were non-randomised. Other variables tested and found not to be significant were whether the study was undertaken as part of a degree; the number of data collection sites (single v multicentre); pharmaceutical funding; the research department undertaking the study; the year of study approval; the classification of study outcome (qualitative v quantitative); and, for clinical trials, placebo control and blinding. For the 218 quantitative studies, positive studies were much more likely to be published than negative studies (hazard ratio 2.32 (95% confidence interval 1.47 to 3.66), P=0.0003) This finding was even stronger for the subgroup of 130 quantitative clinical trials (hazard ratio 3.13 (1.76 to 5.58), P=0.0001).

Figure 1 shows the survival curve plots for time to publication for quantitative studies and figure 2 those for quantitative clinical trials. The median time to publication was 4.82 (3.87 to 5.72) years for studies with significant results, 7.99 (6.91 to ) years for studies with null results, and not reached for studies with indefinite conclusions. The median time to publication was 4.69 (3.75 to 5.72) years for clinical trials with significant results and 7.99 (7.02 to ) years for clinical trials with null results.

View larger version (23K): [in this window] [in a new window]   Fig 1 Proportion of quantitative studies not published, according to type of results

View larger version (22K): [in this window] [in a new window]   Fig 2  Proportion of quantitative clinical trials not published, according to type of results

The survival curve plots for sample size (not shown) show that studies with a larger sample size (100) continued to be published right until the end of the study period, at which time about 90% of studies and clinical trials had been published, compared with only about 64% of studies and clinical trials with a sample size <100.

The odds ratio for publication quantitative studies was 2.66 (1.32 to 5.35) (P=0.003) for significant compared with null studies. For quantitative clinical trials it was 4.19 (1.71 to 10.32) (P=0.0004).

Figure 3 shows the results of multivariate analysis. For quantitative studies the adjusted hazard ratio for publication for positive compared with negative studies was 2.34 (1.47 to 3.73) (P=0.0004); for studies with intermediate results it was 0.43 (0.15 to 1.24) (P=0.12). For quantitative clinical trials the adjusted hazard ratio for publication for positive compared with negative trials was 3.29 (1.84 to 5.90) (P=0.0001); for trials with intermediate results it was 0.50 (0.14 to 1.74) (P=0.27). For quantitative studies the adjusted odds ratio for publication for positive compared with negative studies was 2.93 (1.49 to 5.74); for studies with intermediate results it was 0.34 (0.17 to 0.67). For quantitative clinical trials the adjusted odds ratio for publication for positive compared with negative trials was 4.57 (1.96 to 10.63); for trials with intermediate results it was 0.44 (0.10 to 1.91).

View larger version (35K): [in this window] [in a new window]   Fig 3 Multivariate analysis of outcome in studies with analysed results, according to type of study and results

Quantitative studies with significant study outcomes were much more likely to be submitted for publication than studies with a null study outcome (114/146; 78% v 28/52; 54%) (P=0.0009). By contrast, 130 of the 148 quantitative studies submitted had been published or were in press, and the publication rate for those studies with significant outcomes (99/114; 87%) were similar to those with null results (23/28; 82%) (P=0.54).

	Discussion

Top Abstract Introduction Methods Results Discussion References

We found publication bias, after allowing for confounding factors, in a cohort of studies that were approved by an Australian ethics committee, and our results confirm those of American and British studies.^{20 21 22} To our knowledge, ours is the first study to show the delay in publication of studies with negative results. The evidence was definite for quantitative studies, with a publication rate 2.3 times greater for studies with significant compared with null results. Within the subgroup of clinical trials, the evidence of publication bias was even stronger, with a publication rate 3.3 times greater. These figures represent a higher estimated publication rate at 5 years of 52% v 27% respectively for quantitative studies, and 54% v 23% respectively for clinical trials. Although the trend for qualitative studies was not significant (P=0.21), this may relate to the smaller number of qualitative studies. Other factors in a multivariate analysis that were associated with publication were research design (observational studies being more likely to be published than clinical trials and non-trial experiments) and source of funding (externally funded studies being more likely to be published). The impact of study results on publication was essentially the same after adjusting for these factors. The impact of other factors on publication rate was examined only for the studies with analysed results. Interestingly, clinical trials that used better study designs (randomisation, placebo control, or blinding) were not published more often than other studies but were more likely to be completed and hence more likely to be analysed than other studies. Consequently, these factors may still positively influence the publication rate when all studies, whether analysed or not, are considered.

Other studies reporting on publication bias have analysed quantitative and qualitative studies as a single group,^{20 21 22} but we considered them separately. This was because the categories were not comparable in terms of outcomes and because there was evidence of interaction between these subgroups and the degree of publication bias (P=0.02).

The validity of our study was not affected by including studies with only interim findings because the findings of publication bias were not materially altered by excluding these studies. Consequently, the bias relates to delay in publication of studies with negative results rather than just the premature publication of positive results.

Quantitative studies and clinical trials with an indefinite study outcome (0.05P<0.10) were less likely to be published than studies and clinical trials with a non-significant study outcome (P0.10). A similar trend was seen in the study of Easterbrook et al, who found an adjusted odds ratio of 0.61 (0.26 to 1.59) for non-significant trend studies compared with null studies.²⁰ Thus an indefinite study outcome may be even more likely than a definitely negative outcome to deter researchers from submitting their studies for publication, or editors from accepting them.

The data also imply that publication bias is primarily due to the failure of researchers to submit negative studies for publication rather than to the failure of journal editors to publish them after they have been submitted. However, we cannot exclude that the failure of investigators to submit negative studies for publication may partly relate to editorial bias; previous experience in the submission of negative studies may have conditioned them to expect rejection of such studies for publication.

Sample size
Berlin et al have argued that large scale studies are eventually published and that a meta-analysis restricted to large studies may be free of publication bias.²³ They also argue that small studies should be excluded from meta-analyses on the grounds that they have greater random fluctuation in their estimates and that they may be more subject to bias than large studies.²³ We found that studies with a sample size 100 continue to be published, with more than 90% published at the end of the study period. However, at any time point, since some studies are still in progress, there is always publication bias due to a delay in the publication of negative studies, provided that the results from large studies are not systemically different from other studies and that large negative studies are subject to the same delay in publication as other studies. We did not find significant interaction between study results and sample size; in addition, publication bias was still evident when analysis was restricted to the 103 quantitative studies with a sample size 100, with a hazard ratio for publication of positive compared with negative studies of 2.00 (1.09 to 3.66) (P=0.02). Hence, a strategy that excludes small trials is not a solution to the problem of publication bias.

Implications for the conduct of meta-analysis
The use of meta-analysis in medical research is becoming more common; its results seem to be precise and convincing, and it is beginning to have an impact on clinical practice and on the planning of future research. Consequently, it is important that modest differences in outcome can be validly attributed to the effect of treatment rather than bias. Although an increasing number of meta-analyses are based on controlled clinical trials, a computerised literature search for 1982-9 by Easterbrook et al²⁰ found that about two thirds of meta-analyses are based on observational studies, which are more prone to bias than controlled clinical trials.

Our results have four important implications for the conduct of meta-analysis.

Firstly, the studies used in any meta-analysis should not comprise solely published studies. This is not an unbiased sample, and meta-analysis based on published studies may result in bias in favour of positive results.

Secondly, meta-analyses should be restricted to studies that have started before a certain date. This avoids the problem of the delayed but eventual publication of studies with negative results. To include all studies, regardless of starting date, is likely to result in a selection that is biased in favour of studies with positive results.

Thirdly, retrospective collection of data is difficult—the lower questionnaire completion rate for studies that were undertaken in the early years confirms the findings of Hetherington et al.²⁴ Meta-analyses that are based on retrospective data collection are likely to be subject to significant selection bias.

Finally, restricting meta-analyses to large scale studies, even if they are all eventually published, does not by itself solve the problem of publication bias, since it does not take into account the delay in publishing negative studies.

Prospective registration—a solution to the problem of publication bias
A strategy that provides a solution to publication bias is to register prospectively all trials before their results are known and to select trials from such a registry when undertaking any systematic review.⁴ This approach has been supported by the Cochrane Collaboration and has led to the statement on the need for prospective registration of all controlled trials developed at the second annual Cochrane colloquium in 1994. For evidence based medicine, the support of government bodies to ensure that mechanisms and funding are provided to ensure universal registration is essential and can be argued to be in the interests of developing the highest level of evidence. There are also strong ethical arguments on behalf of participating patients to ensure that the results of all studies are eventually published.^{25 26} The mechanisms for the establishment of universal registration already exist—it has been mandatory since 1985 in Australia, as in many other countries, for all research projects on human subjects to be approved by an institutional ethics committee.²⁷ The identification of studies for national registration at the time of initial approval by each institutional ethics committee is therefore a fairly simple matter. The need for universal prospective registration was recognised over 10 years ago and is being actively pursued by the Cochrane Collaboration, but it is yet to be implemented. With the increasing recognition of the importance of evidence based medicine, the establishment of universal prospective registration is an important and urgent priority.

	Acknowledgements

We thank the ethics committee and investigators from the Royal Prince Alfred Hospital for their support, and Davina Ghersi for insightful comments on the manuscript.

Funding: This study was supported by a unit grant from the National Health and Medical Research Council, Australia.

Conflict of interest: None.

	References

Top Abstract Introduction Methods Results Discussion References

1. Newell DJ. Intention to treat analysis: implications for quantitative and qualitative research. Int J Epidemiol 1992;21:837-41.
2. Chalmers TC, Levin H, Sacks HS, Reitman D, Berrier J, Nagalingham R. Meta-analysis of clinical trials as a scientific discipline: control of bias and comparison with large co-operative trials. Stat Med 1987;6:315-25.
3. Counsell CE, Clarke MJ, Slattery J, Sandercock PA. The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis? BMJ 1994;309:1677-81. [Abstract/Free Full Text]
4. Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;4:1529-41.
5. Simes RJ. Prospective meta-analysis of cholesterol-lowering studies: the prospective pravastatin pooling (PPP) project and the Cholesterol Treatment Trialists (CTT) Collaboration. Am J Cardiol 1995;76:122-6C.
6. Moher D, Fortin P, Jadad AR, Juni P, Klassen T, Le Lorier J. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet 1996;347:363-6.
7. Frank E. Authors' criteria for selecting journals. JAMA 1994;272:163-4. [Abstract]
8. Gøtzche PC. Reference bias in reports of drug trials. BMJ 1987;295:654-6.
9. Sterling TD. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. J Am Stat Assoc 1959;54:30-4.
10. Juhl E, Christensen E, Tygstrup N. The epidemiology of the gastrointestinal randomised clinical trial. N Engl J Med 1977;296:20-2.
11. Begg CB, Pocock LJ, Freedman L, Zelen M. State of the art in comparative cancer clinical trials. Cancer 1987;60:2811-5.
12. Davidson RA. Source of funding and outcome of clinical trials. J Gen Intern Med 1986;155-8.
13. Gøtzsche PC. Methodology and overt and hidden bias in reports in 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. Controlled Clin Trials 1989;10:31-56.
14. Gøtzsche PC. Reference bias in reports of drug trials. BMJ 1987;295:654-6.
15. Mahoney MJ. Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cog Ther Res 1977;1:161-75.
16. Koren G, Klein N. Bias against newspaper reports of medical research. JAMA 1991;266:1824-6.
17. Kotelchuk D. Asbestos research: winning the battle but losing the war. Health/PAC Bulletin 1974;61:1-27.
18. Sommer B. The file drawer effect and publication rates in menstrual cycle research. Psychol Women Q 1987;11:233-42.
19. Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith, H. Publication bias and clinical trials. Controlled Clin Trials 1987;8:343-53.
20. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:867-72.
21. Dickersin K, Min Y-I, Meinert CL. Factors influencing publication of research results. JAMA 1992;267:374-8.
22. Dickersin K, Min Y-I. NIH clinical trials and publication bias. Online J Curr Clin Trials 1993;Document No 50.
23. Berlin JA, Begg CB, Louis TA. An assessment of publication bias using a sample of published clinical trials. J Am Stat Assoc 1989;84:381-92.
24. Hetherington J, Dickersin K, Chalmers I, Meinert CL. Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Paediatrics 1989;84:374-80.
25. Chalmers I, Gray M, Sheldon T. Handling scientific fraud: prospective registration of health care research would help. BMJ 1995;311:262. [Free Full Text]
26. Savulescu J, Chalmers I, Blunt J. Are research ethics committees behaving unethically? Some suggestions for improving performance and accountability. BMJ 1996;313:1390-3. [Free Full Text]
27. National Health and Medical Research Council. NHMRC Statement on human experimentation and supplementary notes. Canberra: NHMRC, 1985.

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review

Anders W Jørgensen, Jørgen Hilden, and Peter C Gøtzsche
BMJ 2006 333: 782. [Abstract] [Full Text] [PDF]

Fate of biomedical research protocols and publication bias in France: retrospective cohort study

Evelyne Decullier, Véronique Lhéritier, and François Chapuis
BMJ 2005 331: 19. [Abstract] [Full Text] [PDF]

Time to publication of studies was not affected by whether results were positive

Michael L Callaham, Ellen Weber, Gary Young, Robert Wears, and Chris Barton
BMJ 1998 316: 1536. [Extract] [Full Text]

This article has been cited by other articles:

• Arshad, A, Arkwright, P D (2008). Status of healthcare studies submitted to UK research ethics committees for approval in 2004-5. J. Med. Ethics 34: 393-395 [Abstract] [Full text]  
• Chan, J. K., Ueda, S. M., Sugiyama, V. E., Stave, C. D., Shin, J. Y., Monk, B. J., Sikic, B. I., Osann, K., Kapp, D. S. (2008). Analysis of Phase II Studies on Targeted Agents and Subsequent Phase III Trials: What Are the Predictors for Success?. JCO 26: 1511-1518 [Abstract] [Full text]  
• Strevel, E. L., Chau, N. G., Pond, G. R., Murgo, A. J., Ivy, P. S., Siu, L. L. (2008). Improving the Quality of Abstract Reporting for Phase I Cancer Trials. Clin. Cancer Res. 14: 1782-1787 [Abstract] [Full text]  
• Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., Rosenthal, R. (2008). Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy. NEJM 358: 252-260 [Abstract] [Full text]  
• Dartnell, J., Hemming, M., Collier, J., Ollenschlaeger, G. (2008). Putting evidence into context: some advice for guideline writers. Evid. Based Nurs. 11: 6-8 [Full text]  
• Garg, A. X., Hackam, D., Tonelli, M. (2008). Systematic Review and Meta-analysis: When One Study Is Just not Enough. CJASN 3: 253-260 [Full text]  
• Berry, A. M., Davidson, P. M., Masters, J., Rolls, K. (2007). Systematic Literature Review of Oral Hygiene Practices for Intensive Care Patients Receiving Mechanical Ventilation. Am J Crit Care 16: 552-562 [Abstract] [Full text]  
• Dartnell, J., Hemming, M., Collier, J., Ollenschlaeger, G. (2007). Putting evidence into context: some advice for guideline writers. Evid. Based Med. 12: 130-132 [Full text]  
• Lawlor, D. A (2007). Quality in epidemiological research: should we be submitting papers before we have the results and submitting more hypothesis-generating research?. Int J Epidemiol 36: 940-943 [Full text]  
• Moher, D. (2007). Reporting research results: a moral obligation for all researchers/Presenter les resultants de recherche : une obligation morale pour tous les chercheurs. Canadian J. Anesthesia 54: 331-335 [Full text]  
• Hall, R., de Antueno, C., Webber, A. (2007). Publication bias in the medical literature: a review by a Canadian Research Ethics Board: [Biais de publication dans la litterature medicale : un compte rendu d'un Comite d'ethique de recherche canadien]. Canadian J. Anesthesia 54: 380-388 [Abstract] [Full text]  
• Lynch, J. R., Cunningham, M. R.A., Warme, W. J., Schaad, D. C., Wolf, F. M., Leopold, S. S. (2007). Commercially Funded and United States-Based Research Is More Likely to Be Published; Good-Quality Studies with Negative Outcomes Are Not. JBJS 89: 1010-1018 [Abstract] [Full text]  
• Ioannidis, J. P.A., Trikalinos, T. A. (2007). The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey. CMAJ 176: 1091-1096 [Abstract] [Full text]  
• Decullier, E., Chapuis, F. (2007). Oral presentation bias: a retrospective cohort study. J. Epidemiol. Community Health 61: 190-193 [Abstract] [Full text]  
• Allison, G. T. (2007). Statistical and Clinical Interpretation of Research Results. J. Am. Podiatr. Med. Assoc. 97: 165-170 [Abstract] [Full text]  
• Bero, L A (2006). Clinical trial registration at Tobacco Control. Tobacco Control 15: 417-418 [Full text]  
• Jorgensen, A. W, Hilden, J., Gotzsche, P. C (2006). Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ 333: 782- [Abstract] [Full text]  
• Liebeskind, D. S., Kidwell, C. S., Sayre, J. W., Saver, J. L. (2006). Evidence of publication bias in reporting acute stroke clinical trials.. Neurology 67: 973-979 [Abstract] [Full text]  
• Falagas, M. E., Rosmarakis, E. S. (2006). Clinical decision-making based on findings presented in conference abstracts: is it safe for our patients?. Eur Heart J 27: 2038-2039 [Abstract] [Full text]  
• Oliveira, G. J., Leles, C. R. (2006). Critical appraisal and positive outcome bias in case reports published in brazilian dental journals.. J Dent Educ 70: 869-874 [Abstract] [Full text]  
• Gluud, L. L. (2006). Bias in Clinical Intervention Research. Am J Epidemiol 163: 493-501 [Abstract] [Full text]  
• Gold, J. L., Studdert, D. M. (2005). Clinical Trials Registries: A Reform that is Past Due. J Law Med Ethics 33: 811-820  
• Littner, Y., Mimouni, F. B., Dollberg, S., Mandel, D. (2005). Negative Results and Impact Factor: A Lesson From Neonatology. Arch Pediatr Adolesc Med 159: 1036-1037 [Abstract] [Full text]  
• Williamson, P R, Gamble, C, Altman, D G, Hutton, J L (2005). Outcome selection bias in meta-analysis. Stat Methods Med Res 14: 515-524 [Abstract]  
• Metcalfe, A. (2005). Universal precautions: A review of knowledge, compliance and strategies to improve practice. Journal of Research in Nursing 10: 549-550  
• Ioannidis, J. P. A. (2005). Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA 294: 218-228 [Abstract] [Full text]  
• Decullier, E., Lheritier, V., Chapuis, F. (2005). Fate of biomedical research protocols and publication bias in France: retrospective cohort study. BMJ 331: 19- [Abstract] [Full text]  
• Hartling, L., McAlister, F. A., Rowe, B. H., Ezekowitz, J., Friesen, C., Klassen, T. P. (2005). Challenges in Systematic Reviews of Therapeutic Devices and Procedures. ANN INTERN MED 142: 1100-1111 [Abstract] [Full text]  
• Cure, S., Chua, W. L., Duggan, L., Adams, C. (2005). Randomised controlled trials relevant to aggressive and violent people, 1955-2000: a survey. Br. J. Psychiatry 186: 185-189 [Abstract] [Full text]  
• Hartling, L., Craig, W. R., Russell, K., Stevens, K., Klassen, T. P. (2004). Factors Influencing the Publication of Randomized Controlled Trials in Child Health Research. Arch Pediatr Adolesc Med 158: 983-987 [Abstract] [Full text]  
• Dowd, M. D. (2004). Breaching the Contract: The Ethics of Nonpublication of Research Studies. Arch Pediatr Adolesc Med 158: 1014-1015 [Full text]  
• Maher, C. G, Sherrington, C., Elkins, M., Herbert, R. D, Moseley, A. M (2004). Challenges for Evidence-Based Physical Therapy: Accessing and Interpreting High-Quality Evidence on Therapy. ptjournal 84: 644-654 [Abstract] [Full text]  
• Landon, B. E., Wilson, I. B., McInnes, K., Landrum, M. B., Hirschhorn, L., Marsden, P. V., Gustafson, D., Cleary, P. D. (2004). Effects of a Quality Improvement Collaborative on the Outcome of Care of Patients with HIV Infection: The EQHIV Study. ANN INTERN MED 140: 887-896 [Abstract] [Full text]  
• Krzyzanowska, M. K., Pintilie, M., Brezden-Masley, C., Dent, R., Tannock, I. F. (2004). Quality of Abstracts Describing Randomized Trials in the Proceedings of American Society of Clinical Oncology Meetings: Guidelines for Improved Reporting. JCO 22: 1993-1999 [Abstract] [Full text]  
• Hirsch, L. (2004). Randomized clinical trials: What gets published, and when?. CMAJ 170: 481-483 [Full text]  
• Krzyzanowska, M. K., Pintilie, M., Tannock, I. F. (2003). Factors Associated With Failure to Publish Large Randomized Trials Presented at an Oncology Meeting. JAMA 290: 495-501 [Abstract] [Full text]  
• Lexchin, J., Bero, L. A, Djulbegovic, B., Clark, O. (2003). Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 326: 1167-1170 [Abstract] [Full text]  
• Melander, H., Ahlqvist-Rastad, J., Meijer, G., Beermann, B. (2003). Evidence b(i)ased medicine--selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ 326: 1171-1173 [Abstract] [Full text]  
• Baird, P. (2003). Getting it right: industry sponsorship and medical research. CMAJ 168: 1267-1269 [Full text]  
• Mabry, I. R., Richmond, T., Bialostozky, A., Rushton, J. (2003). Interpreting Negative Results: Postpartum Interview Position Not Associated With Improved Outcomes. Arch Pediatr Adolesc Med 157: 333-335 [Full text]  
• de Camargo, K. R. Jr (2002). The Thought Style of Physicians: Strategies for Keeping up with Medical Knowledge. Social Studies of Science 32: 827-855 [Abstract]  
• Wyatt, J, Guly, H (2002). Identifying the research question and planning the project. Emerg. Med. J. 19: 318-321 [Abstract] [Full text]  
• Slimmon, D., Bennell, K., Brukner, P., Crossley, K., Bell, S. N. (2002). Long-Term Outcome of Fasciotomy with Partial Fasciectomy for Chronic Exertional Compartment Syndrome of the Lower Leg. Am J Sports Med 30: 581-588 [Abstract] [Full text]  
• Field, T. S., Hill, M. D., Connor, M. D. (2002). Weather, Chinook, and Stroke Occurrence * Editorial Comment. Stroke 33: 1751-1758 [Abstract] [Full text]  
• Olson, C. M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J. W., Zhu, Q., Reiling, J., Pace, B. (2002). Publication Bias in Editorial Decision Making. JAMA 287: 2825-2828 [Abstract] [Full text]  
• Dickersin, K., Olson, C. M., Rennie, D., Cook, D., Flanagin, A., Zhu, Q., Reiling, J., Pace, B. (2002). Association Between Time Interval to Publication and Statistical Significance. JAMA 287: 2829-2831 [Abstract] [Full text]  
• Moher, D., Schachter, H. M. (2002). Potential solutions to the problem of conducting systematic reviews of new health technologies. CMAJ 166: 1674-1675 [Full text]  
• Klassen, T. P., Wiebe, N., Russell, K., Stevens, K., Hartling, L., Craig, W. R., Moher, D. (2002). Abstracts of Randomized Controlled Trials Presented at the Society for Pediatric Research Meeting: An Example of Publication Bias. Arch Pediatr Adolesc Med 156: 474-479 [Abstract] [Full text]  
• Stewart, L. A., Tierney, J. F. (2002). To IPD or not to IPD?: Advantages and Disadvantages of Systematic Reviews Using Individual Patient Data. Eval Health Prof 25: 76-97 [Abstract]  
• Dickersin, K. (2002). Systematic reviews in epidemiology: why are we so far behind?. Int J Epidemiol 31: 6-12 [Full text]  
• Yaphe, J., Edman, R., Knishkowy, B., Herman, J. (2001). The association between funding by commercial interests and study outcome in randomized controlled drug trials. Fam Pract 18: 565-568 [Abstract] [Full text]  
• McNeil, B. J. (2001). Hidden Barriers to Improvement in the Quality of Care. NEJM 345: 1612-1620 [Full text]  
• Haidich, A.-B., Ioannidis, J. P. A. (2001). Effect of Early Patient Enrollment on the Time to Completion and Publication of Randomized Controlled Trials. Am J Epidemiol 154: 873-880 [Abstract] [Full text]  
• McQueen, M. J. (2001). Overview of Evidence-based Medicine: Challenges for Evidence-based Laboratory Medicine. Clin. Chem. 47: 1536-1546 [Abstract] [Full text]  
• Sterne, J. A C, Egger, M., Smith, G. D. (2001). Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis. BMJ 323: 101-105 [Full text]  
• Evans, A., Kalra, L. (2001). Are the Results of Randomized Controlled Trials on Anticoagulation in Patients With Atrial Fibrillation Generalizable to Clinical Practice?. Arch Intern Med 161: 1443-1447 [Abstract] [Full text]  
• Sutton, A. J, Song, F., Gilbody, S. M, Abrams, K. R (2000). Modelling publication bias in meta-analysis: a review. Stat Methods Med Res 9: 421-445 [Abstract]  
• Loewen, P. S., Marra, C. A., Zed, P. J. (2000). 5-HT3 receptor antagonists vs traditional agents for the prophylaxis of postoperative nausea and vomiting. Canadian J. Anesthesia 47: 1008-1018 [Abstract]  
• Gøtzsche, P. C (2000). Why we need a broad perspective on meta-analysis. BMJ 321: 585-586 [Full text]  
• Shields, P. G. (2000). Publication Bias Is a Scientific Problem with Adverse Ethical Outcomes: The Case for a Section for Null Results. Cancer Epidemiol. Biomarkers Prev. 9: 771-772 [Full text]  
• WEST, R R (2000). Evidence based medicine overviews, bulletins, guidelines, and the new consensus. Postgrad. Med. J. 76: 383-389 [Full text]  
• Schwid, S. R., Trotter, J. L. (2000). Lessons from linomide: A failed trial, but not a failure. Neurology 54: 1716-1717 [Full text]  
• Coleman, B. D., Khan, K. M., Kiss, Z. S., Bartlett, J., Young, D. A., Wark, J. D. (2000). Open and Arthroscopic Patellar Tenotomy for Chronic Patellar Tendinopathy: A Retrospective Outcome Study. Am J Sports Med 28: 183-190 [Abstract] [Full text]  
• Mariani, L., Marubini, E. (2000). Content and Quality of Currently Published Phase II Cancer Trials. JCO 18: 429-429 [Abstract] [Full text]  
• Tonks, A. (1999). Registering clinical trials. BMJ 319: 1565-1568 [Full text]  
• Johansen, H. K., Gotzsche, P. C. (1999). Problems in the Design and Reporting of Trials of Antifungal Agents Encountered During Meta-analysis. JAMA 282: 1752-1759 [Abstract] [Full text]  
• Jaakkimainen, R L., Boyle, E., Tudiver, F. (1999). Is Helicobacter pylori associated with non-ulcer dyspepsia and will eradication improve symptoms? A meta-analysis. BMJ 319: 1040-1044 [Abstract] [Full text]  
• Whatling, P. (1999). Having non-medical readers of papers on internet will enhance peer review. BMJ 318: 1144-1144 [Full text]  
• Sykes, R. (1998). Being a modern pharmaceutical company. BMJ 317: 1172-1180 [Full text]  
• Kunz, R., Oxman, A. D (1998). The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 317: 1185-1190 [Abstract] [Full text]  
• Plotnick, L. H, Ducharme, F. M (1998). Should inhaled anticholinergics be added to beta 2 agonists for treating acute childhood and adolescent asthma? A systematic review. BMJ 317: 971-977 [Abstract] [Full text]  
• Rush, J. (1998). Cleaning the umbilical cord with water rather than alcohol shortened the time to separation with no change in colonisation. Evid. Based Nurs. 1: 112-112 [Full text]  
• Rush, J. (1998). Review: topical agents for cord care have not been shown to be effective in newborn infants in developed countries. Evid. Based Nurs. 1: 113-113 [Full text]  
• Misakian, A. L., Bero, L. A. (1998). Publication Bias and Research on Passive Smoking: Comparison of Published and Unpublished Studies. JAMA 280: 250-253 [Abstract] [Full text]  
• McKibbon, K A., Marks, S. (1998). Searching for the best evidence. Part 1: where to look. Evid. Based Nurs. 1: 68-70 [Full text]  
• Clarke, M., Stewart, L., Arida, A. U., Ioannidis, J. P. A., Jadad, A. R., Rennie, D. (1998). Time Lag Bias in Publishing Clinical Trials. JAMA 279: 1952-1953 [Full text]  
• Gøtzsche, P. C, Johansen, H. K. (1998). Meta-analysis of short term low dose prednisolone versus placebo and non-steroidal anti-inflammatory drugs in rheumatoid arthritis. BMJ 316: 811-818 [Abstract] [Full text]  
• Stuck, A. E, Rubenstein, L. Z, Wieland, D., Vandenbroucke, J. P, Irwig, L., Macaskill, P., Berry, G., Glasziou, P., Seagroatt, V., Stratton, I., Egger, M., Smith, G. D., Minder, C., Langhorne, P., Song, F., Gilbody, S. (1998). Bias in meta-analysis detected by a simple, graphical test. BMJ 316: 469-469 [Full text]  
• Jadad, A. R., Rennie, D. (1998). The Randomized Controlled Trial Gets a Middle-aged Checkup. JAMA 279: 319-320 [Full text]  
• Julian, D., Le Lorier, J., Grégoire, G. (1998). Meta-analysis and the meta-epidemiology of clinical research. BMJ 316: 311b-311 [Full text]  
• Egger, M., Smith, G. D. (1998). meta-analysis bias in location and selection of studies. BMJ 316: 61-66 [Full text]  
• Naylor, C D. (1997). Meta-analysis and the meta-epidemiology of clinical research. BMJ 315: 617-619 [Full text]