BMJ  2005;331:19 (2 July), doi:10.1136/bmj.38488.385995.8F (published 20 June 2005)

Paper

Fate of biomedical research protocols and publication bias in France: retrospective cohort study

¹ Clinical Research Unit, DIM des Hospices Civils de Lyon, 162 avenue Lacassagne, 69424 Lyon cedex 03, France, ² CCPPRB Lyon B - Hôpital Hotel-Dieu, place de l'Hopital, 69002 Lyon, ³ French National Confederation of Research Ethics Committees - Hôpital Hotel-Dieu, 69002 Lyon

Abstract

Objectives To describe the fate of protocols approved by the French research ethics committees, a national system created by the French 1988 Huriet-Sérusclat Act; to assess publication bias at a national level.

Design Retrospective cohort study.

Setting Representative sample of 25/48 French research ethics committees in 1994.

Protocols 649 research protocols approved by committees, with follow-up information.

Main outcome measures Protocols' initial characteristics (design, study size, investigator) abstracted from committees' archives; follow-up information (rates of initiation, completion, and publication) obtained from mailed questionnaire to principal investigators.

Results Completed questionnaires were available for 649/976 (69%) protocols. Of these, 581 (90%) studies were initiated, 501/581 (86%) were completed, and 190/501 (38%) were published. Studies with confirmatory results were more likely to be published as scientific papers than were studies with inconclusive results (adjusted odds ratio 4.59, 95% confidence interval 2.21 to 9.54). Moreover, studies with confirmatory results were published more quickly than studies with inconclusive results (hazard ratio 2.48, 1.36 to 4.55).

Conclusion At a national level, too many research studies are not completed, and among those completed too many are not published. We suggest capitalising on research ethics committees to register and follow all authorised research on human participants on a systematic and prospective basis.

Introduction

Biomedical research protocols, once approved by a research ethics committee, have different fates. Some have a linear course—approval, initiation, completion, and publication, whereas others may fail at any step. Publication bias—defined as the tendency to favour publication of research with confirmatory results over research with inconclusive or invalidating results—threatens the reliability of reviews of published literature.¹

Four papers reporting on follow-up of protocols approved by research ethics committees found that 79-93% of approved protocols were initiated and 64-74% of the initiated studies proceeded to completion.^2-5 Three of these papers also showed that confirmatory results are associated with publication.^3-5 The main reason for non-publication was investigators considering their results not interesting. A survey in 1973 showed that in the case of statistically non-significant results, the probability of submission was only 6%.⁶

In France, a national system of 48 research ethics committees was created in 1988. Every research protocol involving humans in France must be approved by a committee. This information has not previously been used for research purposes. Our objective was to describe the fate of clinical protocols after approval and to assess publication bias.

Methods

We surveyed a sample of 25/48 (54%) committees, randomly chosen to ensure a geographical cross section representative of administrative areas. All invited committees agreed to participate. We assessed three main outcomes: study initiation, study completion, and publication as a scientific paper. Our hypothesis was that studies with confirmatory results were more likely to be published than those with either inconclusive or invalidating results. All protocols newly approved between 1 January 1994 and 31 December 1994 by any of the 25 participating French committees were eligible. We collected data either from the committee files or from postal questionnaires. We classified study results as "confirmatory," "invalidating," or "inconclusive." When the investigator did not respond to questions about publication status, we considered this as missing data. We classified as "confidential" protocols describing research that the investigator reported was not intended to be published.

Data collection
Each protocol was anonymised and given an identification number, and completed forms were sent to the coordinating centre. In the case of non-response, a research assistant contacted principal investigators up to six times by mail or phone, and the local committee then contacted them. Complete non-responders were classified as refusal, investigator retired, deceased, or moved away.

Statistical methods
We obtained frequency distributions for all variables and assessed association as needed. We used logistic regression to build explicative models for the three outcomes (initiation, completion, and publication). We restricted analysis of publication to completed studies. We excluded studies when results were not known by the investigator and when not aimed at publication (confidential results or phase I studies). We calculated the time between the date of approval by the committee and the date of first publication and did a survival analysis. See bmj.com for a full description of statistical analysis.

Results

In 1994 the 25 committees evaluated a total of 1143 protocols. After exclusions, 649 approved protocols were included in the analysis.

Study of non-responses
We obtained data for 185/305 (61%) of non-respondents. The reasons were refusal to fill in a follow-up form (n = 74, 40%), unable to find the original file (n = 56, 30%), and investigator not located because he or she had moved (n = 42, 23%) or had retired or died and nobody could locate the protocol archives (n = 13, 7%).

View larger version (41K): [in this window] [in a new window]   Fig 1 Fate of biomedical research protocols

 
Protocols with missing follow-up data (n = 305) did not differ from the included protocols (n = 649) by either type of sponsor or study design, but they more often needed modifications to gain approval (relative risk 1.25, 95% confidence interval 1.01 to 1.55) and were more often multicentre (2.04, 1.66 to 2.50) and international (1.45, 1.18 to 1.78).

Characteristics of approved protocols
The most common characteristics of the 649 approved protocols were drug testing topic (68%), private funding (73%), and conducted nationally only (82%). Experimental designs were most frequent, and 62% of them were randomised. Planned study size was less than 20 patients in 34% of studies, and expected duration of study was less than 18 months in 56% of the studies.

Fate of approved protocols
Figure 1 shows the fate of biomedical research protocols for the three study outcomes. Protocols not initiated tended more often to be national (91% v 82%), to be testing medical devices (9% v 5%), and to have no funding (21% v 8%).

Initiation of protocols
Phase I protocols, protocols with mixed funding, and multinational ones were more likely to begin (table 1). Among the 68 (10%) protocols that were not initiated, reasons given were refusal of the legal sponsor (n = 21, 31%), problems with recruitment of patients (n = 15, 22%), technical aspects and feasibility (n = 9, 13%), absence of funding (n = 8, 12%), decision of the investigator (n = 8, 12%), and a similar study having been published (n = 2, 3%). No reason was given for five studies (7%).

View this table: [in this window] [in a new window]   Table 1 Multivariate analysis of study initiation, completion, and publication*

 

Completion of studies
Among the 581 protocols initiated, 16 were ongoing. Phase I studies and studies without adverse effects were more likely to be completed (table 1). Reasons for stopping studies before their planned completion included patient recruitment problems (n = 28, 44%), results found in the interim analysis (n = 13, 20%), incidence of adverse effects (n = 8, 12%), sponsor's decision (n = 8, 13%), and other (n = 7, 11%).

Publication of results
Results were published in a scientific paper for 190/501 (38%) of completed studies, for 7/16 (44%) of ongoing studies, and for 8/64 (12%) of stopped studies. Among stopped studies, publication rates varied from 0% for studies with recruitment difficulties to 3/8 (37%) for studies with adverse events. Among the 501 completed studies, the publication rate was also heterogeneous; it was lower for the subgroup of phase I studies—21/127 (17%) compared with 169/374 (45%) for others.

Publication bias
Among the 501 completed studies, 248 were included in the analysis of publication bias. Four variables remained in the final model (table 1): direction of results, international versus national scope of the study, study design, and presence of an interim analysis. The stepwise regression confirmed the existence of publication bias; studies with confirmatory results were significantly more likely to be published (odds ratio 4.59, 95% confidence interval 2.21 to 9.54).

Investigators' reasons for non-publication
The main reason for non-publication given by the investigator was invalidating results. Some studies had manuscripts still in the writing or submission stage. Rejection of manuscript was cited for only 5%. The reasons given by the investigator for non-publication corroborate the logistic regression results (confirmatory results were the strongest predictor of publication) (see bmj.com).

View larger version (36K): [in this window] [in a new window]   Fig 2 Time elapsed to publication

 
Delayed publication of invalidating results
We estimated the effect of direction of results on time to publication for the 248 completed studies. Mean time to publication was significantly associated with direction of results (P < 0.001; fig 2): 5.2 years (n = 139, 95% confidence interval 4.8 to 5.6) for confirmatory results compared with 6.9 years (n = 13, 5.9 to 7.9) for invalidating results, and 6.5 years (n = 35, 5.8 to 7.2) for inconclusive results. Cox univariate analysis yielded hazard ratios of 2.48 (1.36 to 4.55) for confirmatory results versus inconclusive results and 0.64 (0.18 to 2.27) for invalidating results versus inconclusive results.

Dissemination of results
Among the 248 protocols used for the analysis of publication bias, 146 (59%) led to scientific papers (table 2). Most studies (92%) with multiple publications had confirmatory results. However, the association between multiple publication and direction of results was not significant.

View this table: [in this window] [in a new window]   Table 2 Reporting of 248 completed studies

 

Fifty five per cent of the studies reported in scientific papers were also presented orally. The 102 remaining studies were not published as scientific papers.

Discussion

Only 38% of completed studies were published. We found evidence for publication bias, favouring publication of confirmatory results. Most (90%) approved protocols were initiated, and 86% of these were completed. Such information has been unavailable until now for interventional research conducted on humans. Previous studies used similar methods to ours but focused on one or two local committees, whereas we collected data on a sample of half the committees over a whole country. This may explain our lower publication rate. Moreover, we added information on major steps in a protocol's life—from approval and initiation to completion and publication. Phase I studies were more likely to be initiated and completed than were others, probably because they are shorter and smaller.

What is already known on this topic Three observational studies have shown evidence of publication bias in biomedical research approved by research ethics committees, but all were done at a local level What this study adds The fate of biomedical research, from acceptance to publication, has been shown at a national level Publication bias has been confirmed; confirmatory results were 4.59 times more likely to be published than inconclusive results

Publication bias
The estimated odds ratio for the association between results and publication in our study was similar to, although higher than, those found in the other studies (range 2.32-2.93). This may be because our study population included 22% descriptive non-experimental protocols, which may be easier to do and more likely to be published. We also excluded the stopped studies and those considered to be confidential, which are less likely to be published. Investigators' decisions to declare a study as confidential were not linked to invalidating or inconclusive results. The leading reason declared for failure to publish was that the investigator did not find the results interesting (26%). This is similar to other studies (range 27-43%).^{4 7 8} Only 5% of studies were not published because of rejection by a journal, similar to other studies (range 5-10%).

A recent report showed that investigators were more likely to report statistically significant outcomes and failed to report others (outcome reporting bias).⁹ Reasons given were similar to those explaining non-publication: 30% were not reported owing to the lack of statistical significance.

Non-response
The major limitation of our study was the non-response rate (31%), similar to those of other studies (range 22-30%).^{4 5 8} Characteristics of protocols lost to follow-up were similar to those never initiated (multicentre and international studies). Non-response may thus be associated with never initiated protocols.

Registering trials
We strongly support prospective registration of protocols—proposed in 1986 and supported by many authors.^{1 10 11} In 2004 the International Committee of Medical Journal Editors decided to require prior recording in a protocol registry.^{12 13}

We propose capitalising on the work done by research ethics committees worldwide. Moreover, the European EC/2001-20 guideline tends towards standardising clinical trial files and procedures across Europe.¹⁴

---

This is the abridged version of an article that was posted on bmj.com on 20 June 2005: http://bmj.com/cgi/doi/10.1136/bmj.38488.385995.8F

We thank Kay Dickersin and Hervé Maisonneuve for their advice on this paper and Yves Matillon and Christian Hervé for their advice on drafting the protocol. We also thank Marie-Pierre Rochette and Françoise Leclet, administrative staff of research ethics committee of Lyon; Patricia Darnand and William Banga, members of the logistic staff; and the chairpersons and members of the participating French research ethics committees: Alsace, Aulnay, Auvergne, Bordeaux, Boulogne, Brest, Dijon, Loire, Lyon A, Lyon B, Lyon C, Marseille1, Marseille2, Montpellier, Nice, Normandie, Paris-Bicetre, Paris-Creteil, Paris-Hotel dieu, Paris-Necker, Paris-Versailles, Poitou, Toulouse1, Toulouse2, and Tours.

Contributors: See bmj.com

Funding: French Ministry of Health (Programme Hospitalier de Recherche Clinique 1998-065) and Hospices Civils de Lyon. French Ministry of Research and Higher Education and Claude Bernard University. Neither of the funding sources was involved at any stage.

Competing interests: None declared.

References

1. Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;4: 1529-41.[Abstract/Free Full Text]
2. Pich J, Carne X, Arnaiz JA, Gomez B, Trilla A, Rodes J. Role of a research ethics committee in follow-up and publication of results. Lancet 2003;361: 1015-6.[CrossRef][Web of Science][Medline]
3. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337: 867-72.[CrossRef][Web of Science][Medline]
4. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315: 640-5.[Abstract/Free Full Text]
5. Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA 1992;267: 374-8.[Abstract/Free Full Text]
6. Greenwald AG. Consequences of prejudice against the null hypothesis. Psychol Bull 1975;82: 1-20.
7. Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279: 281-6.[Abstract/Free Full Text]
8. Easterbrook PJ, Matthews DR. Fate of research studies. J R Soc Med 1992;85: 71-6.[Abstract]
9. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291: 2457-65.[Abstract/Free Full Text]
10. Dickersin K. Why register clinical trials?—Revisited. Control Clin Trials 1992;13: 170-7.[CrossRef][Web of Science][Medline]
11. Easterbrook P. Reducing publication bias. BMJ (Clin Res Ed) 1987;295: 1347.[Free Full Text]
12. Abbasi K. Compulsory registration of clinical trials. BMJ 2004;329: 637-8.[Free Full Text]
13. DeAngelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Med J Aust 2004;181: 293-4.[Web of Science][Medline]
14. Directive 2001/20/EC of the European Parliament and of the Council of 4 April 2001 on the approximation of the laws, regulations and administrative provisions of the member states relating to the implementation of good clinical practice in the conduct of clinical trials on medicinal products for human use. Official journal L121, 01/05/2001, 2001: 34-44.

(Accepted 9 May 2005)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Relevant Articles

Waiving confidentiality for the greater good

An-Wen Chan, Ross Upshur, Jerome A Singh, Davina Ghersi, François Chapuis, and Douglas G Altman
BMJ 2006 332: 1086-1089. [Full Text] [PDF]

Dissemination of results needs to be tracked as well as the funding is

James McCormack, Peter Loewen, and Peter Jewesson
BMJ 2005 331: 456. [Extract] [Full Text]

French ethics committees should tackle publication bias
BMJ 2005 331: 0. [Full Text]

Compulsory registration of clinical trials

Kamran Abbasi
BMJ 2004 329: 637-638. [Extract] [Full Text] [PDF]

Time to register randomised trials

Richard Horton and Richard Smith
BMJ 1999 319: 865-866. [Extract] [Full Text] [PDF]

Publication bias: evidence of delayed publication in a cohort study of clinical research projects

Jerome M Stern and R John Simes
BMJ 1997 315: 640-645. [Abstract] [Full Text]

This article has been cited by other articles:

• Blumle, A, Antes, G, Schumacher, M, Just, H, von Elm, E (2008). Clinical research projects at a German medical faculty: follow-up from ethical approval to publication and citation by others. J. Med. Ethics 34: e20-e20 [Abstract] [Full text]  
• Arshad, A, Arkwright, P D (2008). Status of healthcare studies submitted to UK research ethics committees for approval in 2004-5. J. Med. Ethics 34: 393-395 [Abstract] [Full text]  
• Decullier, E., Chapuis, F. (2007). Oral presentation bias: a retrospective cohort study. J. Epidemiol. Community Health 61: 190-193 [Abstract] [Full text]  
• Chan, A.-W., Upshur, R., Singh, J. A, Ghersi, D., Chapuis, F., Altman, D. G (2006). Waiving confidentiality for the greater good.. BMJ 332: 1086-1089 [Full text]  
• Delgado-Rodriguez, M. (2006). Systematic reviews of meta-analyses: applications and limitations. J. Epidemiol. Community Health 60: 90-92 [Full text]  
• McCormack, J., Loewen, P., Jewesson, P. (2005). Dissemination of results needs to be tracked as well as the funding is. BMJ 331: 456-456 [Full text]  

Rapid Responses:

Read all Rapid Responses

Ethics committees must require trial registration

Andrew Jull
bmj.com, 4 Jul 2005 [Full text]

Publication bias, selective dissemination and trial registries

James P. McCormack, et al.
bmj.com, 15 Jul 2005 [Full text]

Online poll

Find out more >>