BMJ  2004;328:22-24 (3 January), doi:10.1136/bmj.328.7430.22

Paper

Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group

¹ Department of Interdisciplinary Oncology, H Lee Moffitt Cancer Center and Research Institute, University of South Florida, 12902 Magnolia Drive, Tampa, FL 33612, USA, ² H Lee Moffitt Cancer Center and Research Institute, ³ UK Cochrane Centre, Oxford OX2 7LG, ⁴ Statistical Unit, Radiation Therapy Oncology Group, PA 19107, USA

Abstract

Objective To determine whether poor reporting of methods in randomised controlled trials reflects on poor methods.

Design Observational study.

Setting Reports of randomised controlled trials conducted by the Radiation Therapy Oncology Group since its establishment in 1968.

Participants The Radiation Therapy Oncology Group.

Outcome measures Content of reports compared with the design features described in the protocols for all randomised controlled trials.

Results The methodological quality of 56 randomised controlled trials was better than reported. Adequate allocation concealment was achieved in all trials but reported in only 42% of papers. An intention to treat analysis was done in 83% of trials but reported in only 69% of papers. The sample size calculation was performed in 76% of the studies, but reported in only 16% of papers. End points were clearly defined and and errors were prespecified in 76% and 74% of the trials, respectively, but only reported in 10% of the papers. The one exception was the description of drop outs, where the frequency of reporting was similar to that contained in the original statistical files of the Radiation Therapy Oncology Group.

Conclusions The reporting of methodological aspects of randomised controlled trials does not necessarily reflect the conduct of the trial. Reviewing research protocols and contacting trialists for more information may improve quality assessment.

Introduction

Evaluation of the quality of published clinical research is central to informed decision making. Information on trial quality is particularly important during peer review or when results from individual studies are evaluated in systematic reviews or meta-analyses.^{1 2} The quality of research should always be considered when a report is used in decision making in health care. Poorly conducted and reported research seriously compromises the integrity of the research process, especially if biased results receive false credibility.³

Many efforts have been made to improve the quality of studies and their related publications. The best example was the publication of the Consolidated Standards of Reporting of Trials (CONSORT) statement to improve the quality of trial reports.³ Such efforts to improve the quality of clinical research, however, imply that if certain design or methodological features are not reported then they were not done. Ideally, assessment of the quality of clinical research should not only address reporting but also the original design and intended plan for its conduct and analysis as specified in the trial's research protocol. The importance of linking the final report of clinical trials with their original research protocols led some authors to argue that no randomised controlled trial should be conducted without publication of its research protocol.⁴ The reasons behind this are that critical comments may be encouraged leading to improvements in trial design, publication can be coupled with trial registration, the original protocol can be compared with what was subsequently done, and investigators can more easily appreciate what research is being conducted in their areas of interest.⁴ More importantly, publication of research protocols is one of the best ways to minimise bias by explicitly stating a priori hypotheses and methods without the prior knowledge of results.⁵ Many randomised controlled trials are preceded by the preparation of a written protocol, which describes the conduct of the trial more comprehensively than is possible in many journal articles, and making these protocols available would provide much useful additional information. We aimed to test the assumption that poor reporting reflects poor methods by comparing research protocols with the information published in the final reports of a set of randomised controlled trials.

Methods

We studied randomised controlled trials conducted by the Radiation Therapy Oncology Group. This is a national clinical cooperative group with a focus on the development of studies to improve survival and the quality of life of patients with cancer. It was established in 1968 and is publicly funded by the National Cancer Institute in the United States. The group consists of both clinical and laboratory investigators from over 260 institutions across the United States and Canada, and its membership includes nearly 90% of all comprehensive and clinical cancer centres designated by the National Cancer Institute.⁶ Before activation, the group's research protocols must pass through a rigorous peer review process and be reviewed and approved through its own committee system and the National Cancer Institute. Development of a protocol consists of six phases (box).⁶

Our analysis included data related to primary outcomes from all terminated phase III trials conducted by the Radiation Therapy Oncology Group since its establishment in 1968. We extracted data on methodological domains that have been acknowledged as vital for minimising bias in the conduct and analysis of randomised controlled trials.⁷ The effect of chance is usually minimised by appropriate planning of the trial's size, through a statistical power analysis using estimates for the expected differences between the interventions and prespecified type I () and type II () error levels.⁷ To investigate systematic bias we extracted data on the quality of the randomisation process (selection bias) and drop outs (attrition bias).⁸ Since the primary outcome was survival in most of the studies, we did not consider quality related to observer bias, such as the use of placebo or independent reading of outcomes (there were only three placebo controlled trials). We extracted data from all papers and protocols. The accuracy of this data was verified by the group's statistical centre.

Results

Overall, there were 59 terminated phase III randomised controlled trials, three of which had not been published. We found 58 published papers for the remaining 56 protocols for use in our study. The figure summarises the results according to information from the papers, protocols, and the Radiation Therapy Oncology Group's statistical office. This shows that the reporting of methods in the publications does not necessarily reflect the methodological quality of the associated protocols. For example, a priori sample size calculations were performed in 44 (76%) trials, but this information was given in only nine of the 58 published papers (16%). Although all trials had adequate allocation concealment (through central randomisation), this was reported in only 24 (41%) of the papers. From our initial data extraction, we found that 40 (69%) of these trials used an intention to treat analysis. This number was increased to 48 (83%) after verification by the Radiation Therapy Oncology Group. End points were clearly defined, and and errors were prespecified in 44 (76%) and 43 (74%) trials, respectively, but only reported in six (10%) of the papers. Interestingly, reporting of drop outs was meticulous; we found no difference in frequency (91%) between data presented in the papers and those in the original files.

View larger version (44K): [in this window] [in a new window]   Quality of reporting compared with actual methodological quality of 56 randomised controlled trials (58 reports) conducted by the Radiation Therapy Oncology Group, based on information from published reports, protocols, and verification by the group. Absolute numbers of reports are shown

 

Discussion

Poor reporting of randomised controlled trials may not indicate poor quality of the trials themselves. We are aware of two other studies that reported empirical assessments of this relation. One study evaluated the quality of 63 randomised controlled trials of breast cancer treatment. Data were extracted from publications related to these trials and the results compared with the information provided by the principal investigators. The study concluded that faulty reporting reflected faulty methods.⁹ Another study, however, concluded that even well designed and conducted trials may be badly reported.² This conclusion was drawn indirectly from an assessment of three key indicators of quality: adequate allocation concealment, appropriate blinding, and use of intention to treat analysis.² Unlike our study, neither of these studies reported a comparison of the quality of reporting with the methods specified in the original research protocols.

Phases in development of a trial protocol by Radiation Therapy Oncology Group Approval of concept Review and approval of protocol among group members Review by headquarters, including statistics, data management, quality assurance, protocol administrator, and review by the institutional review board Review by National Cancer Institute Activation of protocol Revision of protocol

In general, the Radiation Therapy Oncology Group and, we predict, other cooperative oncology groups sponsored by the National Cancer Institute, have conducted research of good quality. Our study is the first formal investigation of this and, we believe, the first examination of the methodological quality of randomised controlled trials performed by a cooperative oncology group.

The relation between poor reporting and poor methods was raised in 1980 in a report on patient registration, randomisation, and the importance of avoiding bias in cooperative oncology trials.¹⁰ This may have helped the cooperative oncology groups to be especially aware of methodological issues relating to trials and before the start of modern research on methodological quality.¹¹ Consequently, for cooperative oncology groups such as the Radiation Therapy Oncology Group, even if the published description of the methods of a randomised controlled trial is poor, the quality of the trial should not be assumed to be poor. It is important to note, however, that our findings are based on a select sample of trials, which may not be representative of randomised controlled trials. Further studies to confirm the generalisability of our findings are needed and would be useful.

Another important point relates to any assumptions that trials published before the 1996 CONSORT statement are more likely to be of poorer quality than those published after it.³ The CONSORT statement contains several methodological elements that should be followed to eliminate biased results. The intention of this statement was to improve the conduct, integrity, and reporting of randomised controlled trials.³ Our results show that studies conducted by the Radiation Therapy Oncology Group were of high quality even before publication of the CONSORT statement. It was the reports of the randomised controlled trials that showed deficiencies in their description of the methods used in the trials, not the trials themselves. Our findings indicate that although researchers in the Radiation Therapy Oncology Group were cognisant of key features in the design and conduct of good quality trials, they were less aware of the need to report these to a standard that would meet contemporary (CONSORT) requirements.

It is still appropriate to expect that the CONSORT statement will contribute to the conduct of higher quality randomised controlled trials in the future, since it incorporates and highlights many of the elements needed to perform a trial adequately. We agree with the call for all journals to adopt the policy of only publishing the report of a randomised controlled trial if it follows the CONSORT requirements. This is supported by empirical data that are now emerging about the usefulness of the CONSORT statement. For example, one study compared the quality of reports of trials before and after the CONSORT statement and found that the statement was associated with an improvement in the quality of reports.¹² Further improvements in the quality of the conduct and reporting of clinical research would arise with the publication of research protocols.⁴

What is already known on this topic Assessment of the quality of research evidence is central to informed decision making The quality of randomised controlled trials is often based on the quality of reporting What this study adds Poor reporting of methods in randomised controlled trials may not reflect on poor methods themselves Evaluation of research protocols and contacting trialists should be integral to assessing the quality of such trials

---

Contributors: HPS and BD conceptualised the study, were involved in all aspects of the study, and wrote the first draft of the paper. SD and AK contributed to the study design, collection of data, analysis and interpretation of the data, and writing the report. MC contributed to the study design, interpretation of the data, and writing the report. CS and SS contributed to the collection of data and writing the report. BD will act as guarantor for the paper.

Funding: This research was supported by the Research Program on Research Integrity, an Office of Research Integrity/National Institute of Health collaboration, grant No 1R01NS/NR44417-01.

Competing interests: MC is director of the UK Cochrane Centre, which is funded by the NHS research and development programme and part of the international Cochrane Collaboration. The collaboration produces systematic reviews of health care, including randomised trials, but the views expressed here are not necessarily those of the official policy of the Cochrane Collaboration.

Ethical approval: This study was approved by the University of South Florida Institutional Review Board (No 100449).

References

1. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352: 609-13.[CrossRef][ISI][Medline]
2. Huwiler-Muntener K, Juni P, Junker C, Egger M. Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 2002;287: 2801-4.[Abstract/Free Full Text]
3. Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001;285: 1987-91.[Abstract/Free Full Text]
4. Godlee F. Publishing study protocols: making them visible will improve registration, reporting and recruitment. BMC News Views 2001;2: 4.
5. Silagy CA, Middleton P, Hopewell S. Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA 2002;287: 2831-4.[Abstract/Free Full Text]
6. RTOG procedure manual. Philadelphia: Radiation Therapy Oncology Group Headquarters Office, 2003.
7. Egger M, Davey Smith G, Altman D. Systematic reviews in health care. Meta-analysis in context, 2nd ed. London: BMJ Publishing, 2001.
8. Clarke M, Oxman A, eds. Cochrane reviewers' handbook 4.2.0 [updated March 2003]. In Cochrane Library, issue 2. Oxford: Update Software, 2003.
9. Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 1986;4: 942-51.[Abstract/Free Full Text]
10. Herson J. Patient registration in a cooperative oncology group. Control Clin Trials 1980;1: 101-10.[CrossRef][ISI][Medline]
11. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273: 408-12.[Abstract]
12. Moher D, Jones A, Lepage L. Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA 2001;285: 1992-5.[Abstract/Free Full Text]

(Accepted 10 October 2003)

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Related Articles

Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study

Lesley Wood, Matthias Egger, Lise Lotte Gluud, Kenneth F Schulz, Peter Jüni, Douglas G Altman, Christian Gluud, Richard M Martin, Anthony J G Wood, and Jonathan A C Sterne
BMJ 2008 336: 601-605. [Abstract] [Full Text] [PDF]

Are experimental treatments for cancer in children superior to established treatments? Observational study of randomised controlled trials by the Children's Oncology Group

Ambuj Kumar, Heloisa Soares, Robert Wells, Mike Clarke, Iztok Hozo, Archie Bleyer, Gregory Reaman, Iain Chalmers, and Benjamin Djulbegovic
BMJ 2005 331: 1295. [Abstract] [Full Text] [PDF]

Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study

Julie Pildal, An-Wen Chan, Asbjørn Hróbjartsson, Elisabeth Forfang, Douglas G Altman, and Peter C Gøtzsche
BMJ 2005 330: 1049. [Abstract] [Full Text] [PDF]

Quality of randomised controlled trials: Quality of trial methods is not good in all disciplines

Helen H G Handoll
BMJ 2004 328: 286. [Extract] [Full Text]

Poor reporting of trials does not mean poor quality trials
BMJ 2004 328: 0. [Full Text]

This article has been cited by other articles:

• Abrahamyan, L., Johnson, S. R., Beyene, J., Shah, P. S., Feldman, B. M. (2008). Quality of randomized clinical trials in juvenile idiopathic arthritis. Rheumatology (Oxford) 47: 640-645 [Abstract] [Full text]  
• Djulbegovic, B., Kumar, A., Soares, H. P., Hozo, I., Bepler, G., Clarke, M., Bennett, C. L. (2008). Treatment Success in Cancer: New Cancer Treatment Successes Identified in Phase 3 Randomized Controlled Trials Conducted by the National Cancer Institute-Sponsored Cooperative Oncology Groups, 1955 to 2006. Arch Intern Med 168: 632-642 [Abstract] [Full text]  
• Wood, L., Egger, M., Gluud, L. L., Schulz, K. F, Juni, P., Altman, D. G, Gluud, C., Martin, R. M, Wood, A. J G, Sterne, J. A C (2008). Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 336: 601-605 [Abstract] [Full text]  
• Strevel, E. L., Chau, N. G., Pond, G. R., Murgo, A. J., Ivy, P. S., Siu, L. L. (2008). Improving the Quality of Abstract Reporting for Phase I Cancer Trials. Clin. Cancer Res. 14: 1782-1787 [Abstract] [Full text]  
• Mangoush, O., Purkayastha, S., Haj-Yahia, S., Kinross, J., Hayward, M., Bartolozzi, F., Darzi, A., Athanasiou, T. (2007). Heparin-bonded circuits versus nonheparin-bonded circuits: an evaluation of their effect on clinical outcomes. Eur. J. Cardiothorac. Surg. 31: 1058-1069 [Abstract] [Full text]  
• Baron, G., Boutron, I., Giraudeau, B., Ravaud, P. (2007). Reporting of radiographic methods in randomised controlled trials assessing structural outcomes in rheumatoid arthritis. Ann Rheum Dis 66: 651-657 [Abstract] [Full text]  
• Kumar, A., Soares, H. P., Balducci, L., Djulbegovic, B. (2007). Treatment Tolerance and Efficacy in Geriatric Oncology: A Systematic Review of Phase III Randomized Trials Conducted by Five National Cancer Institute-Sponsored Cooperative Groups. JCO 25: 1272-1276 [Abstract] [Full text]  
• Bainbridge, D., Cheng, D., Martin, J., Novick, R., The Evidence-based Peri-operative Clinical Outcome, (2007). Does off-pump or minimally invasive coronary artery bypass reduce mortality, morbidity, and resource utilization when compared with percutaneous coronary intervention? A meta-analysis of randomized trials. J. Thorac. Cardiovasc. Surg. 133: 623-631 [Abstract] [Full text]  
• Bauwens, K., Matthes, G., Wich, M., Gebhard, F., Hanson, B., Ekkernkamp, A., Stengel, D. (2007). Navigated Total Knee Replacement. A Meta-Analysis. JBJS 89: 261-269 [Abstract] [Full text]  
• Bower, P., Gilbody, S., Richards, D., Fletcher, J., Sutton, A. (2006). Collaborative care for depression in primary care: Making sense of a complex intervention: systematic review and meta-regression. Br. J. Psychiatry 189: 484-493 [Abstract] [Full text]  
• Tiruvoipati, R., Balasubramanian, S. P., Atturu, G., Peek, G. J., Elbourne, D. (2006). Improving the quality of reporting randomized controlled trials in cardiothoracic surgery: The way forward.. J. Thorac. Cardiovasc. Surg. 132: 233-240 [Abstract] [Full text]  
• Newton-Howes, G., Tyrer, P., Johnson, T. (2006). Authors' reply. Br. J. Psychiatry 189: 187-187 [Full text]  
• Kober, T., Trelle, S., Engert, A. (2006). Reporting of randomized controlled trials in Hodgkin lymphoma in biomedical journals.. JNCI J Natl Cancer Inst 98: 620-625 [Abstract] [Full text]  
• Gluud, L. L. (2006). Bias in Clinical Intervention Research. Am J Epidemiol 163: 493-501 [Abstract] [Full text]  
• Lai, R., Chu, R., Fraumeni, M., Thabane, L. (2006). Quality of Randomized Controlled Trials Reporting in the Primary Treatment of Brain Tumors. JCO 24: 1136-1144 [Abstract] [Full text]  
• Kumar, A., Soares, H., Wells, R., Clarke, M., Hozo, I., Bleyer, A., Reaman, G., Chalmers, I., Djulbegovic, B. (2005). Are experimental treatments for cancer in children superior to established treatments? Observational study of randomised controlled trials by the Children's Oncology Group. BMJ 331: 1295- [Abstract] [Full text]  
• Stengel, D., Bauwens, K., Rademacher, G., Mutze, S., Ekkernkamp, A. (2005). Association between Compliance with Methodological Standards of Diagnostic Research and Reported Test Accuracy: Meta-Analysis of Focused Assessment of US for Trauma. Radiology 236: 102-111 [Abstract] [Full text]  
• Pildal, J., Chan, A.-W., Hrobjartsson, A., Forfang, E., Altman, D. G, Gotzsche, P. C (2005). Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ 330: 1049- [Abstract] [Full text]  
• Fervers, B., Burgers, J. S., Haugh, M. C., Brouwers, M., Browman, G., Cluzeau, F., Philip, T. (2005). Predictors of high quality clinical practice guidelines: examples in oncology. Int J Qual Health Care 17: 123-132 [Abstract] [Full text]  
• Soares, H. P., Kumar, A., Daniels, S., Swann, S., Cantor, A., Hozo, I., Clark, M., Serdarevic, F., Gwede, C., Trotti, A., Djulbegovic, B. (2005). Evaluation of New Treatments in Radiation Oncology: Are They Better Than Standard Treatments?. JAMA 293: 970-978 [Abstract] [Full text]  
• Grann, A., Grann, V. R. (2005). The Case for Randomized Trials in Cancer Treatment: New Is Not Always Better. JAMA 293: 1001-1003 [Full text]  
• SAUERLAND, S., DAVIS, T. R. C. (2004). The Consolidated Standards of Reporting Trials (CONSORT): Better Presentation of Surgical Trials in the Journal of Hand Surgery. J Hand Surg Eur Vol 29: 621-624 [Abstract] [Full text]  
• Griffin, S. J., Kinmonth, A.-L., Veltman, M. W. M., Gillard, S., Grant, J., Stewart, M. (2004). Effect on Health-Related Outcomes of Interventions to Alter the Interaction Between Patients and Practitioners: A Systematic Review of Trials. Ann Fam Med 2: 595-608 [Abstract] [Full text]  
• Chan, A.-W., Hrobjartsson, A., Haahr, M. T., Gotzsche, P. C., Altman, D. G. (2004). Empirical Evidence for Selective Reporting of Outcomes in Randomized Trials: Comparison of Protocols to Published Articles. JAMA 291: 2457-2465 [Abstract] [Full text]  
• Handoll, H. H G (2004). Quality of randomised controlled trials: Quality of trial methods is not good in all disciplines. BMJ 328: 286-286 [Full text]  

Rapid Responses:

Read all Rapid Responses

"Intention to read"

Patricio Herrera
bmj.com, 6 Jan 2004 [Full text]

Other areas give less encouraging results

Helen HG Handoll
bmj.com, 9 Jan 2004 [Full text]