Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group

Introduction

Evaluation of the quality of published clinical research is central to informed decision making. Information on trial quality is particularly important during peer review or when results from individual studies are evaluated in systematic reviews or meta-analyses.1 2 The quality of research should always be considered when a report is used in decision making in health care. Poorly conducted and reported research seriously compromises the integrity of the research process, especially if biased results receive false credibility.3

Many efforts have been made to improve the quality of studies and their related publications. The best example was the publication of the Consolidated Standards of Reporting of Trials (CONSORT) statement to improve the quality of trial reports.3 Such efforts to improve the quality of clinical research, however, imply that if certain design or methodological features are not reported then they were not done. Ideally, assessment of the quality of clinical research should not only address reporting but also the original design and intended plan for its conduct and analysis as specified in the trial's research protocol. The importance of linking the final report of clinical trials with their original research protocols led some authors to argue that no randomised controlled trial should be conducted without publication of its research protocol.4 The reasons behind this are that critical comments may be encouraged leading to improvements in trial design, publication can be coupled with trial registration, the original protocol can be compared with what was subsequently done, and investigators can more easily appreciate what research is being conducted in their areas of interest.4 More importantly, publication of research protocols is one of the best ways to minimise bias by explicitly stating a priori hypotheses and methods without the prior knowledge of results.5 Many randomised controlled trials are preceded by the preparation of a written protocol, which describes the conduct of the trial more comprehensively than is possible in many journal articles, and making these protocols available would provide much useful additional information. We aimed to test the assumption that poor reporting reflects poor methods by comparing research protocols with the information published in the final reports of a set of randomised controlled trials.

Methods

We studied randomised controlled trials conducted by the Radiation Therapy Oncology Group. This is a national clinical cooperative group with a focus on the development of studies to improve survival and the quality of life of patients with cancer. It was established in 1968 and is publicly funded by the National Cancer Institute in the United States. The group consists of both clinical and laboratory investigators from over 260 institutions across the United States and Canada, and its membership includes nearly 90% of all comprehensive and clinical cancer centres designated by the National Cancer Institute.6 Before activation, the group's research protocols must pass through a rigorous peer review process and be reviewed and approved through its own committee system and the National Cancer Institute. Development of a protocol consists of six phases (box).6

Our analysis included data related to primary outcomes from all terminated phase III trials conducted by the Radiation Therapy Oncology Group since its establishment in 1968. We extracted data on methodological domains that have been acknowledged as vital for minimising bias in the conduct and analysis of randomised controlled trials.7 The effect of chance is usually minimised by appropriate planning of the trial's size, through a statistical power analysis using estimates for the expected differences between the interventions and prespecified type I (α) and type II (β) error levels.7 To investigate systematic bias we extracted data on the quality of the randomisation process (selection bias) and drop outs (attrition bias).8 Since the primary outcome was survival in most of the studies, we did not consider quality related to observer bias, such as the use of placebo or independent reading of outcomes (there were only three placebo controlled trials). We extracted data from all papers and protocols. The accuracy of this data was verified by the group's statistical centre.

Results

Overall, there were 59 terminated phase III randomised controlled trials, three of which had not been published. We found 58 published papers for the remaining 56 protocols for use in our study. The figure summarises the results according to information from the papers, protocols, and the Radiation Therapy Oncology Group's statistical office. This shows that the reporting of methods in the publications does not necessarily reflect the methodological quality of the associated protocols. For example, a priori sample size calculations were performed in 44 (76%) trials, but this information was given in only nine of the 58 published papers (16%). Although all trials had adequate allocation concealment (through central randomisation), this was reported in only 24 (41%) of the papers. From our initial data extraction, we found that 40 (69%) of these trials used an intention to treat analysis. This number was increased to 48 (83%) after verification by the Radiation Therapy Oncology Group. End points were clearly defined, and α and βerrors were prespecified in 44 (76%) and 43 (74%) trials, respectively, but only reported in six (10%) of the papers. Interestingly, reporting of drop outs was meticulous; we found no difference in frequency (91%) between data presented in the papers and those in the original files.

Quality of reporting compared with actual methodological quality of 56 randomised controlled trials (58 reports) conducted by the Radiation Therapy Oncology Group, based on information from published reports, protocols, and verification by the group. Absolute numbers of reports are shown

• Download figure
• Open in new tab
• Download powerpoint

Discussion

Poor reporting of randomised controlled trials may not indicate poor quality of the trials themselves. We are aware of two other studies that reported empirical assessments of this relation. One study evaluated the quality of 63 randomised controlled trials of breast cancer treatment. Data were extracted from publications related to these trials and the results compared with the information provided by the principal investigators. The study concluded that faulty reporting reflected faulty methods.9 Another study, however, concluded that even well designed and conducted trials may be badly reported.2 This conclusion was drawn indirectly from an assessment of three key indicators of quality: adequate allocation concealment, appropriate blinding, and use of intention to treat analysis.2 Unlike our study, neither of these studies reported a comparison of the quality of reporting with the methods specified in the original research protocols.

In general, the Radiation Therapy Oncology Group and, we predict, other cooperative oncology groups sponsored by the National Cancer Institute, have conducted research of good quality. Our study is the first formal investigation of this and, we believe, the first examination of the methodological quality of randomised controlled trials performed by a cooperative oncology group.

The relation between poor reporting and poor methods was raised in 1980 in a report on patient registration, randomisation, and the importance of avoiding bias in cooperative oncology trials.10 This may have helped the cooperative oncology groups to be especially aware of methodological issues relating to trials and before the start of modern research on methodological quality.11 Consequently, for cooperative oncology groups such as the Radiation Therapy Oncology Group, even if the published description of the methods of a randomised controlled trial is poor, the quality of the trial should not be assumed to be poor. It is important to note, however, that our findings are based on a select sample of trials, which may not be representative of randomised controlled trials. Further studies to confirm the generalisability of our findings are needed and would be useful.

Another important point relates to any assumptions that trials published before the 1996 CONSORT statement are more likely to be of poorer quality than those published after it.3 The CONSORT statement contains several methodological elements that should be followed to eliminate biased results. The intention of this statement was to improve the conduct, integrity, and reporting of randomised controlled trials.3 Our results show that studies conducted by the Radiation Therapy Oncology Group were of high quality even before publication of the CONSORT statement. It was the reports of the randomised controlled trials that showed deficiencies in their description of the methods used in the trials, not the trials themselves. Our findings indicate that although researchers in the Radiation Therapy Oncology Group were cognisant of key features in the design and conduct of good quality trials, they were less aware of the need to report these to a standard that would meet contemporary (CONSORT) requirements.

It is still appropriate to expect that the CONSORT statement will contribute to the conduct of higher quality randomised controlled trials in the future, since it incorporates and highlights many of the elements needed to perform a trial adequately. We agree with the call for all journals to adopt the policy of only publishing the report of a randomised controlled trial if it follows the CONSORT requirements. This is supported by empirical data that are now emerging about the usefulness of the CONSORT statement. For example, one study compared the quality of reports of trials before and after the CONSORT statement and found that the statement was associated with an improvement in the quality of reports.12 Further improvements in the quality of the conduct and reporting of clinical research would arise with the publication of research protocols.4