Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publicationsBMJ 2012; 344 doi: http://dx.doi.org/10.1136/bmj.d8141 (Published 03 January 2012) Cite this as: BMJ 2012;344:d8141
- Beate Wieseler, head of department of drug assessment,
- Michaela F Kerekes, data manager,
- Volker Vervoelgyi, researcher,
- Natalie McGauran, researcher,
- Thomas Kaiser, head of department of drug assessment
- 1Institute for Quality and Efficiency in Health Care, Dillenburger Strasse 27, 51105 Cologne, Germany
- Correspondence to: B Wieseler
- Accepted 8 December 2011
Objective To investigate to what extent three types of documents for reporting clinical trials provide sufficient information for trial evaluation.
Design Retrospective analysis
Data sources Primary studies and corresponding documents (registry reports, clinical study reports, journal publications) from 16 health technology assessments of drugs conducted by the German Institute for Quality and Efficiency in Health Care between 2006 and February 2011.
Data analysis We assessed reporting quality for each study and each available document for six items on methods and six on outcomes, and dichotomised them as “completely reported” or “incompletely reported.” For each document type, we calculated the proportion of studies with complete reporting for methods and outcomes, per item and overall, and compared the findings.
Results We identified 268 studies. Publications, study reports and registry reports were available for 192 (72%), 101 (38%), and 78 (29%) studies, respectively. Reporting quality was highest in study reports, which overall provided complete information for 90% of items (1086/1212). Registry reports provided more complete information on outcomes than on methods (overall 330/468 (71%) v 147/468 (31%)); the same applied to publications (594/1152 (52%) v 458/1152 (40%)). In the matched pairs analysis, reporting quality was poorer in registry reports than in study reports for overall methods and outcomes (P<0.001 in each case). Compared with publications, reporting quality was poorer in registry reports for overall methods (P<0.001), but better for outcomes (P=0.005).
Conclusion Registry reports and publications insufficiently report clinical trials but may supplement each other. Measures to improve reporting include the mandatory worldwide implementation of adequate standards for results registration.
A prerequisite of evidence based healthcare is that medical interventions are analysed in clinical trials and the findings from these trials are used to inform decision making in the healthcare system. The selective publication of clinical trials (publication bias) and their outcomes (outcome reporting bias) have been identified as major problems distorting the scientific evidence available. As a result, perception of the effects of healthcare interventions based on published literature is biased towards overestimating benefits and underestimating harms.1 2 3 4 5 This problem of distorted public record is widely prevalent.6
To solve the problem, study registration (disclosure at inception that a study is being conducted) and results registration (posting of results after a study has been completed) have been partly implemented using publicly accessible databases. Usually, the details provided at inception and after completion both include information on study methods.
Initiatives to promote study registration at inception were launched in the 1960s,7 and this practice became widely established in 2004 after the editors of medical journals determined that only registered trials would be published.8 In contrast, results registration (the focus of this paper), lagged behind, but gained momentum in August 2004 after the settlement between the pharmaceutical company GlaxoSmithKline and the New York General Attorney after the company withheld data on paroxetine. This settlement required the company to publish results summaries of all of its sponsored clinical drug trials completed after December 2000.9 10
In January 2005 the associations of the pharmaceutical industry issued a joint position statement (updated in 2009) committing their member companies to post the results of certain clinical trials in results registries (that is, on company websites or in other web based industry registries such as ClinicalStudyResults.org).11 In April 2005 the Ottawa Statement, besides proposing the registration of the original protocol of a study, suggested minimum requirements for results registration, recommending that “results for outcomes and analyses specified in the protocol (as approved by the institutional review boards/independent ethics committees), as well as data on harms should be registered regardless of whether or not they are published.” 12 In September 2007 the Food and Drug Administration (FDA) Amendments Act became effective for ongoing and future studies regulated by the administration and required not only the posting of a defined dataset for study registration but also the posting of “basic results” (such as demographic data and results of primary and secondary efficacy end points) in a new section of the ClinicalTrials.gov database. Basic results were to be entered into the database starting in September 2008. The requirements were extended to include data on adverse events in September 2009.13 14
Despite the steps taken, discussion of the adequate reporting of study results continues, with questions about results registration beyond ClinicalTrials.gov (such as in the European Union Drug Regulating Authorities Clinical Trials database, EudraCT),15 16 availability of data from older studies,14 17 18 and reporting of individual patient data.19
The German Institute for Quality and Efficiency in Health Care (IQWiG) prepares health technology assessment reports for health policy makers. For this purpose, besides searching bibliographic databases, it routinely searches clinical trial registries and requests clinical study reports from manufacturers. The assessment reports form the basis for decisions on the reimbursement of drug and non-drug interventions by statutory health insurance funds.20
The aim of this analysis was to investigate to what extent three types of documents for reporting clinical trials provide sufficient information on methods and outcomes to enable the evaluation of a trial. We compared reports posted in trial results registries (referred to as “registry reports”), clinical study reports submitted to regulatory authorities during regulatory drug approval (referred to as “study reports”), and journal publications. Table 1⇓ describes the characteristics of the three document types. In short, journal publications and registry reports make summaries of studies publicly available (such as to clinicians and authors of systematic reviews) and are used to inform clinical and health policy decision making. In contrast, study reports are detailed accounts of studies and are in general not publicly available. These documents are used to inform regulatory decision making. Details on reporting standards (Consolidated Standards Of Reporting Trials (CONSORT) for publications, the FDA Amendments Act for (some) registry reports, and the International Conference on Harmonisation (ICH E3) Guideline for Structure and Content of Clinical Study Reports) are also outlined in table 1⇓.
The systematic search for primary studies to be included in drug assessments by IQWiG routinely covers bibliographic databases as well as trial registries and trial results databases. In order to obtain the most complete dataset possible, the institute also asks the relevant drug manufacturer to provide an overview of sponsored published and unpublished studies investigating the drug of interest. From this list, IQWiG selects the studies deemed relevant to the assessment and asks the manufacturer for submission of the complete study report. However, since submission of study overviews and study reports by manufacturers is voluntary, these documents are available only for a subset of the drugs assessed. As a result of this search process, IQWiG’s assessment is based on various types of documents reporting information on a trial—full text journal publications, registry reports, and study reports.
Study and document selection
To investigate the quality of reporting in such registry reports, study reports, and journal publications, we selected a pool of primary studies and documents from health technology assessments prepared by IQWiG. We included all drug assessments finalised between 2006 and February 2011 that contained a systematic search for registry reports as part of the information retrieval process (see appendix table A on bmj.com). In our analysis we considered all registry reports, full study reports (including appendices), and full text journal publications available for all primary studies analysed in these assessments. Since all drug assessments were based on randomised controlled trials, our study sample included only this design.
Data extraction and coding
We developed a database for the extraction of characteristics of the assessment report (drug assessed, drug manufacturer), study characteristics (study identification number, study sponsor), and documents available (registry report, study report, journal publication) using MS Access software. In addition, reporting quality was assessed for each study and each available document with regard to items about study methods and outcomes (listed in table 3⇓). For study methods items in available documents, reporting quality was classified as (1) completely reported, (2) partly reported, or (3) not reported. For study outcome items, reporting quality was recorded as (1) completely reported including numerical data, (2) partly reported including numerical data, (3) verbally reported without numerical data, or (4) not reported.
Our requirements for complete reporting were based on the requirements of authors of systematic reviews (that is, provision of adequate information for assessment of risk of bias and adequate data for meta-analyses).21 A definition of all categories is provided in the appendix tables B and C on bmj.com.
All data were extracted and coded by one reviewer (MFK). All data and codings of registry reports were checked by a second reviewer (VV). In addition, a random sample of 10% of the data and codings of study reports and journal publications were also checked by the second reviewer (VV). Discrepancies were resolved by consensus, if necessary, after discussion with a third reviewer (BW).
To investigate the completeness of reporting, we calculated the proportion of studies with complete reporting, per item and overall, for study methods and outcomes for each of the three document types. For this analysis, the categories described above were dichotomised as “completely reported” (category 1 above) or “incompletely reported” (all other categories).
To evaluate in more detail the information content of registry reports compared with study reports or journal publications, studies were identified for which both registry reports and study reports or registry reports and journal publications were available. This analysis was based on these paired samples to avoid bias caused by the comparison of samples including different studies. Within the paired samples, we used the categories described above to determine whether registry reports provided similar (same category in both document types), more (different category with higher information content), or less (different category with lower information content) information than the study reports or journal publications of the corresponding studies. Within the paired samples, the proportion of studies with complete reporting was calculated and compared by means of a McNemar test in order to take the potential dependency of samples into account. The data were analysed using SAS 9.2.
Table 2⇓ shows the characteristics of the included studies and documents: our sample included 268 studies from 16 assessment reports covering seven different therapeutic areas, with about half of the studies investigating antidepressants. Most (72%) of the studies were published in journals. Study reports and registry reports were available for about 40% and 30% of the studies, respectively.
All registry reports were posted in industry registries (mainly on company websites); no reports in public registries were identified. About three quarters of the reports originated from three pharmaceutical companies, while no registry entries from non-industry sponsors were identified. Their volume ranged from two to about 150 pages (median of seven).
To directly compare the reporting quality of a given study in the different types of documents, we identified paired samples of registry reports and study reports (50 studies) or journal publications (47 studies). All three types of documents were available for 29 studies.
Overall reporting quality in study reports, registry reports, and journal publications
Table 3⇓ shows the completeness of information on study methods and outcomes by document type.
In the study reports, which showed the highest reporting quality, complete information was provided for about 90% of items when overall study methods and outcomes were analysed. In the analysis of single items, study reports provided complete information on the primary end point for 77 of 101 studies (76%), which was the lowest rating of all items investigated. The most poorly reported methods items were allocation concealment (80%) and sample size estimation and randomisation (in each case 81%). The best reported methods items were definition of the dataset for “intention to treat” analysis (99%) and blinding (100%), whereas for outcomes, the best reported items were overall withdrawals and reasons for withdrawals (in each case 100%).
In the registry reports, information on study outcomes was far more complete than that on study methods (overall 71% v 31%) (table 3⇑). In particular, none of the registry reports included complete information on randomisation or allocation concealment. As in the study reports, poorer reporting of outcomes mainly affected the information on the primary end point, where complete information was provided for only 44 of 78 studies (56%). The best reported items for methods and outcomes were the patient number in the intention to treat dataset (69%) and overall withdrawals (85%), respectively.
In journal publications the reporting of study outcomes was also more complete than that of study methods (overall 52% v 40%). As with the other two document types, in journal publications poorer reporting of outcomes mainly affected the information on the primary end point, but also (serious) adverse events; in each case complete information was provided for only about a third of the studies. Regarding study methods, the poorest reporting was for allocation concealment (19%). The best reported methods item was blinding (59%), whereas for outcomes, the best reported items were patients withdrawn due to adverse events (75%) and overall withdrawals (77%).
Based on the sample of 29 studies for which all three document types were available, the right column of table 3⇑ shows for which proportion of studies complete information was presented only in the study report. Inversely, this analysis also shows for which proportion of studies complete information could be extracted from the combination of journal publications and registry reports for a given study. For more than 40% of methods items, complete information was available only from study reports. However, the combination of registry reports and publications produced complete information on more than 90% of the outcome items investigated in this analysis.
Detailed comparison of registry reports with study reports or publications using paired samples
To address the reporting quality of registry reports in more detail, table 4⇓ shows the direct comparison of information on study methods and outcomes for a given study in paired samples—that is, studies for which either a registry report and a study report or a registry report and a journal publication were available.
Registry reports versus study reports
In our sample a similar level of information was available in the registry reports and the study reports mainly for study outcomes (overall 81%). However, registry reports mainly provided less information for study methods (overall 64%), in particular on randomisation and allocation concealment, where less information was provided for at least 90% of studies (table 4⇑).
Registry reports versus journal publications
Registry reports also showed deficits in the reporting of methods compared with journal publications. Only a few registry reports provided more information on methods: in most cases, similar or less information was provided. In contrast, the overall reporting of outcomes was more complete in registry reports than in journal publications, particularly with regard to reporting of (serious) adverse events (table 4⇑).
Appendix table D on bmj.com shows the statistical analysis of the comparison of proportions of studies with complete information in the matched pairs sample. Reporting quality (that is, the proportion of studies with complete information) was significantly lower in registry reports than in study reports for both overall methods (P<0.001) and outcomes (P<0.001). Compared with journal publications, registry reports showed significantly lower reporting quality for overall methods (P<0.001) but a better reporting quality for overall outcomes (P=0.005).
Our analysis showed differences in the completeness of three reporting formats for clinical trials. Table 5⇓ summarises our findings, describes the advantages and disadvantages of the three document types, and suggests measures to improve the availability of information on study methods and outcomes for patients, clinicians, and other stakeholders such as health technology assessment bodies.
A considerably higher proportion of study reports prepared for regulatory approval of drugs provided complete information on study methods and results than did registry reports or journal publications. In addition, we detected differences between the two latter types of documents: while journal publications offered more complete information on study methods, registry reports were in part superior in reporting outcomes.
Overall, the completeness of information on a study could be improved by supplementing the journal publication with the registry report (which, unlike the study report, is publicly available), albeit for only about 30% of the studies included in our analysis. The limited availability of registry reports underlines the need for mandatory registration of results. In our sample, registry reports provided additional information mainly on study outcomes, particularly on the primary end point and on adverse events. When registry reports and journal publications were combined, our analysis showed complete reporting of outcomes for a high proportion of studies. However, we analysed only a limited number of outcomes, and we don’t know whether complete information on other outcomes, especially on secondary efficacy outcomes, could also be obtained with this approach.
Incomplete reporting: a question of space?
One reason for more complete reporting in study reports might be that these are not limited in volume (they comprise several hundred to several thousand pages, including appendices). However, with electronic publication in web based databases, there is also no need for volume limits in registry reports, which in our sample had a median volume of only seven pages (although some sponsors posted registry reports of up to about 150 pages). Another reason for the superior reporting in study reports might be that these are often prepared by specifically trained staff following well defined document standards and procedures, thus ensuring a high level of quality. In contrast, reporting standards for registry entries are less rigorous, which may explain reporting deficits.
In contrast to the other document types, journal publications do have page limits. However, reporting standards such as the CONSORT statement22 and the accompanying extended explanation and elaboration document23 enable authors of manuscripts to provide transparent and complete information on a clinical trial. In addition, many print journals allow web appendices, so that space limitations can no longer be regarded as an excuse for incomplete reporting.
Our requirements for complete information did not exceed those of the CONSORT statement. However, our analysis exposed information gaps in journal publications, confirming findings by other researchers: recent studies investigating the impact of CONSORT have shown that, although reporting standards have improved reporting in journal publications, deficits still remain.24 25 26 Further efforts are needed to improve trial reporting in journal publications, including training for authors and peer reviewers with the help of sources such as the EQUATOR network (Enhancing the QUAlity and Transparency Of health Research, which aims to “promote the use of reporting guidelines and good research reporting practices through an education and training programme”27).
Standards for registry reports
In contrast to study reports, which are written according to the ICH E3 guideline,28 and journal publications, which are covered by the CONSORT statement,22 registry reports have no accepted worldwide standard. The settlement by GlaxoSmithKline and the New York State Attorney required the company to publish study results based on the synopsis of a study report prepared according to ICH E3 plus additional information.9 10 The ICH E3 synopsis is suggested by the joint position statement of pharmaceutical associations on trial registration.11 However, the ICH E3 synopsis was developed to accompany a full study report, not to provide a comprehensive representation of a clinical trial. The specification of content in this synopsis in the ICH E3 guideline is too vague and insufficient to ensure full representation of a study when used as a standalone document. Thus, the ICH E3 synopsis is an inadequate standard for results registration.
Since 2004, when results registration was implemented by some pharmaceutical companies, new requirements have been defined for public registries.29 Future registry reports will mostly be based, firstly, on the requirements for trials registration at inception, which include the provision of minimum information on trial methodology, and, secondly, on the requirements for results posting after trial completion (see below).
Standards for registration at inception
The dataset for registration at inception of a clinical trial differs between various registries. Most registries include the World Health Organization’s 20 item minimum dataset, and some registries require or allow provision of additional information. A recent study by Reveiz et al, however, showed that entries in WHO primary registries of randomised controlled trials actively recruiting in 2008 contained only limited methodological information: the extent of information available varied between registries, but was generally insufficient to enable critical appraisal of the registered trials.30 While our analysis only investigated reports from results registries, the above analysis shows that reports from trial registries also insufficiently report methodological information.
If a major aim of study registration is to use the information from studies as a basis for decision making in healthcare, it is necessary to optimise reporting requirements according to the needs of this goal. The lack of information on study methods in current registry entries emphasises the need to post the full study protocol (and any amendments) to provide sufficient detail on a study’s conduct and methods.31 32 The study protocol itself should be prepared in a standardised manner; evidence based recommendations for this purpose are currently being developed.33
Standards for results registration
Mandatory results registration in public registries has been implemented only in ClinicalTrials.gov. On the basis of the FDA Amendments Act of 2007, detailed requirements for results reporting in ClinicalTrials.gov have been defined,34 35 which are far more specific than those for the ICH E3 synopsis. In particular, the posting of results comprises the structured reporting of extended numerical data. In contrast, results registration in Europe, although stipulated by law since 2004, has not yet been implemented. The European clinical trial registry became available in March 2011 but so far does not include study results. However, reporting standards similar to the requirements of ClinicalTrials.gov are under discussion.15 16 36 37
Our sample did not include registry reports prepared according to these new standards. The evaluation of the upcoming registry reports will show whether the shortcomings of the current registry reports will be solved by the new standard.
Closing the evidence gap
While new reporting standards such as the requirements of ClinicalTrials.gov might improve the reporting of newer studies, clinical practice will remain largely based on the results of older studies in the foreseeable future. It is not to be expected that studies completed before the new requirements came into effect in 2007 will be retrospectively reported according to the new standards. Although the pharmaceutical industry associations are phasing out their main results registry (ClinicalStudyResults.org) because of the availability and extensive use of databases such as ClinicalTrials.gov,38 registry reports of the older studies will be available only in their current format. Moreover, since mandatory results registration is currently effective only for FDA regulated studies, a large number of studies conducted in the next few years will have no registry reports or only insufficient ones. This will result in an incomplete evidence base.
Study reports prepared for regulatory authorities might be a solution to partially close the information gap for a large number of studies. As our analysis has shown, these reports provide the most complete information on a clinical trial. However, study reports are generally unavailable except to regulatory authorities. One possibility for improving the evidence base for older studies would be to make the study reports on file at regulatory agencies publicly available.14 17 18 Our analysis strongly supports this suggestion. Study reports could also close the information gap for future studies in countries where mandatory registration of trial results to an adequate standard is yet to be implemented.
Registry reports and study reports as a data source for systematic reviews
Our definition of “complete information” was based on the requirements of authors of systematic reviews. To prepare high quality systematic reviews, authors have to be able to assess the risk of bias in the included studies and thus require comprehensive information on study methods and results. Furthermore, sufficient study data for meta-analysis have to be provided. The final goal in the provision of complete information is to evaluate and summarise evidence on interventions to inform healthcare decisions.
So far, only a minority of systematic reviews have searched registries,39 even though, as our analysis has shown, registry reports may provide additional relevant data, particularly on study outcomes. The Cochrane handbook also emphasises that registries are becoming an increasingly important source of information.40
Consideration of registry reports should reduce the problem of publication bias by identifying studies not published in journals. They could also reduce outcome reporting bias by providing information on additional study outcomes not reported in publications. In our sample, registry reports included more information on study outcomes for a substantial proportion of studies. To decrease outcome reporting bias, valid standards for registry reports have to be developed, and it must be ensured that the content of registry reports reflects the original study protocol and clearly identifies any changes. This is an additional argument for posting full protocols in trial registries. This would also require the posting of statistical analysis plans defining the planned analyses a priori.
Another prerequisite for reducing publication and outcome reporting bias is that results registration becomes mandatory outside the United States.
In our sample, full study reports provided complete information on the large majority of methods and results data items. Such reports are prepared for a large proportion of studies due to regulatory requirements. This raises the question of whether they should also be used for results registration and thus be available for authors of systematic reviews. If necessary, the ICH E3 guideline could be reviewed to assess whether information not needed for registration could be presented in separate modules, which could be detached before posting the report in a results registry.
Limitations of our analysis
Our sample covered only a limited number of medical indications and interventions. Moreover, the registry reports included were prepared by a limited number of companies, with most reports being produced by three companies.
As previously stated, the registry reports included were not prepared according to the new requirements of the FDA Amendments Act or the upcoming European regulation. Future reports in ClinicalTrials.gov may be of better quality. Moreover, our sample of studies was restricted to randomised controlled trials investigating drugs, so we cannot comment on other study designs or studies of non-drug interventions.
Our analysis confirms that publicly available information on clinical trials is often insufficient. Consequently, stakeholders in the healthcare system face difficulties in reaching reliable conclusions on the effect of medical interventions.
For many studies, documents providing the most complete information are available within pharmaceutical companies and regulatory authorities. These study reports prepared for regulatory drug approval should be made publicly available to support decision making outside the approval processes.
Our analysis also indicates that the content of most registry reports currently available is insufficient to provide complete information on a clinical trial, mainly due to shortcomings in the reporting of study methods. On the other hand, these reports are able to supplement journal publications, in particular with information on study outcomes, and should thus become a standard data source for systematic reviews.
There is a need for mandatory registration of all clinical trials and for a mandatory standard for registry reports containing sufficient details on study methods and results to allow full evaluation of the validity of a clinical trial and its outcomes.
What is already known on this topic
Selective publication of clinical trials and their outcomes is a major problem in clinical research and leads to an overestimation of benefits and an underestimation of harms of treatment effects
Standards to improve the reporting of publicly accessible documents on clinical trials have been introduced for journal publications and in part for reports posted in trial (results) registries
Clinical study reports submitted to regulatory authorities during the drug approval process follow detailed reporting standards, but they are generally not publicly available
What this study adds
Our analysis of whether these three types of documents for reporting clinical trials provide sufficient information to enable trial evaluation shows that clinical study reports provide the most complete information for appropriate evaluation of clinical trials
Journal publications and reports posted in trial results registries provide insufficient information on clinical trials but may supplement each other
Reporting of clinical trials could be improved by implementing worldwide mandatory reporting standards for results registries (for new studies) and by making clinical study reports submitted to regulatory authorities publicly available (for older studies)
Cite this as: BMJ 2012;344:d8141
The authors thank Brian and Katrin McGauran for their comments on the manuscript.
Contributors: BW and TK conceived the idea for the analysis. MFK extracted and coded the data. VV checked the data and codings and prepared the statistical analysis. BW drafted the first version of the paper, which NM revised. All authors contributed to data interpretation and to critical revision of the paper and have seen and approved the final version. BW is the guarantor of the paper.
Funding: This work was supported by IQWiG.
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare no support from any company for the submitted work. All authors are employees of IQWiG. In order to produce unbiased health technology assessment reports, the institute depends on access to all of the relevant data on the topic under investigation. The authors therefore support the mandatory worldwide establishment of trial registries and study results databases as well as the implementation of reporting standards for clinical trials.
Data sharing: No additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.