Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analysesBMJ 2017; 356 doi: https://doi.org/10.1136/bmj.j448 (Published 17 February 2017) Cite this as: BMJ 2017;356:j448
All rapid responses
I thank Alessandra Pugi for drawing attention to our article.
We are aware that our meta-analytic results showed that the direction and statistical significance of each summary effect does not changed for any meta-analysis once the new RCTs are included. However, the change in summary statistics varied from 0% to 29% and was greater than 10% for five of 14 systematic reviews and greater than 20% for two.
Despite these results, we believe that is probably worth spending time to search clinical trial registries and that searching trial registries should continue to be a recommendation with strategy to improve adherence to this recommendation. Clinical trial registries have been developed and enforced by editors and policy makers to reduce waste in research and publication bias. They have been considered an important step toward more transparency and increasing research value.
1) Lack of searching trial registries implies that we could have missed evidence (in our study evidence was missing in 41 of 95 systematic reviews) while the objective of a systematic review is to collate all empirical evidence (Cochrane handbook, JPT Higgins, 2011).
2) Knowing the amount of unpublished and not available data is very important information as to understand what is under “the tip of the iceberg”. This step is important to grade the quality of evidence in systematic review with meta-analysis. In our study, among the 122 RCTs identified, 63 (52%) had results available.
3) Searching trial registries allowed the inclusion of new RCTs in meta-analyses in 14 of 95 (15%) systematic reviews in our study. However, the addition of new RCTs in these meta-analyses did not change the statistical significance but increased precision. Further, some trials results were still missing and could not be included in the new analyses. Searching registry may become even more important if posting results improve and the protocol and statistical analysis plan become available.
Finally, searching in trial registries represents a low burden. The number of records to screen by systematic review was limited (median [Q1-Q3] = 23 [6-150]) except for 3 systematic reviews with more than 1000 records to screen (respectively 1661, 2680 and 3576).
Actually, the effort of searching trial registry is counterbalanced by the lack of results. In our study, in half of the 14 selected meta-analyses, only a part of the RCTs retrieved by the trial registry search had available results, resulting in “incomplete evidence meta-analysis”.
In our study, among the 122 RCTs retrieved from trial registry search, 1) 41 had their results posted; 2) 21 had a publication available identified by a reference reported on the registry and 10 from a complementary search and 3) 31 had results available on the company’s Web site.
This should change in the near future because the policy of trial registration changes in favour the posting of results and of protocol. In the final rule of trial registration in Clinicaltrial.gov, the FDAAA of 2007, made publicly available on September 16, 2016, requires submission of full protocol and statistical analysis plan at the same time as submission of results information (Zarin nejm 2016). Registry could be a more important source of results in the future.
Amelie Yavchitz , MD, PhD
Centre de Recherche Epidémiologie et Statistique,
INSERM U1153, Paris, France
Competing interests: No competing interests
We read with great interest the article "Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses" by Baudard M. et al1. It analysed the impact of a search strategy including clinical trial registries on the results of published systematic reviews of pharmacological treatments.
The authors reported that 52% of the selected systematic reviews did not report a search of trial registries. Throughout the consultation of the World Health Organization International Trials Registry Platform (WHO ICTRP) search portal, they found 122 additional randomised clinical trials (RCT). Finally, 14 systematic reviews were reanalysed, including 45 RCTs from registries. The addition of data allowed for increasing the number of patients but none of the changes in the summary effect estimates had a qualitative impact in the interpretations of the results. However, the conclusion of the authors was: “Searching clinical trial registries is essential for identifying additional trials that could increase the value of systematic reviews”.
This conclusion seems to be in contrast with the results of the paper.
We need to consider that the aim of a systematic review is providing evidences of efficacy to support clinical decisions. The value of a systematic review should be measured in relation to the capability of detecting the appropriate clinical behaviour. In fact, the methodology of guidelines development2, as the best tool for clinical appropriateness, puts systematic reviews at the highest level in the hierarchy of evidence.
Therefore, considering that the new research strategy reported by Baudard M et al1, did not affect the qualitative results of the systematic review, it is safe to conclude that this research method did not add value to the clinical results and had no implications for clinical practice.
In our opinion, an exhaustive literature search in systematic reviews could be acceptable if it has an impact on clinical outcomes. Sagliocca L. et al3 compared the traditional systematic reviews research method with a more pragmatic approach based on the review of a selected number of core journals. The authors demonstrated, in a sample of systematic reviews, that reproduction of the literature search restricted to major medical journals in the fields, as well as to the most relevant generalist journals, does not alter the quantitative estimates of efficacy, nor the clinical indications.
Finally, we strongly believe that systematic reviews and guidelines are the most valid instrument in supporting clinical decisions and should be conducted according to an accurate and reliable method. The comprehensiveness of a literature search is the main factor influencing the accuracy but should be assessed considering the implications for clinical practice.
1. Baudard M, Yavchitz A, Ravaud P, et al. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ 2017;356:j448
2. Developing NICE guidelines: the manual. October 2014
3. Sagliocca L, De Masi S, Ferrigno L, et al. A pragmatic strategy for the review of clinical evidence. J Eval Clin Pract. 2013 Aug;19(4):689-96
Competing interests: No competing interests