Effect of redundant clinical trials from mainland China evaluating statins in patients with coronary artery disease: cross sectional study
BMJ 2021; 372 doi: https://doi.org/10.1136/bmj.n48 (Published 02 February 2021) Cite this as: BMJ 2021;372:n48
- Yuanxi Jia, postdoctoral fellow1,
- Jiajun Wen, doctoral student2,
- Riaz Qureshi, doctoral student2,
- Stephan Ehrhardt, associate professor of epidemiology2,
- David D Celentano, professor of epidemiology2,
- Xin Wei, clinical fellow3,
- Lori Rosman, informationist4,
- Yumeng Wen, clinical fellow5,
- Karen A Robinson, professor of medicine1
- 1Department of Medicine, School of Medicine, Johns Hopkins University, 1830 East Monument Street, Suite 8068, Baltimore, MD 21287, USA
- 2Department of Epidemiology, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, USA
- 3Department of Cardiology, Virginia Commonwealth University, Richmond, VA, USA
- 4Welch Medical Library, Johns Hopkins University, Baltimore, MD, USA
- 5Division of Nephrology, School of Medicine, Johns Hopkins University, Baltimore, MD, USA
- Correspondence to: K A Robinson krobin{at}jhmi.edu
- Accepted 3 December 2020
Abstract
Objective To identify redundant clinical trials evaluating statin treatment in patients with coronary artery disease from mainland China, and to estimate the number of extra major adverse cardiac events (MACEs) experienced by participants not treated with statins in those trials.
Design Cross sectional study.
Setting 2577 randomized clinical trials comparing statin treatment with placebo or no treatment in patients with coronary artery disease from mainland China, searched from bibliographic databases to December 2019.
Participants 250 810 patients with any type of coronary artery disease who were enrolled in the 2577 randomized clinical trials.
Main outcome measures Redundant clinical trials were defined as randomized clinical trials that initiated or continued recruiting after 2008 (ie, one year after statin treatment was strongly recommended by clinical practice guidelines). The primary outcome is the number of extra MACEs that were attributable to the deprivation of statins among patients in the control groups of redundant clinical trials—that is, the number of extra MACEs that could have been prevented if patients were given statins. Cumulative meta-analyses were also conducted to establish the time points when statins were shown to have a statistically significant effect on coronary artery disease.
Results 2045 redundant clinical trials were identified published between 2008 and 2019, comprising 101 486 patients in the control groups not treated with statins for 24 638 person years. 3470 (95% confidence interval 3230 to 3619) extra MACEs were reported, including 559 (95% confidence interval 506 to 612) deaths, 973 (95% confidence interval 897 to 1052) patients with new or recurrent myocardial infarction, 161 (132 to 190) patients with stroke, 83 (58 to 105) patients requiring revascularization, 398 (352 to 448) patients with heart failure, 1197 (1110 to 1282) patients with recurrent or deteriorated angina pectoris, and 99 (95% confidence interval 69 to 129) unspecified MACEs.
Conclusions Of more than 2000 redundant clinical trials on statins in patients with coronary artery disease identified from mainland China, an extra 3000 MACEs, including nearly 600 deaths, were experienced by participants not treated with statins in these trials. The scale of redundancy necessitates urgent reform to protect patients.
Introduction
When investigators overlook existing evidence, clinical trials might be initiated to deal with treatment uncertainty that has already been solved by previous studies.1 Failing to establish equipoise, such clinical trials are deemed redundant and inappropriate by the research community as they waste resources and put patients at risk of harm. This unnecessary replication is highly problematic in the context of placebo controlled trials, which are legitimate only if no known treatment option exists.23 Otherwise, patients who only take placebo in the control group are denied a known effective treatment, violating the ethical principles of conducting clinical trials.4
Cumulative meta-analysis can be used to show the chronological change of the overall estimate of a treatment effect as individual trials are added to the evidence pool.5 Cumulative meta-analysis has been adapted to evaluate when sufficient evidence has accrued to reach a conclusion and, subsequently, to identify the redundancy of additional trials.5 On the one hand, it is challenging for investigators to decide the adequacy of existing evidence solely based on cumulative meta-analysis owing to the varied interpretation of the results, as well as the possibility of the overall estimates at early stages being overridden by subsequent trials.6 On the other hand, clinical practice guidelines, which ideally are both consensus oriented and based on systematic reviews, take into account the balance of harms and benefits as well as feasibility and patient values. Therefore, clinical practice guidelines are more comprehensively adopted by the research community to deal with treatment uncertainty on the basis of cumulative meta-analysis.7
The research community has witnessed a recent proliferation of scientific publications from mainland China.8 However, there are concerns over the redundancy of those research activities.9 We evaluated the potential redundancy of clinical trials from mainland China. Specifically, we identified randomized clinical trials evaluating statins for the treatment of coronary artery disease that were conducted after the benefits of statins were affirmed by clinical practice guidelines and cumulative meta-analysis. We estimated the number of major adverse cardiac events (MACEs) that were experienced by patients who did not receive statins in the redundant trials.
Methods
The data collected and analyzed in our study were retrieved from publications of clinical trials. We followed the strengthening the reporting of observational studies in epidemiology (STROBE) reporting guideline.10
Eligibility criteria
We conducted a literature review to identify eligible trials, defined as randomized clinical trials comparing statins with placebo or no treatment among patients recruited from mainland China who were diagnosed as having coronary artery disease, including stable angina pectoris and acute coronary syndrome. Acute coronary syndrome consisted of unstable angina pectoris and myocardial infarction (eg, acute myocardial infarction, history of myocardial infarction, non-ST elevation myocardial infarction, ST elevation myocardial infarction). We excluded ischemic heart failure because the benefits of statins had not been confirmed among patients with this condition.11 Seven types of statins were included: lovastatin, simvastatin, atorvastatin, rosuvastatin, pravastatin, fluvastatin, and pitavastatin. We also included clinical trials that compared statins and other drug treatments as a combination with placebo or no treatment. Clinical trials were excluded if surgical procedures, such as percutaneous coronary intervention and coronary artery bypass grafting, were conducted as part of the treatment.
We included clinical trials published as journal articles in Chinese or English until December 2019. Protocols, systematic reviews, and meta-analyses were excluded.
Literature search
We searched three English bibliographic databases (PubMed, Embase, and Cochrane Controlled Register of Trials (CENTRAL)), plus four Chinese bibliographic databases (SinoMed (formerly known as Chinese Biomedical Database), the China National Knowledge Infrastructure (CNKI), Wanfang Data, and the China Science and Technology Journal Database (VIP, http://qikan.cqvip.com)).12 The supplemental file presents the search terms. We performed the initial search on 1 January 2020 and completed an updated search on 19 October 2020. Two authors (YJ and JW) independently screened the titles or abstracts, or both, and full text articles. Discrepancies were resolved through discussion with a third author (XW).
Definition of redundant trials
In March and April 2007, two clinical practice guidelines developed by the Chinese Society of Cardiology of the Chinese Medical Association were published that strongly recommended (benefits outweigh harms based on “grade A” evidence) statin treatment for patients with stable angina pectoris and acute coronary syndrome, respectively.1314 We defined redundant trials as those that were either initiated or continued recruiting new patients at least one year after the clinical practice guidelines were published. Therefore, March 2008 was the cut-off point for stable angina pectoris and April 2008 was the cut-off point for acute coronary syndrome or a mixture of stable angina pectoris and acute coronary syndrome. The one year lag was added to allow trialists to adopt the clinical practice guidelines and take action.
The rationale behind the choice of March and April 2008 as cut-off points was further supported by two assumptions. First, cumulative meta-analysis based on eligible trials had established the benefits of statins for coronary artery disease by March and April 2008, and, second, the findings of eligible trials published before March and April 2008 were consistent, undermining the treatment uncertainties to justify subsequent clinical trials.
Correspondingly, we conducted cumulative meta-analyses with random effect to establish the time points when statins were shown to statistically significantly reduce the incidence of a combination of death, new or recurrent myocardial infarction, and revascularization. We conducted three cumulative meta-analyses for patients with stable angina pectoris, unstable angina pectoris, and myocardial infarction.15 The conclusion of eligible trials on the efficacy and safety of statins was abstracted, for example, whether statin treatment was effective or safe based on the findings.
Statistical analysis
Descriptive analysis—Among eligible trials, we report the distribution of coronary artery disease by type, type of statins, language of publication, funding source, and whether approval from an ethics committee was reported. We also calculated the number of patients enrolled, the number of patients treated in the control group, and the person years of patients treated in the control group.
Primary outcome—The primary outcome was the number of extra MACEs experienced by patients who did not receive statins in the redundant trials. To accommodate the wide range of clinical events reported by individual trials, we defined MACE broadly to include all cause mortality or cardiac related mortality, new or recurrent myocardial infarction, stroke, heart failure, revascularization, and recurrent or exacerbated angina pectoris. We estimated the extra MACEs by risk difference between the statin and control groups in individual trials—that is, the difference between the observed MACEs in the control group and the expected MACEs if patients in the control group were treated as in the statin group. When a redundant trial was initiated before the cut-off time point, we only included the patients recruited after the cut-off point, assuming a constant recruiting process. Bootstrapping was used to construct the 95% confidence intervals of extra MACEs. Supplemental file 2T reports the method in detail. The analysis was conducted in SAS 9.4.
Sensitivity analysis—We explored multiple cut-off time points to define redundant trials, including immediately, six months, two years, and five years after publication of the clinical practice guidelines, as well as the ones established by cumulative meta-analyses. When the patients in a trial were covered by multiple cumulative meta-analyses, we used the most recent time point.
Patient and public involvement
Patients or the public were not involved in the design, implementation, interpretation, or writing up of the study. The study was based on published literature without direct contact of patients or the public.
Results
Characteristics of eligible trials
In total, 2577 eligible trials had been published by December 2019 (fig 1). Of 250 810 enrolled patients, 121 722 were treated without statins in the control group for 32 428 person years. Of the 2577 trials, 1022 (39.7%) recruited patients with unspecified types of coronary artery disease, and more than a half (1474, 57.2%) evaluated atorvastatin (table 1). Most of the trials (2560, 99.3%) were published in Chinese; only 98 (3.8%) trials reported funding sources, mainly from government agencies (86, 3.3%); 272 (10.6%) reported approval from an ethics committee; and 548 (21.3%) reported MACEs. None of the eligible trials were registered in trial registries. Figure 2 presents the number of eligible trials and the number of patients treated in the control group by year of publication.
Selection of eligible trials. CENTRAL=Cochrane Controlled Register of Trials
Characteristics of eligible trials
Number of eligible clinical trials on statin treatment in patients with coronary artery disease and number of patients treated in control group by year of publication
Characteristics of redundant trials
Among 2577 eligible trials, 2045 (79.4%) were considered redundant because they were either initiated or continued recruiting after March or April 2008. In total, 207 317 (82.7%) patients were enrolled in redundant trials, of whom 101 486 were treated in the control group without statins for 24 638 person years. Overall, 962 (47.0%) of 2045 redundant trials recruited patients with unspecified types of coronary artery disease, and more than a half (1291, 63.1%) evaluated atorvastatin (table 1). Most redundant trials (2039, 99.7%) were published in Chinese; only 85 (4.1%) reported funding sources, mainly from government agencies (74, 3.6%); 265 (13.0%) reported approval from an ethics committee; and 360 (17.6%) reported MACEs.
Validation on March and April 2008 as cut-off points
The assumptions supporting March and April 2008 as the cut-off points were validated. First, cumulative meta-analyses showed that statins could statistically significantly reduce the incidence of a combination of death, new or recurrent myocardial infarction, and revascularization among patients with unstable angina pectoris by 2002 and among patients with stable angina pectoris or myocardial infarction by 2004 (supplemental files 3-5), several years earlier than March and April 2008.
Second, 375 (14.6%) of 2577 eligible trials were published before March or April 2007, all of which consistently reiterated that statins were beneficial and safe for patients with coronary artery disease. Nearly all (2573, 99.8%) the eligible trials reached this conclusion.
Primary outcome
In 360 redundant trials that reported MACEs, 22 380 patients were treated in the control group for 6766 person years. In total, 3470 (95% confidence interval 3323 to 3619) extra MACEs were reported, including 559 (95% confidence interval 506 to 612) deaths, 973 (95% confidence interval 897 to 1052) patients with new or recurrent myocardial infarction, 161 (132 to 190) patients with stroke, 83 (58 to 105) patients requiring revascularization, 398 (352 to 448) patients with heart failure, 1197 (1110 to 1282) patients with recurrent or deteriorated angina pectoris, and 99 (95% confidence interval 69 to 129) unspecified MACEs (table 2).
Number of extra major adverse cardiac events (MACEs)
Sensitivity analysis
Even allowing for a five year lag after the clinical practice guidelines were released (ie, after March or April 2012), 1478 redundant trials were identified, of which 207 reported 1913 (95% confidence interval 1801 to 2014) extra MACEs, including 275 (95% confidence interval 238 to 310) deaths, 584 (95% confidence interval 524 to 648) patients with new or recurrent myocardial infarction, 110 (88 to 132) patients with stroke, 16 (6 to 25) patients requiring revascularization, 215 (183 to 251) patients with heart failure, 683 (609 to 752) patients with recurrent or deteriorated angina pectoris, and 28 (9 to 45) unspecified MACEs (table 2).
Using 2002 and 2004 as the cut-off points established by cumulative meta-analyses, 2356 redundant trials were identified, of which 458 reported 4535 (4367 to 4702) extra MACEs, including 711 (95% confidence interval 652 to 772) deaths, 1292 (95% confidence interval 1198 to 1382) patients with new or recurrent myocardial infarction, 203 (170 to 236) patients with stroke, 130 (100 to 158) patients requiring revascularization, 495 (442 to 546) patients with heart failure, 1560 (1459 to 1665) patients with recurrent or deteriorated angina pectoris, and 144 (95% confidence interval 111 to 177) unspecified MACEs (table 2).
Discussion
Our study identified 2045 redundant clinical trials conducted in mainland China, with 3470 extra MACEs experienced among participants with coronary artery disease who were not treated with statins in the trials. The unexpected scale of redundancy raises concerns over the ethical foundation of clinical research in mainland China.
Such a large scale of redundant trials might arise when multiple system failures occur. First, investigators might not be trained to consider existing evidence before initiating clinical trials. We did not formally assess how the authors justified their trials, but, as others have reported,1617 we noticed that few redundant trials cited systematic reviews or previous trials, suggesting a lack of appreciation about previous evidence.1819 Second, clinical practitioners are under pressure to produce publications.2021 This could also explain why only 20% of included trials reported clinical events, because it is easier and faster to conduct clinical trials on surrogate laboratory outcomes. Third, when ethics approval was reported, the committees reviewing the trial protocols failed to check the scientific foundation and protect participants from enrolling in harmful trials.322 Currently, ethics committee approval required by the China State Food and Drug Administration only covers clinical trials for the marketing license of drugs.23 Moreover, many redundant trials were conducted in primary care settings where approval by an ethics committee was not feasible. Therefore, some redundant trials might fail to obtain ethical approval. Fourth, some journal editors fail to evaluate the scientific value of the publications adequately. By accepting manuscripts from such trials, those journals provided a means for redundant trials to be published, thereby validating the redundancy as acceptable. Those journals might be more interested in pursuing profits rather than scientific merits.24 Fifth, only a small proportion of trials reported the funding source, most of which were either central or local government agencies, which failed to evaluate the scientific value of the redundant trials.25 Last, none of the included trials were registered in trial registries, a function of which is to reduce resource waste by declaring and presenting what has been conducted and what is currently being done in the clinical trial community.26
We also confirmed a large gap between Chinese and English literature.27 Most eligible trials were published in Chinese and only indexed in Chinese bibliographic databases. This might explain why systematic reviews that fail to search Chinese bibliographic databases often fail to include Chinese trials, even when a large amount of Chinese trials on the clinical question exist.28
Our findings might only cover a fraction of the problem. First, nearly 80% of eligible trials did not report clinical events, even with follow-up of many years. It is unclear whether no clinical events occurred in those trials, the events were not collected, or the events were simply not reported. Second, clinical trials with limited follow-up might not capture the long term benefits of statins, leading to underestimated treatment effects and extra MACEs.29 Third, the evidence is scarce on the publication rate of clinical trials in mainland China, and it is challenging to estimate how many trials might have been conducted but not published. Fourth, our study was limited to only one drug class, one disease condition, one type of comparator, and randomized clinical trials. Future studies are needed to evaluate the existence of research redundancy over the entire clinical trial community and the general biomedical research in mainland China.
Comparison with other studies
Previous studies have suggested that external evidence, including from systematic reviews, might be overlooked by researchers before new clinical trials on similar topics are initiated.303132 However, even if systematic reviews confirm the benefits of a treatment in the early stages of clinical research, it is challenging to know whether future trials could modify, or even reverse, that early conclusion because early trials tend to exaggerate positive findings whereas later trials often report reduced effect sizes.33 Consequently, it might not be appropriate to label some clinical trials as redundant or as unethical. In our study, these concerns may be alleviated because the eligible trials consistently reaffirmed the benefits of statins among patients with coronary artery disease. This is unusual and differs from other studies using cumulative meta-analyses, in which a proportion of subsequent trials were in favor of the control group.531 The consistency of findings undermines the treatment uncertainties required to support the clinical equipoise to initiate subsequent clinical trials on similar topics.
In general, five arguments support our classification of 2045 trials as redundant. First, the release of clinical practice guidelines strongly recommending statins to all patients with coronary artery disease based on “abundant evidence” should disrupt the clinical equipoise required to justify starting new trials; second, the results of our cumulative meta-analyses suggested that the cut-off points of March and April 2008 were conservative; third, the consistency of the results from eligible trials undermined the treatment uncertainties; fourth, all eligible trials were conducted in mainland China so the redundancy could not be justified by the unsatisfactory representativeness of Asian populations recruited in clinical trials in Western countries34; and fifth, it is noticeable that the landmark clinical trials outside China, such as the 4S trial (Scandinavian Simvastatin Survival Study) published in 1994,35 were overlooked by redundant trials.
Limitations of this study
Our study has several limitations. First, we did not conduct a risk of bias assessment because our primary goal was to estimate the number of extra clinical events rather than the precise treatment effect at the population level. In case of information bias (eg, lack of masking), the extra clinical events might still be attributable to the treatment assignment—for example, patients in the control group did not receive as much attention as patients in the treatment group, therefore this would not invalidate our findings.36 On the other hand, selection bias, which might arise from inappropriate randomization,37 was not addressed. The extent to which this bias might impact our findings is unclear. Second, there are three reasons why we might have underestimated the extra clinical events: we only included journal articles in our study (more redundant trials could have been identified if we expanded our search to other publications, such as gray literature); we excluded trials that were not clearly specified as randomized clinical trials, but there are likely many non-randomized clinical trials that also assigned participants to statins or placebo; and it is likely that some redundant trials failed to report clinical events, but we believe it is inappropriate to extrapolate the number of events by assuming a similar incidence between the trials reporting and not reporting such events. Third, we did not evaluate the quality of the two clinical practice guidelines as the anchor to define redundancy. It is possible that some researchers do not consider those guidelines as trustworthy, even if they were developed by a leading national organization and published in a reputable Chinese journal.
We found redundant clinical trials initiated as recently as 2018. More redundant trials will be published in the near future unless actions are taken by stakeholders in mainland China, which might include, but are not limited to, altering the method to evaluate the academic performance of researchers, legislating the responsibilities and requirements of ethic committees, reaching consensus about publishing requirements for journals, adapting the funding system, and mandating trial registration. Our findings should enlighten researchers in other countries, especially developing countries that share certain characteristics with mainland China. However, the discussion should be on a nation specific, case-by-case basis because of the varied incentives behind research redundancy.
Conclusions
More than 2000 redundant clinical trials on statins among patients with coronary artery disease were identified from mainland China. More than 3000 extra MACEs, including nearly 600 deaths, were experienced by participants not treated with statins in these trials. The scale of redundancy necessitates urgent reform to protect patients.
What is already known on this topic
Redundant clinical trials waste resources and potentially harm patients who might be denied effective treatment, especially in the setting of placebo controlled trials
A concern is that redundancy has become a serious challenge in clinical trials from mainland China, the biggest producer of scientific publications
What this study adds
In mainland China, more than 2000 redundant clinical trials have been initiated since clinical practice guidelines recommended statins to all patients with coronary artery disease in 2008
More than 3000 extra major adverse cardiac events, including nearly 600 deaths, were experienced by participants not treated with statins in these trials, including 973 patients with myocardial infarction and 161 patients with stroke
The redundancy of clinical trials necessitates urgent reform to protect patients
Footnotes
Contributors: YJ and JW conceived the study. They contributed equally to this manuscript as joint first authors. YJ, JW, and KAR designed the study. YJ, JW, LR, XW, and YW acquired and analyzed the data, which was interpreted by YJ, JW, RQ, SE, DDC, XW, LR, YM, and KAR. YJ drafted the manuscript. JW, RQ, SE, DDC, XW, LR, YM, and KAR revised the article critically for important intellectual content. YJ, JW, RQ, SE, DDC, XW, LR, YM, and KAR read the final version of the manuscript and gave approval for it to be published. KAR had access to the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. YJ is the guarantor.
Funding: None.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organization for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: Technical appendix, statistical code, and dataset available from the corresponding author at krobin@jhmi.edu.
The lead author (YJ) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
Dissemination to participants and related patient and public communities: We did not include members of the public mainly because this study was not funded. However, the idea of how redundant research harms patients was well received by our friends who did not work in the specialty of biomedical research. Upon publication, we will disseminate this research through social media such as WeChat to encourage further discussion in the public.
Provenance and peer review: Not commissioned; externally peer reviewed.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.