Comparative effect sizes in randomised trials from less developed and more developed countries: meta-epidemiological assessment
BMJ 2013; 346 doi: https://doi.org/10.1136/bmj.f707 (Published 12 February 2013) Cite this as: BMJ 2013;346:f707
- Orestis A Panagiotou, research associate1,
- Despina G Contopoulos-Ioannidis, clinical associate professor23,
- John P A Ioannidis, director4, C F Rehnborg professor in disease prevention, professor of medicine, and professor of health research and policy5
- 1Clinical Trials and Evidence-Based Medicine Unit, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, University Campus, Ioannina, Greece
- 2Division of Infectious Diseases, Department of Pediatrics, Stanford University School of Medicine, Stanford, CA, USA
- 3Health Policy Research, Palo Alto Medical Foundation Research Institute, Palo Alto, CA, USA
- 4Stanford Prevention Research Center, Department of Medicine
- 5Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305, USA
- Correspondence to: J P A Ioannidis jioannid{at}stanford.edu
- Accepted 30 January 2013
Abstract
Objective To compare treatment effects from randomised trials conducted in more developed versus less developed countries.
Design Meta-epidemiological study.
Data sources Cochrane Database of Systematic Reviews (August 2012).
Data extraction Meta-analyses with mortality outcomes including data from at least one randomised trial conducted in a less developed country and one in a more developed country. Relative risk estimates of more versus less developed countries were compared by calculating the relative relative risks for each topic and the summary relative relative risks across all topics. Similar analyses were performed for the primary binary outcome of each topic.
Results 139 meta-analyses with mortality outcomes were eligible. No nominally significant differences between more developed and less developed countries were found for 128 (92%) meta-analyses. However, differences were beyond chance in 11 (8%) cases, always showing more favourable treatment effects in trials from less developed countries. The summary relative relative risk was 1.12 (95% confidence interval 1.06 to 1.18; P<0.001; I2=0%), suggesting significantly more favourable mortality effects in trials from less developed countries. Results were similar for meta-analyses with nominally significant treatment effects for mortality (1.15), meta-analyses with recent trials (1.14), and when excluding trials from less developed countries that subsequently became more developed (1.12). For the primary binary outcomes (127 meta-analyses), 20 topics had differences in treatment effects beyond chance (more favourable in less developed countries in 15/20 cases).
Conclusions Trials from less developed countries in a few cases show significantly more favourable treatment effects than trials in more developed countries and, on average, treatment effects are more favourable in less developed countries. These discrepancies may reflect biases in reporting or study design as well as genuine differences in baseline risk or treatment implementation and should be considers when generalising evidence across different settings.
Introduction
The predominant share of the global burden of disease is concentrated in less developed countries.1 However, until recently relative few trials were being conducted in these nations.2 3 Evidence on the management of many diseases affecting less developed countries often had to be tentatively extrapolated from studies performed in more developed countries with a longer standing tradition of conducting clinical research. This situation is currently changing. As participation rates in clinical trials decrease and cost increases in Western countries,4 5 6 many organisations providing contracts for research are focusing on eastern Europe,7 Asia,8 and South America,9 where the cost of recruiting participants is low. By 2015, for example, 15% of clinical trials are expected to be conducted in India.8 10 11 With the globalisation of international health,12 results of studies done in countries without a longstanding tradition of clinical research are becoming important to clinical practice in more developed nations.2
Trials carried out in less developed countries may differ in important aspects from those done in countries with stronger traditions in clinical research.13 Firstly, publication dynamics and biases may differ.14 Investigators in less developed countries may face a higher barrier against publication of “negative” results. Trials may remain unpublished or appear in domestic or local journals,15 16 and language biases may exist.17 18 19 Some national literatures on specific disciplines contain only significant results (for example, Chinese and Russian studies on acupuncture).20 Secondly, treatment effects may genuinely vary between countries owing to differences in study populations, baseline risk, concomitant diseases, background management, and clinical settings.
We performed a large scale assessment of meta-analyses on topics with randomised evidence from more developed and less developed countries. We assessed how often randomised trials performed in these countries with different levels of development and different traditions in clinical research give different results, whether treatment effects are systematically larger in one setting than the other, and whether discordant effects are the result of bias or genuine differences.
Methods
Definition of countries
There are varying definitions and no perfect consensus on what countries should be included in the lists of “more developed” and “less developed,” so their separation is not absolute.21 22 These categorisations try to take into account the per capita income but also other factors such as the composite human development index. For research purposes it is also helpful to know whether a country has a longstanding tradition in modern clinical research, uses critical scientific thinking, and in general applies empirical modern methods. We considered more developed countries to be those with both longstanding established market economies and longstanding traditions in clinical research.22 Such countries included the United States, Canada, Australia, New Zealand, Israel, and Japan, and western European countries. All other countries except for those in eastern Europe were considered as less developed.
Our definition is consistent with the list of less developed countries of the International Monetary Fund, except that Israel (considered less developed until 2001 according to the International Monetary Fund) is classified among the more developed countries, given its strong longstanding research tradition, and eastern European countries (less developed according to the International Monetary Fund, except for Slovenia after 2007, the Czech Republic and Slovakia after 2009, and Estonia after 2011) are excluded. Eastern European countries were excluded from our analyses as they may have unique differences7 and are considered to be a separate group of countries in transition.
We also performed sensitivity analyses where we excluded four Asian “tigers” (Hong Kong, Taiwan, Singapore, and South Korea) that evolved from less developed countries into advanced economies according to the International Monetary Fund in 1997, although their tradition of clinical research is not as longstanding as the main more developed countries. Sensitivity definitions excluding nations with a high per capita income but no tradition of clinical research (for example, several Arabian nations) yielded similar results, since few trials were identified that had been done in these countries.
Eligible meta-analyses
We identified meta-analyses that included data from one or more randomised trials conducted in a less developed country and one or more randomised trials conducted in a more developed country. Trials were classified on the basis of the countries in which participants were recruited; countries were considered with the names (for example, United Kingdom) or geographical indicators (for example, North America) as these were reported in the eligible Cochrane review. For consistency we focused on mortality, the most serious outcome.
We searched the Cochrane database of systematic reviews (last update 27 August 2012) using terms for mortality (death OR mortality OR survival) in the title, abstract, or keywords. The reviews included in this database are considered to be thorough in searching for eligible studies.23 We excluded protocols and reviews that had been withdrawn, had no statistical synthesis on mortality, and had no country information of individual trials. Only reviews including randomised and pseudorandomised trials were eligible. Whenever a systematic review contained two or more different pertinent intervention comparisons we considered these separately.
We also excluded reviews in which all the randomised trials from less developed or more developed countries had zero deaths. Multicentre international trials were eligible if all the countries were either less developed or more developed.
Data extraction
From each eligible trial we extracted the publication year, country of origin, number of participants, and number of deaths in each trial arm. We selected deaths from all causes; if, however, there were no data for all cause mortality we used cause specific mortality. Whenever there were several forest plots on mortality, we selected the one that reported overall data rather than subgroups. Whenever separate forest plots pertained to non-overlapping events for the same participants (for example, stillbirths and neonatal deaths), we selected whichever analysis included more deaths. For many conditions deaths are uncommon, conferring low power to show differences in effect sizes for mortality. Therefore, for each eligible topic we also examined separately the meta-analysis on the primary binary outcome (mortality or other). Whenever several eligible binary outcomes existed, we selected whichever had the largest number of studies regardless of whether this was mortality. Only meta-analyses with data from one or more trials from a less developed country and one or more trials from a more developed country were eligible. For each eligible trial we recorded the publication year, country, characteristics of participants, and events per arm.
Two investigators (OAP, DGCI) independently extracted data. Any disagreements were resolved after discussion with the third investigator (JPAI).
Statistical analysis
As a metric of relative risk, we used the odds ratio for our analyses when this could be estimated from available 2×2 tabular data for each trial. When this estimation was not possible we used the hazard ratio or risk ratio estimates as provided in the forest plots. Effect estimates for each trial were coined consistently to represent the odds of death or unfavourable primary outcome for the experimental (newer) intervention versus control. When survival or favourable primary outcomes were reported, we took the complementary mortality or unfavourable primary outcome event counts.
Whenever two or more trials per country group were included in a forest plot, we synthesised them by fixed effects and random effects models.24 Fixed effects assume a common effect across the combined studies, whereas random effects assume that the true treatment effect may differ in each trial, and the summary aims at identifying an average treatment effect. Subsequently, for each topic we calculated the relative relative risk, with corresponding 95% confidence intervals, by dividing the summary relative risk from trials in more developed countries by that in less developed countries on the same topic. A relative relative risk >1.00 means that the experimental intervention has more favourable outcomes in trials from less developed countries versus more developed countries. Furthermore, we calculated the summary relative relative risk across all topics of more developed versus less developed countries, by synthesising the relative relative risks of more developed versus less developed countries from each individual topic using a random effects model.25 Heterogeneity was probed using the Q statistic and I2 metric with corresponding 95% confidence intervals.26 27 In each eligible forest plot we also examined whether estimated intervention effects in smaller studies differed beyond chance from those estimated in larger studies (small study effects) using the Harbord’s test28 when 2×2 data were available and the Egger’s test29 otherwise; both test are considered to be significant for P<0.10.
For each topic where the results of trials from less developed countries differed beyond chance from trials from more developed countries, we examined whether there was evidence for small study effects. We also evaluated the constituent trials to examine whether there was any reason for anticipating ceiling effects related to the standard of care and the mode that an intervention was implemented in less developed or more developed countries—for example, whether an intervention was difficult to apply or required other concomitant interventions or background care to be effective. Moreover, we examined for each topic whether the baseline event risk in the control arms differed significantly between the two country groups by synthesising the baseline risks per country group by random effects using the Freeman-Tukey arcsin transformation.30 Such differences may mean that patients in these two settings vary in severity of disease, concomitant care, or other risk factors that can influence outcomes. Finally, for all topics where there were significant differences in treatment effects in more developed versus less developed countries, we also examined the risk of bias in the reported study design (mode of randomisation, allocation concealment, blinding, intention to treat, losses to follow-up). Quality deficits for these characteristics are associated with potential inflation in treatment effects in randomised controlled trials.31
In estimating the summary relative relative risk, sensitivity analyses were performed limited to topics where a nominally significant treatment effect for mortality had been found, when all trials were combined. Moreover, old trials may have different characteristics and less relevance to current practice.32 Therefore, we also performed sensitivity analyses excluding meta-analyses with any trials published before 1970. Finally, we performed sensitivity analyses excluding trials from countries that evolved from less developed to more developed countries.
All analyses were done in Stata version 11.2. P values are two sided.
Results
Eligible meta-analyses for mortality outcomes
The electronic search identified 2025 reviews. After exclusions (see supplementary fig A1), 131 eligible systematic reviews with 139 meta-analyses for mortality outcomes were considered (see supplementary table A1).
The 139 meta-analyses included 1297 eligible trials (312 were conducted in less developed countries and 985 in more developed countries, see supplementary fig A2). The median publication year of eligible trials was 1997 (interquartile range 1990-2002). Each meta-analysis included a median of 13 trials (interquartile range 8-19) and 2856 participants (interquartile range 1355-11593). Trials from more developed countries did not have substantially larger sample sizes (median 117; interquartile range 54 to 319) than trials from less developed countries (105; 54 to 365): P=0.93, Mann-Whitney U-test.
By fixed effects synthesis, 31 meta-analyses favoured (P<0.05) the experimental or new intervention, five the control, and 103 showed no statistically significant difference. By random effects synthesis, the respective numbers were 27, 1, and 111. Significant evidence was found for small study effects in a total of 16/139 (12%) meta-analyses: 15/133 (11%) meta-analyses with Harbord’s test and 1/6 meta-analyses with Egger’s test.
Significant differences in effect sizes for mortality
By using fixed effects to combine the relative risks from individual trials within the same country group, we identified 11 topics where the treatment effects between trials from more developed and less developed countries differed beyond chance (95% confidence intervals for relative relative risk excluded 1.00). For all these topics the experimental intervention had significantly less favourable results in the more than less developed countries (relative relative risk >1.00, table 1⇓). Antenatal corticosteroids33 noticeably reduced fetal and neonatal deaths when given to women at risk of preterm birth in trials conducted in less developed countries, but had a modest, non-nominally significant benefit in trials conducted in more developed countries (fig 1⇓). A similar pattern was observed for corticosteroids in the treatment of sepsis or septic shock (fig 2⇓),34 systemic antifungals in non-neutropenic critically ill patients (fig 3⇓),35 calcium antagonists in aneurysmal subarachnoid haemorrhage (fig 4⇓),36 intravenous immunoglobulin for preventing infection in preterm or low birthweight infants (fig 5⇓),37 and transarterial embolisation in unresectable hepatocellular carcinoma (fig 6⇓).38 Moreover, antioxidants, given for diverse conditions (fig 7⇓),39 or specifically for prevention of gastrointestinal cancers (fig 8⇓),40 and postoperative radiotherapy for non-small cell lung cancer (fig 9⇓)41 conferred a significantly increased risk of mortality in trials from more developed countries but not in trials from less developed countries (fig 1). Additionally, admission to hospital for bed rest for women with multiple pregnancy (fig 10⇓)42 tended to increase mortality in trials from more developed countries and decrease mortality in trials from less developed countries. Finally, altered fractionation radiotherapy compared with conventional radiotherapy resulted in nominally significant decreases in total mortality from oral cavity and oropharyngeal cancer in both less and more developed countries, although this was larger in trials from less developed countries (fig 11⇓).43
Fig 1 Fetal and neonatal deaths with antenatal corticosteroids in women at risk of preterm birth
Fig 2 All cause mortality at 28 days with corticosteroids for treating sepsis and septic shock
Fig 3 Mortality with systemic antifungals in non-neutropenic critically ill patients
Fig 4 Case fatality with calcium antagonists in aneurysmal subarachnoid haemorrhage
Fig 5 All cause mortality with intravenous immunoglobulin for preventing infection in preterm or low birthweight infants
Fig 6 All cause mortality with transarterial embolisation in unresectable hepatocellular carcinoma. Only the effect estimates from each trial were retrievable for the corresponding meta-analysis
Fig 7 Mortality with antioxidants for preventing various diseases
Fig 8 Mortality with antioxidants for preventing gastrointestinal cancers
Fig 9 Mortality after postoperative radiotherapy for non-small cell lung cancer
Fig 10 Perinatal deaths with admission to hospital for bed rest for women with multiple pregnancy
Fig 11 Total mortality with altered fractionation radiotherapy for oral cavity and oropharyngeal cancer. Only the effect estimates from each trial were retrievable for the corresponding meta-analysis
Statistically significant differences in treatment effects in less developed countries versus more developed countries
Evidence for small study effects was strong in the meta-analyses of steroids and antifungals and possibly also antenatal corticosteroids. The interventions were simple and easy to administer in diverse settings, regardless of the availability of other concomitant interventions and standards of care. The one possible exception was postoperative radiotherapy, where better outcomes might be expected in countries with higher standards of technology, although, if anything, the opposite was seen.
The baseline risk of death was significantly higher in less developed countries in the meta-analyses of antenatal corticosteroids (33% v 16%), corticosteroids for sepsis (76% v 34%), systemic antifungals for non-neutropenic critically ill patients (54% v 28%), and intravenous immunoglobulin in preterm infants (19% v 13%), whereas it was significantly higher in more developed countries in meta-analyses of preventive antioxidants for various conditions (8% v 4%) or for gastrointestinal cancers (12% v 5%), and postoperative radiotherapy for non-small cell lung cancer (39% v 31%) (see supplementary table A2).
Table 2⇓ shows the number of trials from the countries that had unclear or high risk of bias for randomisation sequence generation, allocation concealment, and blinding for the topics where significant differences in the treatment effects for mortality were documented. It is difficult to make comparisons within single topics, given the limited number of trials. However, summing the data across all trials, the proportion of trials with an unclear or high risk of bias was not significantly different in trials from more developed versus less developed countries for sequence generation (27% v 33%, P=0.45), allocation concealment (27% v 30%, P=0.68), or blinding (39% v 33%, P=0.53).
Risk of bias in studies from more developed versus less developed countries based on reported features
Summary of comparisons for mortality
When summary relative risks from trials within each country group were synthesised by fixed effects, on average the results from more developed countries were significantly less favourable than those from less developed countries, with a summary relative relative risk of 1.12 (95% confidence interval 1.06 to 1.18, P<0.001, I2=0%, 95% confidence interval 0% to 21%, Q statistic P=0.709). When data were synthesised within each country group by random effects, inferences were similar (1.08, 1.02 to 1.14, P=0.005, I2=0%, 0% to 21%, Q statistic P=0.922), but confidence intervals were wider and only the differences for antenatal corticosteroids, systematic antifungals, calcium antagonists, transarterial embolisation, and altered fractionation radiotherapy were beyond chance. Additionally, summary relative relative risks per fixed effects were 1.10 (1.04 to 1.18, P=0.002, I2=10.5%, 10% to 42%, Q statistic P=0.303) per fixed effects and 1.11 (1.01 to 1.21, P=0.023) per random effects for more developed countries versus China, and 1.21 (1.13 to 1.30, P=0.003, I2=92.3%, 90% to 94%, Q statistic P<0.001) per fixed effects and 1.26 (0.91 to 1.74, P=0.158) per random effects for more developed countries versus India (the two less developed countries with the largest number of trials).
Results were similar when analyses were limited to the 36 meta-analyses that had found nominally significant mortality effects overall at 1.15 (1.08 to 1.23, P<0.001) per fixed effects and 1.17 (1.06 to 1.30, P=0.002) per random effects, I2=17%, 0% to 45%, Q statistic P=0.19, fig 12⇓), the 124 meta-analyses where all trials had been published after 1970 (1.14, 1.08 to 1.21, P<0.001, I2=0%, 0% to 22%, Q statistic P=0.81), and when excluding from calculations the four Asian countries (Hong Kong, Taiwan, Singapore, and South Korea) evolving into more developed countries (1.12, 1.06 to 1.18, P<0.001, I2=0%, 0% to 22%, Q statistic P=0.624).
Fig 12 Relative relative risks and 95% confidence intervals for mortality outcomes between more developed and less developed countries in meta-analyses with nominally significant effects. Relative relative risk estimates and 95% confidence intervals are shown for the 36 topics for which the respective meta-analyses had found nominally significant effects overall per fixed effects synthesis
Eligible meta-analyses for primary binary outcomes
Overall, 127 meta-analyses had primary binary outcomes and available data from at least one more developed country and at least one less developed country (see supplementary table A3); for 58 of those the primary binary outcome was mortality. These 127 meta-analyses included a total of 1312 trials; 319 conducted in less developed countries (median sample size 121, interquartile range 58-318) and 993 in more developed countries (114, 53-310). The median number of trials per meta-analysis overall was 14 (interquartile range 9-21).
By fixed effect synthesis 55 meta-analyses were overall in favour (P<0.05) of the experimental intervention when all trials in the respective forest plot were considered, seven favoured the control, and 65 showed non-significant differences. By random effects, the respective numbers were 46, 5, and 76. In 26/127 (20%) meta-analyses evidence for small study effects was significant: 24/122 with Harbord’s test and 2/5 with Egger’s test.
Significant differences in effect sizes for any primary outcome
Combining by fixed effects model, treatment effects for the experimental intervention in trials from more developed and less developed countries, respectively, the relative differences were beyond chance (relative relative risk and 95% confidence intervals excluding 1.00) in 20 cases, of which six pertained to mortality and 14 to non-mortality outcomes (table 1; also see supplementary fig A3). In 15 of the 20 cases, results were more favourable (or less unfavourable) in trials from less developed countries (relative relative risk >1.00).
Antibiotic prophylaxis for bacterial infections in cirrhotic patients with upper gastrointestinal bleeding,44 lipid lowering regimens for peripheral arterial disease of the leg,45 vaccination for pneumococcal infection in adults,46 and intravenous immunoglobulin for sepsis in preterm or low birthweight infants37 had beneficial effects on the respective outcomes in trials from both less developed and more developed countries, although the benefit was considerably larger in less developed countries. Antioxidants47 and vitamin E48 prevented pre-eclampsia, antioxidants prevented gastrointestinal cancers,40 and β blockers decreased the rate of caesarean sections49 in trials from less developed countries but not in trials from more developed countries. Finally, the reduced number of antenatal visits or goal oriented visits only marginally increased the risk of preterm birth in more developed countries, but showed a small non-significant reduction in trials from less developed countries.50
Evidence of small study effects was strong for the meta-analyses of antioxidants for pre-eclampsia, antioxidants for gastrointestinal cancers, and intravenous immunoglobulin for sepsis. All interventions were simple and easy to apply in diverse settings and background standards of care. The baseline risk of the outcome was significantly higher in less developed countries in the meta-analyses of corticosteroids for treating sepsis (76% v 34%) and systemic antifungals for non-neutropenic critically ill patients (54% v 28%), and it was significantly higher in more developed countries in the meta-analyses of preventive antioxidants in various conditions (8% v 4%) (see supplementary table A2).
For the remaining five non-mortality related primary outcomes, the treatment effects were more beneficial in trials from more developed countries. These included antiplatelet agents to prevent proteinuric pre-eclampsia,51 probiotics in hepatic encephalopathy,52 isoniazid prophylaxis against active tuberculosis in people not infected with HIV,53 prophylactic fluconazole for invasive fungal infections in very low birthweight infants,54 and rotavirus vaccine for the prevention of diarrhoea.55
For the meta-analysis of antiplatelets, evidence for the presence of small study effects was strong and the larger trials showed no clear benefit in either country group. All four interventions were simple and easy to administer in diverse settings. The risks in the control groups at baseline were significantly lower in more developed countries for active tuberculosis (2% v 15%), gastrointestinal cancer (2% v 3%), and rotavirus diarrhoea, whereas the risk was higher in more developed countries for bacterial infections in cirrhotic patients with upper gastrointestinal bleeding (43% v 28%) (see supplementary table A2). The proportion of trials with an unclear or high risk of bias was not significantly different in trials from more developed countries for sequence generation (42% v 45%, P=0.76), allocation concealment (52% v 52%, P=0.95), or blinding (51% v 41%, P=0.22, table 2).
When the relative relative risks were synthesised across all 127 topics, there was some between topic heterogeneity with fixed effects summary relative odds ratio 1.07 (95% confidence interval 1.02 to 1.12, P=0.009) and random effects summary relative relative risk 1.09 (1.01 to 1.18, P=0.034, I2=37%, 95% confidence interval 22% to 49%, Q statistic P<0.001).
Discussion
We evaluated 139 meta-analyses with mortality outcomes, which included trials performed in less developed and more developed countries. In 11 cases, experimental interventions had significantly more favourable results in less developed countries than in more developed ones, whereas the opposite was never seen. On average, trials conducted in less developed countries had 1.12-fold more favourable effect sizes than trials done in more developed countries. When focusing only on interventions with an overall statistically significant impact on mortality the difference was 1.15-fold. Given that even effective interventions rarely achieve more than 1.15-fold to 1.20-fold reductions in the relative risk of mortality,56 relative differences of 1.10-fold to 1.15-fold may confound the presence or not of a genuine effect of many interventions. When we considered any primary binary outcome, in 20 topics treatment effects varied significantly based on the country group of included trials, and in 15/20 results were more favourable in less developed countries. Totally ineffective treatments may spuriously seem effective based on research published from less developed countries. As an increasingly larger share of clinical research is being done in less developed countries without strong research traditions, this may create a flood of spurious evidence.
Possible explanations
Given the systematic preponderance of more favourable results in trials from less developed countries, one potential explanation is that the available randomised evidence from developed countries is more biased. An empirical evaluation of 307 published randomised trials from China, 117 from India, and 304 from Western countries showed that Indian and Chinese trials were of much lower methodological quality.57 Another empirical study highlighted that authors of Chinese trials often mislabelled basic study designs. Among 3137 studies indexed in the China national knowledge infrastructure database and claimed by their authors to be randomised, only 207 were indeed randomised.58 Most Chinese trials do not adhere to the CONSORT guidelines for reporting59 and many trials from less developed countries are not registered in ClinicalTrials.gov or even the composite World Health Organization trials registry.60 Trials from less developed countries tend to report on average more significant results.20 57 For example, such publication bias has been previously seen for Chinese (but not Indian) trials.20 57 61
Publication bias or selective analysis and outcome reporting biases62 63 may be influential in shaping this picture. A higher barrier to publication for authors from less developed countries that do not have a longstanding tradition in clinical research may further boost selective reporting.17 Of course mortality is a hard endpoint and more difficult to manipulate than other endpoints, but even for mortality, selective analysis may achieve inflated effects, as recently shown by corticosteroid trials.64 The presence of patterns showing small study effects is also suggestive (not conclusive) of selective reporting biases.65 Small study effects may also influence the literature in nations with strong traditions of running clinical trials. For example, this may be the case for antiplatelet agents to prevent proteinuric pre-eclampsia,51 where small trials suggest substantial benefits (more so in more developed than less developed countries), but the largest trials66 67 in both more and less developed countries have shown no benefits. Large, well conducted trials are needed to probe the claims for country specific major benefits and they may demonstrate that many of these claims are spurious. For example, after the publication of the examined Cochrane reviews, a recent large trial68 conclusively found no benefit from antioxidants in the prevention of pre-eclampsia (odds ratio 1.00), as opposed to the extremely large benefit that previous small trials had suggested (0.38).
Additionally, in the topics where results between more and less developed countries differed, there was no overall pattern of having a higher or lower proportion of trials with unclear or high risk of bias in sequence generation, allocation concealment, or blinding, when research was compared between more developed and less developed countries. There is some evidence that these quality deficits are associated with inflated treatment effects, although the impact is lesser when the outcome is mortality.31 Although we cannot exclude the possibility that these quality aspects may have played a role in explaining the difference in some specific topics, they do not seem to be the main answer for the discrepancies overall. It should also be acknowledged that reported quality may not necessarily reflect the true quality of trials.69
Differences in treatment effects in less developed versus more developed countries may also be due to genuine differences rather than to biases. Low income and middle income countries face substantial financial barriers to the total healthcare budget,70 which may limit the implementation of expensive interventions.71 This might hold especially true for trials designed by the same body (institute, industry, etc) and conducted in more and less developing countries, as study quality and biases are usually not expected to be different, except maybe for selective reporting of negative results. However, we did not identify any discrepancies where the implicated intervention was expensive or difficult to administer and its efficacy may have depended largely on sophisticated background standards of care. The one exception was postoperative radiotherapy, but then the observed benefit was larger in less developed countries, a paradox that suggests that bias is a more likely explanation than differences in standards of care and technological ability. Trials with different results sometimes studied populations with different baseline risks. For example, mortality in newborns is on average higher in less developed countries72 and we cannot exclude the possibility that corticosteroids may result in a larger benefit in these locations. Similarly, the larger benefit of isoniazid prophylaxis in more developed countries may be explained by an increased burden of multidrug resistant tuberculosis,73 lower rates of treatment compliance,74 and limited access to healthcare74 in less developed countries.
Potential limitations of the study
Some caveats should be acknowledged. Firstly, we worked with available meta-analyses that may have already removed some biases from the primary literature. Meta-analysts may have contacted the authors of primary trials and obtained outcome information not reported in published reports, or they may have standardised outcomes, for example, to include all cause mortality and all available follow-up, whereas primary papers may have focused on subset analyses or other secondary analyses such as cause specific deaths.64 Thus bias may be larger in the reports of primary trials than that seen in meta-analysis based data. Secondly, agreement in treatment effects between more developed and less developed countries does not necessarily mean that both estimates are correct; occasionally both may be equally biased. Some recorded treatment effects in meta-analyses may simply reflect bias.75 For example, some investigators have argued convincingly that pneumococcal vaccination is ineffective in adults and that the apparent benefits in preventing pneumonia in the respective meta-analysis (seemingly larger in less developed countries) are entirely spurious.76 Thirdly, the endpoints of primary trials may not be the same as the respective primary endpoints in meta-analyses, and mortality might not be the primary endpoint for several considered trials. Even so, mortality is a major outcome and trials with favourable mortality results should attract attention regardless of whether this was a primary or secondary endpoint. In fact, an unexpectedly large number of small trials claim significant differences in mortality.77 Fourthly, not all organisations agree on what countries are less developed, and the status of countries has changed over time, with several previously less developed countries adopting market economies. However, these countries still can be separated from countries with longstanding traditions in clinical research, and sensitivity analyses using different definitions yielded similar results. We cannot separate whether per capita income or tradition in clinical trials research is the decisive factor that makes the difference, since few trials were done in countries without strong longstanding traditions in clinical research, where per capita income has increased dramatically in the past decades. Finally, some trials performed in less developed countries may be designed and coordinated by investigators in more developed countries. If anything this would tend to diminish differences between the two groups. Nevertheless, none of the trials implicated in the seven topics with statistically significant differences in mortality had such collaborative patterns. Moreover, it is possible that industry sponsorship may also affect the results of trials, in particular for expensive interventions where large markets are at stake. However, most of the interventions where discrepancies were identified were not expensive and sponsors would not have major invested interests.
Conclusions and implications for future research
Overall, in a globalised world, evidence from less developed countries will increasingly influence decisions in more developed countries and vice versa. It is important to generate randomised evidence in diverse settings including populations with differences in baseline risk, comorbidities, and access to healthcare. It is also important to improve the quality and minimise the biases of randomised trials around the world. Biases could be reduced through more thorough registration of trials from less developed countries, strengthening ethical standards,78 and a global view in the design and interpretation of the overall clinical research agenda.79 Meta-analyses of the available evidence can routinely explore differences and potential explanations thereof for trials performed in countries with different economies and traditions of clinical research. This information should be taken into consideration in guidelines and in the adoption of these interventions.
What is already known on this topic
An increasing number of trials are performed in less developed countries with no longstanding tradition in clinical research
It is unclear whether the results of trials in more developed versus less developed countries are similar for the same intervention and conditions
What this study adds
Randomised trials from less developed countries in a few cases show significantly different treatment effects from randomised trials in more developed countries
The reported treatment effects are larger in the less developed countries
These discrepancies may often reflect biases as well as genuine differences and should be taken into account when generalising evidence across different settings
Notes
Cite this as: BMJ 2013;346:f707
Footnotes
Contributors: JPAI conceived the original idea. OAP, DGC-I, and JPAI designed the study. OAP and DGC-I identified the eligible reviews and performed the data extraction. OAP and JPAI performed the statistical analyses. OAP, DGC-I, and JPAI interpreted the data and wrote the manuscript. All authors have critically commented on and approved the final version of the manuscript. JPAI is the guarantor. All authors, external and internal, had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.
Funding: This study received no funding.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: The statistical code and datasets are available from the corresponding author at jioannid@stanford.edu.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.