Intended for healthcare professionals

Analysis

PREDIMED trial of Mediterranean diet: retracted, republished, still trusted?

BMJ 2019; 364 doi: https://doi.org/10.1136/bmj.l341 (Published 07 February 2019) Cite this as: BMJ 2019;364:l341

Linked opinion

A Mediterranean diet trial’s retraction and republication leaves a trail of questions

  1. Arnav Agarwal, research associate1,
  2. John P A Ioannidis, professor2
  1. 1Department of Medicine, University of Toronto, Toronto, Ontario, Canada
  2. 2Stanford Prevention Research Center and Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, CA 94305, USA
  3. Correspondence to: J P A Ioannidis jioannid{at}stanford.edu

Arnav Agarwal and John P A Ioannidis consider what we can learn from the retraction and republication of an influential trial of Mediterranean diet

The Prevención con Dieta Mediterránea (PREDIMED) trial1 is one of the most influential randomised trials ever. It was cited 3364 times in Google Scholar in the five years after its publication. However, in June 2018 the trial was retracted and republished because serious protocol deviations were detected. Moreover, the repercussions of these protocol deviations and of the correction process raise many additional important questions. How do you correct one of the most influential trials and the large universe of its secondary publications?

Initial results and early raised concerns

PREDIMED was originally published in 2013.1 Heralded as a pioneer effort in nutrition,2 it randomised 7447 participants to a Mediterranean diet supplemented with extra virgin olive oil, a Mediterranean diet supplemented with nuts, or a control diet. It showed a 30% relative risk reduction in a composite clinical endpoint of cardiovascular death, stroke, or myocardial infarction in the Mediterranean diet groups.1 PREDIMED was an important effort and one of the few long term trials with clinical outcomes in nutrition.

However, some concerns were raised at the time of the publication. The interventions tested were not a typical Mediterranean diet but single food supplementations. The control group experience was not necessarily non-Mediterranean. The primary outcome was a composite of three endpoints,34 and significant differences were driven by a single endpoint (stroke) without differences in other cardiovascular disease or death. Effect sizes were probably inflated because the trial was stopped early after interim analyses showed benefit. Several systematic reviews and guidelines have either omitted PREDIMED56 or have rated it as having serious risk of bias and being difficult to interpret.7 Moreover, secondary PREDIMED analyses reported results that were deemed implausible.89

Retraction and republication

Recent developments questioned PREDIMED at its core. An analysis of reported baseline characteristics in 5087 trials by Carlisle identified trials in which the compared randomised groups were too similar or too dissimilar, raising questions of potential fraud or non-random sampling.10 PREDIMED stood out for implausible P value patterns when comparing the baseline characteristics of the three arms. An audit of the trial found serious irregularities: enrolment of household members without randomisation; assignment of participants to study arms based on clinic site rather than true randomisation; and inconsistent use of randomisation tables. These deviations affected 1588 participants (21% of the total).11

The randomised trial was no longer a randomised trial. The original paper was retracted and replaced with a reanalysis that treated PREDIMED as a non-randomised study and excluded participants who were not truly randomised.11 The reanalyses gave similar point estimates for the primary endpoint.

Is republication justified?

Whether republication is justified in such cases is controversial. Retracting a paper and replacing it with a republication is an uncommon choice. In theory, it can be used when there is an error that significantly affects parts of the study but does not completely refute it. Therefore, the validity of this course of action depends on the nature of the errors and on whether there is really a way to fully correct them.12

In PREDIMED, the detected irregularities may not entirely explain the peculiar baseline characteristics and they also raise questions about the quality of other aspects of the conduct of the trial, such as data collection, data arbitration, and adjudication. Participants, investigators, and assessors were not blinded, and the unmasked design can further compound any bias—for example, if investigators and sponsors favour specific interventions or when assessors are collecting outcome data. Here, it is unclear if the correction of the specific identified errors also corrected all the potential accompanying problems and consequences of these errors.

Additionally, the title of the republication does not make it clear that it is a reanalysis and republication, and many readers may be confused. Showing clearly that the new paper is an amendment and numbering versions would help.13

Continuing follow-up

If randomisation problems had been detected while the study was ongoing, would it have been stopped early because of perceived benefit? The reanalysis excluding improperly randomised patients does not satisfy the P value boundary required for early stopping for each intervention arm. PREDIMED has in fact continued follow-up, and the investigators have published papers with 1.2 additional years of follow-up, during which the number of participants experiencing an event included in the composite primary endpoint increased by 19%. However, information is not provided on the comparison of the three arms for the primary endpoint with this extended follow-up. The full follow-up data should be disclosed and analysed for an intention-to-treat comparison to determine whether there is still benefit.

Full follow-up would also allow more complete assessment of mortality differences. Mortality is linked to a national registry and thus cannot be biased from subjective interpretation by the local investigators who collated data on clinical events before sending them to an adjudication committee. In the original follow-up, the control arm had 114 deaths, versus 116 and 118 in the two experimental arms, respectively.1 More deaths have occurred during the additional 1.2 years’ follow-up. Analysis of the full mortality data would provide the most convincing evidence from this trial and is essential to understand the robustness of its conclusion.

We have asked the corresponding authors of PREDIMED papers and (on their recommendation) the head of the steering committee to provide the number of primary outcome events and deaths in each arm in the extended follow-up. PREDIMED investigators responded that “information you have asked for is the main topic of ongoing analyses on PREDIMED data, as already approved by the steering committee.” We asked them again whether they could share minimal information for this article but have not heard back from them over the past six months.

What about secondary publications?

Importantly, the original PREDIMED paper had already generated 267 secondary publications before its retraction and republication.14 Thirty two of them have already received over 100 citations each in Google Scholar. Most of the publications come from the network of investigators who performed the original trial and their extended teams, with three investigators having each published over 150 articles from PREDIMED. In July 2018 we identified 203 secondary papers with data (excluding reviews, editorials, and commentaries); 194/203 (95%) first authors, 201/203 (99%) last authors, and 223/225 (99%) corresponding authors (some papers have more than one corresponding author) belong to the original PREDIMED investigator team in the 2013 paper or are affiliated with related Spanish institutions (see supplementary data on bmj.com).

The analyses presented in secondary papers use the data that led to retraction of the original. These publications should probably have notices of concern (as has been done for a secondary analysis published in CMAJ15) until they are properly re-evaluated. The PREDIMED authors have started correcting some of their work and have published several letters to this effect (at least five letters pertaining to eight secondary publications1617181920).

Re-evaluation should be truly independent. Given the circumstances, it cannot be done only by the PREDIMED investigators and other sympathetic investigators who hold similar views on the importance of specific foods to modulate disease risk and on the agreement between the results of non-randomised studies (what PREDIMED in perceived to be now) and randomised trials. Involvement of investigators with contrary views is pivotal for this reassessment to be fair and balanced. Even the best intentioned and most knowledgeable investigators in nutrition may still favour their beliefs and thus inadvertently introduce some bias in the reanalyses.21

Instead of trying to correct one paper at a time, it may be more efficient for an independent team to make a centralised effort and report its findings to all relevant journals. Audits of bodies of contested literature are not uncommon. Since the work usually pertains to single investigators they are normally done by university appointed committees. For a major multi-investigator effort like PREDIMED, the independent assessors should have international provenance.

Secondary publications that compare outcomes in the randomised arms are directly affected. However, even when secondary publications deal with the study dataset or subsets as an observational cohort, the clustering of recruited participants (eg, household members co-randomised or a whole village recruited in one step) still affects their results.22 The clustering effect also needs to be incorporated in observational analyses and may lead to different estimates and conclusions.

Inconsistencies in secondary publications

Even without in-depth re-evaluation of the raw data underlying PREDIMED publications, there are some inconsistencies in the reported data in these papers that suggest broader and more generalised problems that are not fixable by a single reanalysis.

To illustrate this point, we searched PubMed using the keyword “PREDIMED” and identified English language PREDIMED publications that included data on over 7000 participants reporting either the primary composite endpoint or mortality (all causes and cause specific). The number of events varies widely even when publications have similar follow-up (see supplementary table on bmj.com).

Some discrepancies may simply reflect missing data and different eligibility criteria in different analyses. However, table 1 shows a sample of contradictions in the reported data in paired papers that cannot be explained in this way and point to errors in one or both of the papers. The original publication1 reported 87 cardiovascular deaths for the full trial population and follow-up, while Henríquez-Sánchez and colleagues’ secondary analysis of dietary antioxidants and mortality24 reports 102 such deaths despite a more limited sample and follow-up. Two publications2324 have identical follow-up, but one23 has a larger number of total deaths, while the other24 has a higher number of cardiovascular deaths. Two other publications2526 have identical total number of deaths, but one has five more cardiovascular deaths (a component of the primary endpoint)26 while the other has five more deaths from other causes (not a component of the primary endpoint).25 The original publication reported only cardiovascular deaths and total deaths.1 However, if the 166 deaths from cancer reported by Henríquez-Sánchez and colleagues24 are added to the 117 deaths from non-cancer, non-cardiovascular causes reported by Martínez-González and colleagues25 and 87 deaths from cardiovascular causes in the original paper,1 the total deaths are 370, exceeding the total deaths (n=348) reported in the original publication despite identical length of follow-up.1

Table 1

Examples of inconsistent data on main endpoints across some PREDIMED publications

View this table:

These discrepancies may point to poor reporting, erroneous or inconsistent statistical analyses, or deeper problems related to problematic data collection and curation. They are superimposed on a PREDIMED literature that shows all the hallmarks of data dredging, given the huge number of secondary publications. Most analyses are not prespecified or are specified imprecisely. In fact, it is unclear which (if any) secondary analyses were clearly and unambiguously prespecified. For example, one secondary publication states that invasive breast cancer was a prespecified outcome,27 but the published trial protocol states that all cancer—not just invasive breast cancer—was a secondary outcome. Data are reported on one cancer subtype (claiming huge benefits) but not on dozens of others.

Publicly available data and reanalysis needed

PREDIMED investigators have stated that the data that went into the recent republication can be requested by interested parties, but the concerns identified make a strong case for the complete PREDIMED dataset, not just the data that went into the recent republication, to become publicly available. A truly independent audit should examine the original data records, adjudication, and statistical analyses that underlie this voluminous published literature. Updated follow-up results should be independently assessed and reported, including all-cause mortality data (the most objective outcome). Box 1 summarises some proposed actions. Some may wonder whether it is worth investing so much effort auditing a single study when there is a vast literature on nutrition with potential errors and biases. However, PREDIMED is an iconic example of using randomisation in the field (despite all the caveats discussed above) and it has major repercussions. An independent reanalysis effort may be more efficient and convincing than piecemeal attempts by the authors to defend each secondary publication separately.

Box 1

Potentially useful actions for PREDIMED

  • Disclose full long term outcomes with updated follow-up

  • Issue notices of concern for secondary publications until they are reassessed

  • Consider centralised effort to re-evaluate all PREDIMED publications together

  • Include both sympathetic and contrarian researchers in the re-evaluation

  • Correct obvious inconsistencies that already violate plain logic rules

  • Audit raw data, not just the clean data

  • Audit data collection and curation procedures

  • Correct or retract secondary publications, as appropriate

  • Make raw data widely available (not only those pertaining to recent reanalysis)

RETURN TO TEXT

Lessons for future pivotal multicentre trials

PREDIMED may offer useful lessons about how to run future large multicentre trials that aim to revolutionise an entire field. The original PREDIMED publication1 had 18 authors and 223 collaborators. Many secondary papers also feature impressive numbers of coauthors. However, studies with hundreds of investigators may still have blind spots where no one really is responsible or knowledgeable enough to avert major mistakes and protocol deviations (box 2).

Box 2

Potential safeguards for large scale multicentre clinical trials

  • Include strong methodologists in the statistical team

  • Include strong methodologists in the data and safety monitoring board (DSMB)

  • Investigators responsible for randomisation, data curation, and analysis should be designated and have sufficient methodological expertise

  • Ensure that several people can independently safeguard against each type of error

  • Monitoring board should include people who do not support the study hypothesis

RETURN TO TEXT

PREDIMED investigators have a unique opportunity to disclose details on how the process failed, so that other trialists may avoid similar problems. Strong methodological expertise at all pivotal steps and function, both site specific and centralised, is essential. Also the inclusion of people who do not support the study hypothesis in the monitoring board is key to ensure balance and avoid bias.

PREDIMED offers useful lessons that can be applied to many other trials.28 The problems should not lead to a reduction in funding of diet related research to improve health. Conversely, the same or even higher funding should be diverted to well executed large trials in nutrition. This will require getting together people with different expertise and skill sets. We also have an opportunity to probe how large volumes of published literature can be reassessed and corrected, as appropriate, when serious problems are identified. Finally, we can ponder on what expertise and better safeguards are needed to run such important multicentre trials reliably in the future.

Key messages

  • PREDIMED, a highly influential trial of nutrition, was recently retracted and republished after major protocol deviations were noted

  • Republication may not solve multiple problems that remain, including the inappropriateness of stopping early given the revised results and the effects on over 200 secondary publications

  • Multiple contradictions between data reported across PREDIMED publications suggest a more generic problem with the trial’s quality.

  • PREDIMED may provide useful lessons on how to reassess and correct large volumes of published literature and on what methodological safeguards are needed for pivotal multicentre trials

Footnotes

  • Contributors and sources: JPAI conceived the idea for this article and both authors elaborated on it. Both authors extracted and evaluated data, wrote and revised the paper, and approved the final version. JPAI is guarantor.

  • Competing interests: We have read and understood BMJ policy on declaration of interests and declare that the Meta-Research Innovation Center at Stanford (METRICS) has been funded by the Laura and John Arnold Foundation. JPAI’s work is funded by an unrestricted gift from Sue and Bob O’Donnell. JPAI loves olive oil, nuts, and the Mediterranean diet.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

References

View Abstract