Re: European drug agency backtracks on plan to give researchers access to clinical trial reports
The German Institute for Quality and Efficiency in Health Care (IQWiG) has been deeply involved in the debate on data transparency, as health technology assessment (HTA) agencies such as IQWiG require full information on clinical studies and their results to be able to provide appropriate and meaningful assessments of drugs.
The European Medicines Agency’s (EMA’s) initiative for data transparency, initiated in 2012, has been a major development in this debate. IQWiG participated in EMA’s advisory committees and provided comments on EMA’s draft policy document entitled “Publication and access to clinical-trial data” [1]. However, we now share Doshi’s and Jefferson’s concerns about EMA’s U-turn [2]. In our opinion, the data sharing model now suggested by EMA is jeopardizing what could have been a major improvement in health care. We call on EMA to return to the approaches outlined in the initial draft policy on data transparency. The reasons for our concerns are as follows:
Our experience shows: we need clinical study reports to be fully informed about a drug’s benefits and harms
IQWiG has been working with clinical study reports (CSRs) since 2005. The availability of full CSRs has been crucial for our understanding of the studies included in our assessments and for the completeness of data on patient-relevant outcomes describing the benefits and harms of the drugs under assessment. We quantified the information gain from CSRs in two analyses published in 2012 and 2013: the comparison of CSRs with publicly available journal publications and reports from study registries (“registry reports”) showed that CSRs provided complete information on 88% of relevant methods items; the corresponding rates for journal publications and registry reports were 40% and 31%, respectively [3]. Concerning study results, CSRs provided complete information on 86% of patient-relevant outcomes; the corresponding rates for journal publications and registry reports were 23% and 22%, respectively (39% in the combined publicly available sources) [4].
Additional information from CSRs has challenged published evidence on health care interventions or even reversed conclusions drawn on the basis of publicly available information [5 6].
On the basis of this experience, we have been strongly supporting EMA during the discussions on the new transparency policy. However, we consider EMA’s most recent suggestions insufficient to provide adequate information on clinical studies.
“View on screen only” prevents reliable research
EMA plans to provide CSRs in a “view-on-screen-only” mode. From our point of view, CSRs that can only be viewed on screen (and are not downloadable or printable) are in fact not available in a manner that allows the conduct of reliable research. In our opinion, “view-on-screen-only” CSRs represent nothing more than a placebo intervention against publication bias.
To be able to work with this complex and lengthy type of document (which is often several thousand pages long), researchers have to be able to print the relevant parts of the CSRs to have a constant reference during their work, to copy specific contents (e.g. outcome definitions), to organize information across studies, or to bookmark documents so that important information can be identified in a reproducible manner. The annotated content of the CSRs has to be shared within the research team to achieve a mutual understanding of the studies under discussion. These important steps in the research team’s daily work would not be possible with EMA’s “view-on-screen-only” approach. This makes as much sense as suggesting that EMA accepts regulatory submission dossiers in this format.
Redaction of CSRs prevents the use of important study results for decision making in health care
Against the notion of the new EU clinical trials regulation, which clearly states that “in general the data in a clinical study report should not be considered commercially confidential once a marketing authorization has been granted …”, the CSR redaction principles suggested by EMA specify study information that could constitute commercially confidential information. First of all, this step questions ethical principles of research in humans [7]. Furthermore, the legal defensibility could be challenged [8 9]. In addition, this approach contradicts EMA’s own policy in other areas.
The examples provided in the draft “Redaction principles” include exploratory study outcomes that were not used to support a label claim and did not contribute to the overall benefit-risk evaluation. However, this ignores the fact that study results are needed beyond regulatory decision making and thus must be made fully publicly available. In IQWiG’s assessments we routinely use exploratory outcomes from CSRs (which are often not available in journal publications), as they are relevant for the assessment of the benefits and harms of a drug. For example, health-related quality-of-life outcomes or symptoms are often defined as exploratory and not used to support the label claim. However, they are highly relevant for HTA and for individual decision making by patients and their physicians.
We are currently working with other HTA agencies and EMA to develop procedures to provide joint scientific advice to drug manufacturers with the aim of shaping drug development programmes that not only meet the requirements of drug regulators, but also of HTA agencies. Thus, in the future, CSRs will hopefully contain much more information than needed for the label claim or for the regulatory benefit-risk evaluation. The redaction of this content devaluates the efforts to improve the evidence base for future drugs. It even questions whether the joint advice efforts are worthwhile at all.
Planned restrictions of use jeopardize comparative effectiveness research and health technology assessment
In the whole of Europe, comparative effectiveness research (CER) and HTA are used to inform decision making on a health care system level. These processes require extensive and detailed data. At the same time, many of the decisions made within this framework are on drug costs. It is inevitable that within these processes many stakeholders, such as drug manufacturers, have commercial interests. According to the current terms of use, CSR data could not be used for these assessments.
In many countries, Germany included, drug manufacturers have to submit dossiers comparing their new drugs with existing treatment options. In many cases this is done by using indirect comparisons. This approach has often been unsuccessful or produced results of limited certainty, as journal publications have provided insufficient information on studies to be included in indirect comparisons. Besides academic or other non-commercial researchers, drug manufacturers would therefore also need to use CSRs of studies with possible comparators so that they could provide HTA agencies with the information requested. As the results of these assessments may have an effect on the new drug’s revenue, preparing drug dossiers for HTA might be considered a commercial purpose of data use and would be prohibited according to EMA’s terms of use. This means that EMA’s proposed policy would prevent what we are requesting from manufacturers: to support decision making on a health care system level.
At the same time, the restricted use of data also jeopardizes the informed involvement of other stakeholders in these discussions, such as medical societies or patient organizations.
A lost opportunity to improve patient care
In November 2012, Guido Rasi opened the EMA workshop on clinical data and transparency with the words “We are not here to decide if we publish clinical trial data, but how …”. This statement fell nothing short of a paradigm change in EMA’s position in the data transparency debate. Since then, the perspective of extensive availability of study information has prompted ideas for future research on the use of these data to improve patient care.
One and a half years later, the “if” is at stake. Data we cannot work with are still hidden – even if we see them on a screen.
References
1. Institute for Quality and Efficiency in Health Care. Submission of comments on 'Policy 0070 on publication and access to clinical-trial data' [online]. https://www.iqwig.de/download/IQWiG_comments_on_EMA_Draft_Policy_0070.pdf.
2. PLOS Guest Blogger. Is the EMA poised to make a major U-turn on its transparency initiative? [online]. In: PLOS Medical Journals' Community Blog. May 20, 2014. http://blogs.plos.org/speakingofmedicine/2014/05/20/ema-poised-make-majo....
3. Wieseler B, Kerekes MF, Vervölgyi V, McGauran N, Kaiser T. Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications. BMJ 2012;344:d8141.
4. Wieseler B, Wolfram N, McGauran N, Kerekes MF, Vervölgyi V, Kohlepp Pet al. Completeness of reporting of patient-relevant clinical trial outcomes: comparison of unpublished clinical study reports with publicly available data. PLoS Med 2013;10:e1001526.
5. Eyding D, Lelgemann M, Grouven U, Harter M, Kromp M, Kaiser Tet al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 2010;341:c4737.
6. Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJet al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev 2014:CD008965.
7. Strech D, Littmann J. Lack of proportionality: seven specifications of public interest that override post-approval commercial interests on limited access to clinical data. Trials 2012;13:100.
8. O`Reilly E. Ombudsman concerned about change of policy at Medicines Agency as regards clinical trial data transparency [online]. May 16, 2014. http://europa.eu/rapid/press-release_EO-14-13_en.htm.
9. Association Internationale de la Mutualité (AIM), International Society of Drug Bulletins (ISDB), Medicines in Europe Forum (MiEF), Nordic Cochrane Centre. Backpedalling on EMA`s "proactive publication of clinical-data" draft policy: was it all just a window-dressing exercise? Who or what is the EMA afraid of? [press release] [online]. May 20, 2014. http://english.prescrire.org/Docu/DOCSEUROPE/20140520_EMATransparencyPol....
Competing interests:
All authors are employees of the Institute for Quality and Efficiency in Health Care (IQWiG). To produce unbiased HTA reports, the
Institute depends on access to all of the relevant data on the topic under investigation. The authors therefore support public access to clinical study reports.
23 May 2014
Beate Wieseler
Head of Department of Drug Assessment
Natalie McGauran, Thomas Kaiser.
Institute for Quality and Efficiency in Health Care
Rapid Response:
Re: European drug agency backtracks on plan to give researchers access to clinical trial reports
The German Institute for Quality and Efficiency in Health Care (IQWiG) has been deeply involved in the debate on data transparency, as health technology assessment (HTA) agencies such as IQWiG require full information on clinical studies and their results to be able to provide appropriate and meaningful assessments of drugs.
The European Medicines Agency’s (EMA’s) initiative for data transparency, initiated in 2012, has been a major development in this debate. IQWiG participated in EMA’s advisory committees and provided comments on EMA’s draft policy document entitled “Publication and access to clinical-trial data” [1]. However, we now share Doshi’s and Jefferson’s concerns about EMA’s U-turn [2]. In our opinion, the data sharing model now suggested by EMA is jeopardizing what could have been a major improvement in health care. We call on EMA to return to the approaches outlined in the initial draft policy on data transparency. The reasons for our concerns are as follows:
Our experience shows: we need clinical study reports to be fully informed about a drug’s benefits and harms
IQWiG has been working with clinical study reports (CSRs) since 2005. The availability of full CSRs has been crucial for our understanding of the studies included in our assessments and for the completeness of data on patient-relevant outcomes describing the benefits and harms of the drugs under assessment. We quantified the information gain from CSRs in two analyses published in 2012 and 2013: the comparison of CSRs with publicly available journal publications and reports from study registries (“registry reports”) showed that CSRs provided complete information on 88% of relevant methods items; the corresponding rates for journal publications and registry reports were 40% and 31%, respectively [3]. Concerning study results, CSRs provided complete information on 86% of patient-relevant outcomes; the corresponding rates for journal publications and registry reports were 23% and 22%, respectively (39% in the combined publicly available sources) [4].
Additional information from CSRs has challenged published evidence on health care interventions or even reversed conclusions drawn on the basis of publicly available information [5 6].
On the basis of this experience, we have been strongly supporting EMA during the discussions on the new transparency policy. However, we consider EMA’s most recent suggestions insufficient to provide adequate information on clinical studies.
“View on screen only” prevents reliable research
EMA plans to provide CSRs in a “view-on-screen-only” mode. From our point of view, CSRs that can only be viewed on screen (and are not downloadable or printable) are in fact not available in a manner that allows the conduct of reliable research. In our opinion, “view-on-screen-only” CSRs represent nothing more than a placebo intervention against publication bias.
To be able to work with this complex and lengthy type of document (which is often several thousand pages long), researchers have to be able to print the relevant parts of the CSRs to have a constant reference during their work, to copy specific contents (e.g. outcome definitions), to organize information across studies, or to bookmark documents so that important information can be identified in a reproducible manner. The annotated content of the CSRs has to be shared within the research team to achieve a mutual understanding of the studies under discussion. These important steps in the research team’s daily work would not be possible with EMA’s “view-on-screen-only” approach. This makes as much sense as suggesting that EMA accepts regulatory submission dossiers in this format.
Redaction of CSRs prevents the use of important study results for decision making in health care
Against the notion of the new EU clinical trials regulation, which clearly states that “in general the data in a clinical study report should not be considered commercially confidential once a marketing authorization has been granted …”, the CSR redaction principles suggested by EMA specify study information that could constitute commercially confidential information. First of all, this step questions ethical principles of research in humans [7]. Furthermore, the legal defensibility could be challenged [8 9]. In addition, this approach contradicts EMA’s own policy in other areas.
The examples provided in the draft “Redaction principles” include exploratory study outcomes that were not used to support a label claim and did not contribute to the overall benefit-risk evaluation. However, this ignores the fact that study results are needed beyond regulatory decision making and thus must be made fully publicly available. In IQWiG’s assessments we routinely use exploratory outcomes from CSRs (which are often not available in journal publications), as they are relevant for the assessment of the benefits and harms of a drug. For example, health-related quality-of-life outcomes or symptoms are often defined as exploratory and not used to support the label claim. However, they are highly relevant for HTA and for individual decision making by patients and their physicians.
We are currently working with other HTA agencies and EMA to develop procedures to provide joint scientific advice to drug manufacturers with the aim of shaping drug development programmes that not only meet the requirements of drug regulators, but also of HTA agencies. Thus, in the future, CSRs will hopefully contain much more information than needed for the label claim or for the regulatory benefit-risk evaluation. The redaction of this content devaluates the efforts to improve the evidence base for future drugs. It even questions whether the joint advice efforts are worthwhile at all.
Planned restrictions of use jeopardize comparative effectiveness research and health technology assessment
In the whole of Europe, comparative effectiveness research (CER) and HTA are used to inform decision making on a health care system level. These processes require extensive and detailed data. At the same time, many of the decisions made within this framework are on drug costs. It is inevitable that within these processes many stakeholders, such as drug manufacturers, have commercial interests. According to the current terms of use, CSR data could not be used for these assessments.
In many countries, Germany included, drug manufacturers have to submit dossiers comparing their new drugs with existing treatment options. In many cases this is done by using indirect comparisons. This approach has often been unsuccessful or produced results of limited certainty, as journal publications have provided insufficient information on studies to be included in indirect comparisons. Besides academic or other non-commercial researchers, drug manufacturers would therefore also need to use CSRs of studies with possible comparators so that they could provide HTA agencies with the information requested. As the results of these assessments may have an effect on the new drug’s revenue, preparing drug dossiers for HTA might be considered a commercial purpose of data use and would be prohibited according to EMA’s terms of use. This means that EMA’s proposed policy would prevent what we are requesting from manufacturers: to support decision making on a health care system level.
At the same time, the restricted use of data also jeopardizes the informed involvement of other stakeholders in these discussions, such as medical societies or patient organizations.
A lost opportunity to improve patient care
In November 2012, Guido Rasi opened the EMA workshop on clinical data and transparency with the words “We are not here to decide if we publish clinical trial data, but how …”. This statement fell nothing short of a paradigm change in EMA’s position in the data transparency debate. Since then, the perspective of extensive availability of study information has prompted ideas for future research on the use of these data to improve patient care.
One and a half years later, the “if” is at stake. Data we cannot work with are still hidden – even if we see them on a screen.
References
1. Institute for Quality and Efficiency in Health Care. Submission of comments on 'Policy 0070 on publication and access to clinical-trial data' [online]. https://www.iqwig.de/download/IQWiG_comments_on_EMA_Draft_Policy_0070.pdf.
2. PLOS Guest Blogger. Is the EMA poised to make a major U-turn on its transparency initiative? [online]. In: PLOS Medical Journals' Community Blog. May 20, 2014. http://blogs.plos.org/speakingofmedicine/2014/05/20/ema-poised-make-majo....
3. Wieseler B, Kerekes MF, Vervölgyi V, McGauran N, Kaiser T. Impact of document type on reporting quality of clinical drug trials: a comparison of registry reports, clinical study reports, and journal publications. BMJ 2012;344:d8141.
4. Wieseler B, Wolfram N, McGauran N, Kerekes MF, Vervölgyi V, Kohlepp Pet al. Completeness of reporting of patient-relevant clinical trial outcomes: comparison of unpublished clinical study reports with publicly available data. PLoS Med 2013;10:e1001526.
5. Eyding D, Lelgemann M, Grouven U, Harter M, Kromp M, Kaiser Tet al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 2010;341:c4737.
6. Jefferson T, Jones MA, Doshi P, Del Mar CB, Hama R, Thompson MJet al. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev 2014:CD008965.
7. Strech D, Littmann J. Lack of proportionality: seven specifications of public interest that override post-approval commercial interests on limited access to clinical data. Trials 2012;13:100.
8. O`Reilly E. Ombudsman concerned about change of policy at Medicines Agency as regards clinical trial data transparency [online]. May 16, 2014. http://europa.eu/rapid/press-release_EO-14-13_en.htm.
9. Association Internationale de la Mutualité (AIM), International Society of Drug Bulletins (ISDB), Medicines in Europe Forum (MiEF), Nordic Cochrane Centre. Backpedalling on EMA`s "proactive publication of clinical-data" draft policy: was it all just a window-dressing exercise? Who or what is the EMA afraid of? [press release] [online]. May 20, 2014. http://english.prescrire.org/Docu/DOCSEUROPE/20140520_EMATransparencyPol....
Competing interests: All authors are employees of the Institute for Quality and Efficiency in Health Care (IQWiG). To produce unbiased HTA reports, the Institute depends on access to all of the relevant data on the topic under investigation. The authors therefore support public access to clinical study reports.