Postmarketing studies of drug safetyBMJ 2011; 342 doi: https://doi.org/10.1136/bmj.d342 (Published 08 February 2011) Cite this as: BMJ 2011;342:d342
- Sebastian Schneeweiss, associate professor,
- Jerry Avorn, professor
- 1Division of Pharmacoepidemiology and Pharmacoeconomics, Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School, Boston, MA 02120, USA
In the early days of randomised clinical trials, their results could be manipulated in several ways—protocols could be altered in light of early findings, sponsors could exert undue influence over what could be published, and some “unfavourable” results could be suppressed entirely. In the United States, the creation of the government clinical trials website (www.clinicaltrials.gov) greatly contributed to minimising these threats to honest science.1 But requiring similar consistency, rigour, and transparency has been more difficult with observational studies, because any person or company with modest resources can purchase a large database of health insurance claims and perform a variety of epidemiological analyses with little or no accountability for the transparency, rigour, or visibility of such work⇓.
In 2006, the European Medicines Agency took on this problem by creating the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) to provide registration, standardisation, and quality assurance for observational studies of the effects of drugs (www.encepp.eu/). To qualify for the “ENCePP seal,” study organisers must agree to a code of conduct and transparency, meet a checklist of methodological standards, and agree to publicly post the study protocol as well as its results.2
“Best practices” for the conduct of epidemiological studies of the safety of drugs are less well standardised than those developed over the decades for clinical trials. Because of the multitude of designs and analytical choices available for observational studies, each with its own strengths and limitations, the validity of pharmacoepidemiological research findings is harder to appraise. People responsible for decisions about drug safety, who must often make difficult policy calls based on scanty non-randomised data, have long wished for a user-friendly way of determining the validity of a given study. The creation of the ENCePP process will do a great deal to make study designs more transparent and might provide some useful methodological guidance in the process.
Although the ENCePP approach will promote transparency in non-randomised drug safety studies, it cannot (and does not) claim to guarantee their quality. A methods checklist can remind researchers of areas that should be dealt with in any pharmacoepidemiological study protocol—an important development in the evolution of this still young field. But it does not ensure guidance or best practice beyond that. Other guidelines, including good pharmacoepidemiological practice,3 purposely list only higher level principles of good science, but they stop short of making specific recommendations about methodological choices.
As with the registration of randomised trials, the idea of registering an observational study protocol is that predefined endpoint analyses are likely to be more valid and less easy to manipulate. But some have questioned this approach.4 Is the association between a drug and a predefined end point (such as a given adverse event) necessarily more important or valid than a stronger and more precise effect estimate for an important end point (such as another unexpected adverse event) that was discovered during the analysis of the same study? Confining all analyses to a predefined protocol without allowing for additional considerations may reduce the potential for in depth discovery, because important new insights may emerge only during data analysis.5 A more important advantage of registration may be that researchers are likely to work harder on a study design that they know will become public. In the end it is the quality of individual studies—that is, the control of biases—that most needs to be improved. When biases are ruled out in a credible way, by the original or subsequent studies, epidemiological data can lead to sensible regulatory decisions, irrespective of what was predefined in a protocol.6 7 Although the ENCePP seal will confer a degree of legitimacy regarding transparency and adherence to basic standards, studies that do not go through the ENCePP approval process should not be seen as necessarily yielding lower grade evidence.
The first ENCePP listed study will tackle the effectiveness and safety of long acting β agonists in patients with chronic obstructive pulmonary disease.8 As required, the authors have posted their study design and pledged that they will adhere to ENCePP’s code of conduct and research standards. So far, so good. But a review of the posted protocol raises several methodological concerns. Firstly, the study will combine prevalent and incident users of inhaled long acting β agonists. But ongoing users of these drugs are likely to represent “survivors” who have done well on that drug (and not had to stop taking it because of side effects).9 Furthermore, unlike in clinical trials where participants’ characteristics are assessed before treatment, the characteristics of these people will be assessed during treatment and therefore be subject to the effects of the study drugs being used.10 In addition, the total duration of drug use may not be fully known from the proposed claims database, so that shorter term users might be compared with longer term users.11
Secondly, one comparison group will be “non-users” of any study drug. Yet untreated patients with chronic obstructive pulmonary disease are likely to have less severe disease than those who are treated.12 If the investigators cannot determine the time of onset of the disease or of drug use, it may be difficult to be sure that the underlying differences between treatment and non-user comparison groups can be adequately adjusted for in the data analysis, because it can be difficult to measure the severity of chronic obstructive pulmonary disease with claims data.
An important advantage of the ENCePP process is that study designs will now be publicly available, providing an opportunity for feedback by peer researchers and revisions by the investigators, which will ultimately lead to better science. ENCePP deserves much credit for its successful implementation of this process to ensure improved transparency of study methods. But the quality of each study will still need to be judged on the basis of a detailed description of the final design and analytical approach, which ENCePP now makes possible.
An important next step for ENCePP could be to make study data, from source data to the final analytical datasets, available to peer reviewers. This would make it possible for findings to be reconsidered using different analytical strategies. Doing so would also encourage the original investigators to work even more thoughtfully, knowing that their analyses may be replicated and reassessed by their peers.
Cite this as: BMJ 2011;342:d342
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review: Commissioned; not externally peer reviewed.