Clinical and economic consequences of a reimbursement restriction of nebulised respiratory therapy in adults: direct comparison of randomised and observational evaluations
BMJ 2004; 328 doi: https://doi.org/10.1136/bmj.38020.698194.F6 (Published 04 March 2004) Cite this as: BMJ 2004;328:560
- Sebastian Schneeweiss, assistant professor (schneeweiss{at}post.harvard.edu)1,
- Malcolm Maclure, professor2,
- Bruce Carleton, associate professor3,
- Robert J Glynn, associate professor1,
- Jerry Avorn, associate professor1
- 1 Brigham and Women's Hospital and Harvard Medical School, Division of Pharmacoepidemiology and Pharmacoeconomics, 1620 Tremont St (Suite 3030), Boston, MA 02120, USA
- 2 Harvard School of Public Health, Boston, University of Victoria, Victoria, Canada
- 3 University of British Columbia, Vancouver, Canada
- Correspondence to: S Schneeweiss
- Accepted 7 January 2004
Abstract
Objective To compare the results of a randomised and an observational evaluation of the same policy that restricted reimbursement for nebulised respiratory medications in adult patients in a community setting.
Designs Cluster randomised controlled trial and observational time series with historical controls.
Setting Pharmacare, the government funded drug benefits plan for elderly people and patients receiving social assistance in British Columbia, Canada.
Participants In the randomised controlled trial 104 clusters of medical practices, pair matched by geography and approximately by practice size, were randomised to the intervention group (449 patients affected by the policy on 1 March 1999), and the control group (offered a six month exemption, affecting 386 patients). The observational analysis included all Pharmacare beneficiaries (excluding the 386 exempt patients) who had used any nebulised drugs six months before the policy (4624 patients).
Intervention Pharmacare restricted reimbursement for nebulised bronchodilators, steroids, and cromoglycate to patients whose doctors applied for an individual patient's exemption, giving an appropriate clinical reason.
Main outcome measures Number of contacts with doctors and services, emergency admissions to hospital, and utilisation of and expenditure for respiratory drugs in databases of British Columbia's Ministry of Health.
Results Contacts with doctors or emergency admissions to hospital did not increase in association with the restriction, regardless of the analytical approach. In the observational analysis, we found a reduction of $C24 per patient month in all nebulised drug use (95% confidence interval 19 to 29) and an increase of $C3 per patient month in all expenditure for inhalers (1.4 to 4.5). The randomised evaluation found savings of $C8 per patient month for nebulisers (P = 0.24) and no increase in spending on inhalers (P = 0.79). Correcting for 60% non-compliance by exempt doctors in a sensitivity analysis yielded similar results as the observational evaluation.
Conclusions Observational as well as randomised analyses found moderate net savings and no increase in unintended healthcare outcomes after restricting reimbursement for nebulised respiratory drugs. Randomised policy trials are feasible and, if carefully implemented, likely to be concordant with observational evaluations.
Introduction
Total expenditure on drugs in the United Kingdom (£5.5bn; $10.1bn; €8bn) has grown by 30% over the past four years, 50% faster than other expenditure on health services.1 Spending in continental Europe and North America has escalated similarly.2 To contain costs, drug benefit programmes are introducing increasingly restrictive policies that are, however, intended to have no impact on access to effective care.3 4 Critics claim that restrictions cause patients to switch to less effective treatments, resulting in reduced compliance,5 6 more contacts with doctors and procedures, and more admissions to hospital.7 8
As drug cost containment policies evolve the need grows for direct, valid, and timely evidence of their benefits and risks.9 Only a few well designed observational evaluations have been undertaken—no randomised trials—of the clinical and economic consequences of restricting the reimbursement of drugs.10–14 One observational study15 16 that received publicity17 purported to show by comparing six health maintenance organisations that restrictions lead to more admissions to hospitals and doctors' services, thus increasing net costs. Seriously confounded by unmeasured characteristics of the health maintenance organisation and patient selection bias,18–20 the study was valuable mainly for highlighting the need for rigorous methods in this field. Randomised controlled trials provide the gold standard of evidence because they eliminate selection bias and baseline confounding if enough units are randomised.21 However, a major barrier to policy trials is the belief—which Chalmers called the biggest myth22—that randomisation is expensive. An opportunity to assess the feasibility of randomised evaluation of drug policy and to compare it with a well designed observational evaluation arose in British Columbia, when the provincial government's drug plan, Pharmacare, restricted reimbursement for nebulised respiratory drugs in 1999.
Current guidelines from the British Thoracic Society say that nebulised respiratory drugs are indicated for adult patients with a limited number of conditions, including chronic persistent asthma in elderly people, exacerbations of chronic obstructive pulmonary disease, palliative care, or specific infections.23 On the other hand, a Canadian asthma consensus conference found that “nebulised medication is rarely, if ever, indicated in the management of asthma in adults.”24 Metered dose inhalers are more efficient than nebulisers at delivering drugs to the lungs, and drug costs are usually lower with inhalers. Many drug benefit plans therefore limit reimbursement of nebulised drugs.25 When such a policy was developed in British Columbia we persuaded Pharmacare to delay it for six months in a randomised control group of 10% of doctors and patients, which would cost Pharmacare no more than if the long delayed policy were further delayed by just 18 days (10% of six months).26
This enabled us to compare a cluster randomised evaluation with an observational time trend evaluation in the same target population.
Methods
Intervention
Pharmacare stopped covering the cost of nebulised bronchodilators, steroids, and cromoglycate on 1 March 1999, except if a doctor applied for an exemption for an individual by stating an appropriate clinical reason. Six weeks in advance, all doctors in British Columbia received a letter on the new policy, with brief mention of supporting evidence and the Canadian treatment guidelines.24
Randomised study design
To avoid the problem of “doctor shopping” we identified 10% of all British Columbia's primary care doctors in clusters of rural doctors and used administrative data provided by the Ministry of Health to match them to similar clusters of doctors in British Columbia by size of location, number of patients potentially affected by the policy, and number of clinicians at the same postal code. This yielded 53 pairs of clusters of doctors. We randomly assigned one cluster of each pair to either the policy implementation group or the policy exempted control group. Randomising isolated but matched clusters of doctors minimised the risk of contamination and reduced imbalances owing to chance. We requested informed consent only from the doctors in the policy exempted group.27 Only one doctor declined to participate and was excluded from the study, together with the matched doctor. As figure 1 shows, we studied both groups from the time of the province-wide policy implementation on 1 March 1999 and followed them for six months. After that point the policy delayed group was also affected by the new policy. We did not use a formal sample size calculation to determine the sample size but negotiated it on the basis of the expected lost savings from exempting doctors.
Time line of the study and illustration of the randomised and observational analyses. The arrows () indicate the direction of comparisons in the randomised and observational analyses. The pink person-time areas are subject to the new policy; the green areas represent control groups not subject to the new policy. The group that was exempted from the policy is excluded in the observational analysis to mimic an evaluation without the presence of a randomised policy trial
Observational study design
The observational study followed all people affected by Pharmacare's policy in March 1999 (including the randomised intervention group but excluding the exempted control group) over the same period as the randomised groups were followed and compared their experience with that of two historical control groups defined the same way, but starting one and two years earlier (figure 1)
Cohorts of patients
Altogether 5463 of all patients 18 years or older and covered by Pharmacare28 had filled at least one prescription for a nebulised respiratory drug in the preceding 12 months (3 March 1998 to 28 February 1999) and had not died or emigrated before the policy was introduced. The randomised cohorts comprised 449 (8.2%) patients in the intervention group and 386 (7.0%) in the control group.
The observational cohorts comprised all 4624 adult patients affected by the policy (after excluding the 386 in the control group) who had filled a prescription for nebulised medications during the preceding 12 months and had not died or emigrated before 1 March 1999, and two historical control cohorts defined the same way, but starting one and two years earlier (nhis1 = 4474; nhis2 = 4256; fig 1).
Study end points
We used de-identified data from the databases of British Columbia's Ministry of Health to monitor use of and expenditure for respiratory drugs, doctors' contacts and services, and emergency admissions to hospital. Analysts in the Ministry of Health used unique personal health numbers to extract each patient's data on drug dispensings, doctors' services, and admissions to hospital and linked these. They replaced personal health numbers with unique study numbers before giving the data to the investigators. We drew Pharmacare's drug claims data from PharmaNet, the single computer that captures the substance, route, dose, quantity, and costs of all dispensings by community pharmacies in British Columbia. We linked the date, type, and cost of medical services and one ICD-9-CM diagnostic code for each service as well as hospital admissions with up to 16 ICD-9-CM discharge diagnoses. Previous reports found high accuracy and completeness of the data fields we used.29 30
Statistical analysis
For each month during the study period we calculated event rates by dividing the number of events by the number of all non-censored cohort members in that month. We used χ2 tests and t tests to compare baseline characteristics.31
Randomised intention to treat analysis
We used generalised linear models and assumed gamma distributed errors for cost outcomes32 and Poisson errors for count outcomes (contacts with doctors; admissions to hospital), with correction for overdispersion. We used generalised estimating equations to adjust standard errors for clustering of patients (unit of analysis) within doctors' clusters (unit of randomisation).33 We repeated the analysis by using simple t tests34 as well as non-parametric bootstrap analysis with 2000 replications.35 All three analytical approaches produced almost identical results.
Observational longitudinal data analysis
We used generalised linear models for repeated events to estimate sudden changes in levels and slopes of trends of study end points. We treated the date the intervention started as an instrumental variable.36 We censored patients from all analyses, starting from the date of death or emigration (the month after the last claim in the database). Regression models included a constant term, a term for a linear baseline time trend before the intervention (September 1998 to February 1999), and a binary term and linear time trend for the period afterwards (March 1999 to October 1999).37 38
Time varying covariates included age in years at the beginning of each month and the Charlson comorbidity score39 computed from ICD-9-CM diagnosis codes for hospital and ambulatory care in the specified month and the preceding five months. We adjusted for overdispersion by including a scale variable in the model40 and accounted for intrapatient correlation between the repeated observations by generalised estimating equations.33 On the basis of the Akaike's information criterion41 an autocorrelated covariance structure in patients over time, with a one month lag period, fitted the data best.
We first adjusted for seasonality separately, using each of the two historical control cohorts. Membership of a cohort (observational cohort versus historical control cohort) was indicated by a binary variable in the above model. We then tested whether a multiplicative interaction term of cohort membership and the binary indicator for the post period was statistically significantly-different from zero. A significant interaction would mean that changes in end point trends because of the policy were different from the control cohorts, independent of seasonality.
Analyses followed the predefined plan in the grant proposal, and the protocol was approved by the ethics boards of the Brigham and Women's Hospital and the University of British Columbia.
Results
The baseline distributions of age, sex, drug use, comorbidity score, and use of healthcare did not differ between the randomised groups (table 1; all P values > 0.1). The observational cohorts had slightly more women than the randomised groups (62% v 59%). Otherwise the observational cohorts were comparable to each other.
Baseline characteristics of patients before randomisation (randomised analysis) or before cohort start (observational analysis) for the evaluation of the reimbursement restriction for nebulised respiratory drugs in British Columbia. Values are numbers (percentages) of patients unless otherwise indicated
Dropout of patients because of death or emigration during the trial period was comparable between intervention group (5%) and control group (8%). These dropout rates were comparable to dropout rates at six months in the observational and historical cohorts (6%, 7%, and 8%; table 2).
Characteristics of patients six months after the start of the formulary restriction for nebulised respiratory drugs in British Columbia. Values are numbers (percentages) of patients unless otherwise indicated
After the policy was implemented the observational analysis indicated that significantly fewer patients used nebulised drugs only (7% v 14%, P < 0.001; table 2 and table 1, respectively) or nebulised in combination with inhaled drugs (21% v 46%, P < 0.001; table 2 and table 1, respectively) compared with levels of use before the policy. By contrast, more moderate reductions became obvious in the randomised analysis for nebulised drug use (6% v 11%, P < 0.01) or nebulised in combination with inhaled drug use (23% v 37%, P < 0.001, table 2). The historical control groups showed some seasonal variation, with higher use of respiratory drugs in the six months preceding 1 March compared with the subsequent six months (table 1 and table 2), underlining the importance of adjusting for seasonal effects in observational analyses (table 3).
Randomised intention to treat analysis and observational repeated measures analysis of the same formulary restriction of nebulised respiratory drugs in adults
In the analysis of the observational cohorts we found that expenditure for nebulised drugs increased by about $C25 per patient month in the month preceding the new policy, followed by an equally lower use in the first month afterwards (see “difference” line in figure 2a). When we excluded these two months of stockpiling and transition the average economic effects in the observational analysis were savings of $C24 per patient month for nebulised drugs and an increase of $C3 per patient month for inhaler drugs (table 3).
Expenditure for nebulised respiratory drugs: a) observational analysis, b) randomised analysis. The vertical lines represent the policy change on 1 March 1999
In the analysis of the randomised groups the estimated savings were $C8 per patient month for nebulised drugs (non-significant, P = 0.24) and spending was $C1 per patient month higher for inhalers (fig 2b and table 3). These estimates are low because many control doctors did not exercise their right to an exemption. In the control group as a whole, prescribing of nebulised drug expenditure dropped about by 60% as much as in the intervention group. Under the reasonable assumption that “non-compliance” with the exemption (or crossover) by control doctors was unrelated to patients' characteristics, we corrected for this misclassification in a sensitivity analysis by using the method of Zelen (p 887).42 This corrected estimate of savings from the randomised analysis was then about $C21 per patient month, very close to the observational estimate.
Both study designs consistently showed no increase in contacts with doctors or admissions to hospital, including emergency admissions (table 3, fig 3).
Rates of contacts with doctors or emergency admissions to hospital in the observational analysis. The vertical lines represent the policy change on 1 March 1999
P values did not change for any of the analyses by more than 0.02 after additional adjustment for clustering of patients within doctors' practices.
Discussion
Restricting reimbursement for nebulised respiratory drugs did not lead to serious adverse effects in adult patients in British Columbia. The rates of contacts with doctors and admissions to emergency departments did not increase as a result of the policy, although the observational study had the statistical power to detect a difference of 7.5 contacts per 100 patient months and 0.5 admissions to hospital per 100 patient months.43 Delayed effects of the policy are unlikely. Our results are in accordance with published guidelines for the treatment of asthma24 and consistent with continued efforts to reduce the use of nebulised drugs in hospitals.44 45 Our randomised and observational study shows that reducing reimbursement in a community setting will lead to a moderate substitution of nebulised respiratory drugs with inhalers without increases in severe adverse effects.
Practicability of a randomised drug policy trial
Pharmacare was surprised by how smoothly the randomised trial melded into the normal implementation of the policy. In 1993 Pharmacare rejected the idea of randomised controls but in 1997, after our opinion research,46 it cautiously approved the present trial. Success with this trial led to approval of two more policy trials yet to be implemented.
This study also shows that in a health system with centralised databases, designing randomised control groups in advance enables scientifically rigorous results almost as soon as the data become available. As the data on drug claims were only one week late and the billing data for medical services about one month late, we were able to present scientific findings seven weeks after the trial ended. Normally observational analysis is delayed because historical controls are assembled after the policy, and complex adjustments are required.
Limitations of this trial
The main surprise for the investigators was the non-compliance in the control group, causing the estimates of drug savings to be $C8 in the randomised analysis and $C24 per patient month in the observational analysis. The non-compliance resulted from a change that Pharmacare made to the protocol at the last minute, which can be avoided in future policy trials. Wanting to underscore the independence of the evaluation from the government, Pharmacare cautiously sent letters announcing the policy change six weeks in advance to all doctors in the province. Control doctors were not told of their exemption until they received a separate letter from the investigators two weeks later. Many control doctors either overlooked the second letter or decided to switch patients to inhaler drugs in anticipation of the new policy. This is evidence of the need for close collaboration between programme managers and investigators when implementing policy trials. Although our cluster randomised, delayed control design is open to such bias, it is the most practical approach that can be integrated in the dynamic process of policy making.
What is already known on this topic
Metered dose inhalers are less expensive than nebulised drugs and deliver respiratory medications to the lung better
Randomised trials in selected patients show that inhalers have similar or better outcomes than nebulisers
Randomised policy trials that evaluate restrictions on reimbursement for drugs are regarded as not feasible or expensive
What this study adds
Restricting reimbursement of nebulisers in favour of inhalers does not trigger adverse outcomes in adults
Randomised drug policy trials are feasible and likely to be concordant with observational evaluations if implemented carefully
Sensitivity analysis
In a sensitivity analysis correcting the randomised analysis for non-compliance, the two evaluation designs gave remarkably concordant results. Longitudinal designs such as our observational analysis are threatened by other differences between historical periods. For example, in the first month after the policy was implemented doctors reduced their office hours to meet budget caps at the end of the fiscal year, possibly more than in the previous year. The randomised analysis was protected from this bias since the concurrent control patients were equally as affected as the intervention patients. The concordance of results shows that, in the absence of randomisation, contemporary techniques for longitudinal data analysis have a good chance of producing valid evaluations of the impacts of drug policies.
Conclusion
Ideally both types of evaluations can be done. A randomised evaluation (assuming no control contamination) can yield solid results within days of the data becoming available, and these can be supported later by observational analyses requiring several months of multivariate statistical methods. We have shown the feasibility of a low cost randomised policy trial and its concordance with a parallel observational study and conclude that randomised evaluation is a promising new avenue for collaboration between decision makers and researchers that can produce rapid, rigorous, relevant evidence.
Footnotes
-
Contributors SS conceived the idea of comparing a randomised policy trial with an observational analysis, was responsible for the overall study design, data collection, statistical analysis, and writing the paper. MM and BC conceived the idea of a randomised policy trial and were responsible for conducting the randomisation and data collection and assisted in the statistical analysis and writing the paper. RJG was responsible for the statistical analysis. JA assisted in the study design, data interpretation, and writing the paper. SS is guarantor.
-
Funding Research grant to SS from the National Institutes of Health (AG19463), Department of Health and Human Services, Rockville, MD, and a grant to BC from the Canadian Health Services Research Foundation. MM is a distinguished scholar of the Michael Smith Foundation for Health Research.
-
Competing interests MM was an employee of British Columbia PharmaCare and is an employee of the Ministry of Health on extended leave. BC has received contracts from PharmaCare to advise on drug policy. RJG has received unrestricted research funding from AstraZeneca unrelated to this research. The authors do not perceive any of these funds competing interests
-
Ethical approval The institutional review boards of the Brigham and Women's Hospital and the University of British Columbia approved the study protocol.