Letters

Meta-analysis of increased inhaled steroid or addition of salmeterol in asthma

Researchers can learn from industry based reporting standards

Greening et al's study should not have been included

Study should have been more thorough

Authors' reply

Researchers can learn from industry based reporting standards

EDITORThe meta-analysis of inhaled salmeterol compared with inhaled steroids by Shrewsbury et al¹ is a fine counter example to recent controversial claims that the standard of industry sponsored meta-analyses in asthma is inferior to that of other meta-analyses.² Nevertheless, the paper raises one general difficulty with the reporting of meta-analyses.

In referring to the quality of individual studies the authors write: "In all studies, appropriate methods were used for summarising and comparing treatments, and methods for handling missing data were preplanned" (p 1368). If, as was almost certainly the case, the studies were carried out to the pharmaceutical industry's standards, the analyses will also have been preplanned as required by the guidelines of the International Conference on Harmonisation.³ It does not follow, however, that this is necessarily the analysis that found its way into print, or even the one that was used for this meta-analysis.

The version published by investigators is often inferior, as regards statistical analysis, to that presented to the regulator. This is because editors are less exacting and readers are less tolerant when it comes to statistics than are regulators; sponsors cannot control publications to the same extent as they do trial reports.

It seems unlikely that the scale on which Shrewsbury et al chose to summarise outcomes will have been the same as that in the original trials, which may have varied from trial to trial. This raises the question of prespecification of methods for meta-analysis. For example, the authors chose the binary probability scale for the outcome "exacerbation." But the original scale was "ordered categorical," which suggests logistic regression. Furthermore, a summary on the log-odds scale (the usual default) for the data presented in their paper would disagree with their finding that salmeterol is significantly better than steroid. For example, a fixed effects analysis gives an odds ratio of 0.87 with a 95% confidence interval of 0.74 to 1.03, straddling 1.

I hope that pharmaceutical companies will not only follow Sykes's call to publish all clinical trials⁴ but also have detailed summaries (as provided to regulators) available on the web as well. Not only would this increase confidence in industry based meta-analyses but it would allow researchers to learn from industry based reporting standards.

Stephen Senn, professor of pharmaceutical and health statistics. 
University College London, London WC1E 6BT stephens{at}public-health.ucl.ac.uk

Competing interests: Professor Senn is a consultant to the pharmaceutical industry.

Greening et al's study should not have been included

EDITORThe introduction to Shrewsbury et al's meta-analysis begins: "The 1997 British Guidelines on Asthma Management acknowledged the landmark study of Greening et al."¹ The paragraph in the methods section headed "Quality assessment" begins: "All included studies were sponsored by GlaxoWellcome and all met company-wide minimum quality thresholds." The paragraph finishes with a sentence beginning: "In all studies, appropriate statistical methods were used for summarising and comparing treatments." I am surprised that the authors included Greening et al's study in the meta-analysis.² It is apparent from its methodology that a confounder was introduced that rendered the results uninterpretable.

The study states, "During the 6-month treatment period patients received either beclomethasone diproprionate (BDP) (200 µg twice daily by metered dose aerosol inhaler (MDI)) plus salmeterol (50 µg twice daily by Diskhaler) or higher dose BDP (500 µg twice daily by MDI) plus placebo."² These two arms of the treatment phase were randomised after an initial two week run in period, during which patients took inhaled beclomethasone diproprionate (200 µg twice daily), while bronchodilators were replaced with salbutamol (by Diskhaler) to be used as required.

Both arms of the study used beclomethasone diproprionate via a metered dose inhaler, but salmeterol via a Diskhaler was selected for addition to the lower dose arm. Thus the treatment phase compared an increased dose of beclomethasone diproprionate via metered dose inhaler with added salmeterol via a Diskhaler.

It would be difficult to conclude whether any significant difference between the two groups was due to the salmeterol or the use of a breath actuated Diskhaler. In other words, any benefits to the patients may be assumed to have been due to the added salmeterol but may have been due to the better inhaler device.

The dry powdered breath actuated Diskhaler has inherent advantages over the aerosol metered dose inhaler, and it is therefore difficult to interpret this study's conclusions. I have discussed the methodological flaw with many of GlaxoWellcome's representatives, all of whom acknowledge the methodological design fault. It is therefore surprising to see this study included in Shrewsbury et al's meta-analysis. An independently conducted meta-analysis reviewing all long acting ₂ agonists would be more credible.

Finally, as stated in the competing interests statement, two of the three authors work for GlaxoWellcome and the third has shares in the company. I am concerned about the potential conflict of interests when pharmaceutical companies perform meta-analyses of their own sponsored trials in reputable journals.

A D Greig, general practitioner. 
Bourne Valley Practice, Ludgershall, Andover, Hampshire SP11 9TA roskoff{at}msn.com

Competing interests: Dr Greig has been paid as a speaker by both Allen and Hanbury and Astra Pharmaceuticals.

Study should have been more thorough

EDITORShrewsbury et al's meta-analysis of studies sponsored by GlaxoWellcome was not systematic and has many pitfalls that need to be addressed.¹ The authors restricted themselves to only three databases in their search for studies. Electronic searches of the literature may identify as few as half of the relevant studies.² Use of the Cochrane Airways Group (St George's Hospital Medical School, London) database of randomised controlled trials could have resulted in the inclusion of further studies. This database not only includes studies from other electronic databases but also incorporates systematic hand searching (retrospective and prospective) of many core journals in respiratory disease in an attempt to improve the thoroughness of electronic searching in this subject.

The authors failed to apply independently validated methodological quality assessment criteria to the included studies³ and instead used in-house assessments, for which little detail is provided. Readers are unable to judge the quality of included studies. The authors noted heterogeneity in symptom scores and rescue treatment but failed to conduct any sensitivity analysis (for example, dose or type of inhaled steroid, severity of asthma, or study quality). A meta-analysis should be planned and conducted with predefined subgroup analyses based on clinical predictions.

The authors noted that the heterogeneity is "almost certainly clinically unimportant" since the difference between the fixed and random effects models is small. Changing from a fixed to a random effects model tells you little if anything about the clinical importance of heterogeneity. If it does not make statistical sense (significant heterogeneity) to combine results it certainly won't be worth deriving any clinical conclusions from the outcome in question. Use of additional rescue bronchodilator is an important indicator of the efficacy of salmeterol, and symptoms are an important patient oriented outcome measure. A formal sensitivity analysis based on at least some of the above criteria should be explored.

As with any trial, when a meta-analysis is published it should provide readers with enough information for them to be able to duplicate the study. In this case most readers would have difficulty since unreported in-house quality assessment criteria were used and little information was provided on studies that were excluded. This meta-analysis needs to be conducted again with more thoroughness and perhaps based more closely on the superior methodology used by the Cochrane Collaboration for systematic reviews.⁴

Felix S F Ram, respiratory research fellow. 
St Luke's Hospital, Bradford BD5 0NA felix.ram{at}emailbradh-tr.northy.nhs.uk

Competing interests: None declared.

Authors' reply

EDITORIn planning and conducting this trial we have aimed to adhere to current best practice, conducting our review to a predefined protocol (itself modelled on the guidance offered in the Cochrane Reviewers' Handbook¹) and reporting it as recommended by the QUOROM statement.² Our search was comprehensive: the Cochrane database yielded 263 777 references for salmeterol recently; 17 mentioned both inhaled steroid and asthma in their title. Hand searching these showed no missing studies in our analysis.

We made effective use of our access to detailed study level materials, obtaining full clinical study reports and in many cases the raw data, which enabled a more comprehensive and standardised comparison across studies. The scale of measurement was prespecified, although in the case of exacerbations a change to odds ratio has no material effect on the conclusion presented: exacerbation rates were not increased with add-in salmeterol. Nine studies were identified by this review, and therefore subgroup analyses would have been of limited value. The more conservative random effects model showed greater benefit with added salmeterol.

Many doctors regard exacerbations of asthma as a reasonable guide to airway inflammation when measurement of bronchial hyperreactivity or biopsies are unavailable. Our finding of no increase in exacerbation rates with add-in salmeterol is supported by two bronchial biopsy studies by Li et al³ and Sue-Chu et al.⁴ In these, patients receiving salmeterol plus low dose inhaled corticosteroid (compared with increased corticosteroid) showed no increase in inflammatory cells (and fewer eosinophils³ or mast cells⁴) in the respiratory epithelium.

Salmeterol costs more than theophyllines. If cost were the only consideration in asthma treatment, however, we would still be using prednisolone for maintenance. Doctors will instead wish to weigh the potential for increased costs against the benefits of increased efficacy or improved safety, or both, when judging what is best treatment. A Cochrane review supports numerous studies showing superior efficacy and tolerability of salmeterol compared with theophyllines.

The weight that readers of our paper attach to it is for them to judge in the knowledge of our affiliations, openly stated. The value of this work is, we believe, that it addresses the dilemma at step 3 of the British Guidelines on Asthma Management. Other papers have compared the efficacy and safety of salmeterol with regular salbutamol, while concerns have been expressed about the overuse of short acting agents. We believe that our findings support the choice of adding in salmeterol as a superior alternative to increasing the dose of inhaled corticosteroids.

Stephen Shrewsbury, associate medical director. 
Stephen Pyke, section head, respiratory statistics. 
Mark Britton, consultant respiratory physician. 
GlaxoWellcome UK, Uxbridge, Middlesex UB11 1BT Sbs40926{at}glaxowellcome.com

Stephen Shrewsbury and Stephen Pyke are full time employees of GlaxoWellcome; Mark Britton has been taken to international conferences, has received fees for speaking at conferences and for research and a respiratory nurse, and has shares in GlaxoWellcome.