Rapid Responses to:
|
|
Rapid Responses: Rapid Responses published:
-
![[Read Rapid Response]](/icons/misc/sectionDOWN.gif)
Problems with the summary estimates
- Christopher Cates (18 October 1999)
-
Why exclude foreign language publications?
- Adam Jacobs (19 October 1999)
-
Helicobacter pylori and non-ulcer dyspepsia
- Brian Scott (19 October 1999)
-
Why being a reviewer in the Cochrane Collaboration is a better deal than working outside it!
- Monica Fischer (20 October 1999)
-
A potentially misleading meta-analysis
- Brendan Delaney (25 October 1999)
-
Re: Problems with the summary estimates
- Liisa Jaakkimainen, Eleanor Boyle, Fred Tudiver (2 November 1999)
-
Authors'' responses
- Liisa Jaakkimainen, Eleanor Boyle, Fred Tudiver (2 November 1999)
-
H.pylori vs NUD: A mis-trial
- Deirdre McNamara (9 November 1999)
|
|
|||
|
Christopher Cates, GP and Criticism Editor Cochrane Airways Review Group Manor View Practice, Bushey Health Centre, London Road, Bushey, Herts WD2 2NN, UK
Send response to journal:
|
EDITOR - This meta-analysis concludes that " An improvement in dyspeptic symptoms occurred among patients with non-ulcer dyspepsia in whom H pylori was eradicated. " Unfortunately this is untrustworthy due to a small but crucial problem at the heart of the analysis. The authors report that the summary estimates are statistically homogenous, but the reported p values indicate a very low likelihood that this statement is correct. Thus in the observational studies the p value of <0.001 indicates massive heterogeneity between the results of the included studies. In the therapeutic trials the p value of 0.046 also indicates significant heterogeneity. Meta-analysts faced with such heterogeneity have three choices. They may ignore the heterogeneity and pool the results with a fixed effects model, or they may use a random-effects model which takes the heterogeneity into account or they may decide not to pool the results. In this instance the authors chose to use a fixed effects model in spite of the heterogeneity. In consequence the confidence intervals of the pooled estimates are very narrow and statistical significance is imputed where a random model might well lead to a summary estimate that does not reach significance. At present the extracted data from the individual trials has not been published on the web-site so it is not possible to check this. The authors have sensibly carried out a sensitivity analysis of the therapeutic trials and noted that the single regimen trials only measured short term outcomes and were of lower methodological quality. The two trials on triple therapy measured outcomes at 12 months and were of higher quality; these triple therapy trials showed a much smaller summary estimate of eradication that barely reaches significance (Odds Ratio 1.4 , 95% Confidence Interval 1.0 to 2.3). There is clinical as well as statistical heterogeneity between these two groups of studies and a summary estimate which combines both is of doubtful meaning. Thus in my opinion the best option faced with this data is not to pool the triple therapy trials and single therapy trials. For this reason the conclusion that "Eradication of H Pylori was associated with an almost two-fold increase in dyspeptic symptoms among patients referred to specialist clinics" is misleading because it is based on a summary estimate that makes no clinical sense and is statistically questionable. Until a systematic review is carried out with a wider search strategy and more robust statistical analysis I do not think this meta-analysis should influence guidelines or clinical practice. |
|||
|
|
|||
|
Adam Jacobs, Director Dianthus Medical Limited
Send response to journal:
|
I note that Jaakkimainen et al mention that they excluded articles published in languages other than English. Perhaps they had a perfectly valid reason for this; if so, I invite them to share it with us. It is more worrying if the reason was simply that their budget did not extent to the costs of having papers translated. That would be an arbitrary exclusion of potentially relevant and important research, which seems to go against the idea of a systematic review. It is true that the cost of doing a meta-analysis could be substantially increased if many papers need to be translated. However, when compared with the costs of the original research, the costs of translation are vanishingly small. It seems a shame if those costs are allowed to stand in the way of an inclusive an meaningful synthesis of the research. |
|||
|
|
|||
|
Brian Scott
Send response to journal:
|
EDITOR - While not wishing to criticise the thorough systematic review and meta-analysis by Jaakkimainen and colleagues (1), I would like to draw attention to the lack of diagnostic precision for non-ulcer dyspepsia found in many published papers. Most papers accept a diagnosis of non-ulcer dyspepsia when there has been a normal endoscopy whether or not it was done when the patient was symptomatic. This is unacceptable when trying to exclude intermittent pathology like peptic ulceration. I have drawn attention to this problem elsewhere (2) and illustrated the problem by reference to a paper by McColl and colleagues (3) in which at least four patients, who were diagnosed as non-ulcer dyspepsia, subsequently were found to have peptic ulcer disease. It is thoughtless and misleading to arrange endoscopy, with the intention of confirming or refuting peptic ulceration as a cause of symptoms, at a time when they are not symptomatic. As a result of this meta-analysis, based on faulty data, many GPs will be encouraged to eradicate H pylori needlessly. I draw their attention rather to a recent well conducted study (4) in which all patients labelled as non-ulcer dyspepsia were symptomatic on at least three days during the week before endoscopy. No advantage was shown by eradication in such patients. It is worth stressing that currently there are only two conditions in which there is good evidence favouring eradication - peptic ulceration and the very rare gastric maltoma. Brian Scott 1) Jaakkimaimen RL, Boyle E, Tudiver F. Is Helicobacter pylori associated with non-ulcer dyspepsia and will eradication improve symptoms? A meta-analysis. BMJ 1999;319:1040-4. 2) Scott BB. Helicobacter pylori and non-ulcer dyspepsia. NEnglJM 1999;340:1509. 3) McColl K, Murray L, El-Omar E, et al. Symptomatic benefit from eradicating Helicobacter pylori infectionin patients with nonulcer dyspepsia. NEnglJM 1998; 339:1869-74. 4) Talley NJ, Vakil N, Ballard ED, Fennerty MB. Absence of benefit of eradicating Helicobacter pylori in patients with nonulcer dyspepsia. NEnglJ Med 1999;341:1106-11. |
|||
|
|
|||
|
Monica Fischer, Associate Director The Nordic Cochrane Centre
Send response to journal:
|
The first author of this meta-analysis was notified by a Cochrane reviewer that he suspected that some of the results reported in the article were wrong due a problem with the statistical package used for analysing the data, the Cochrane Collaboration's review writing tool, Review Manager 4.0 (RevMan 4.0) [1]. It must have been frustrating for the authors when they realised that there was a possibility that RevMan 4 might have produced faulty data. This could have been avoided if the authors had been working with the relevant Collaborative Review Group (CRG) in the Cochrane Collaboration, in this case the Cochrane Upper Gastrointestinal and Pancreatic Diseases Group. RevMan is a review writing tool designed and developed by the Cochrane Collaboration. It includes the software MetaView [2] that calculates the statistical data and displays the results both in a tabular and graphical format. In order to have a Cochrane Review published in the Cochrane Database of Systematic Reviews [3], reviewers must use RevMan and therefore also MetaView. The software is available free of charge to everyone who has access to the Internet. It should be noted, however, that when new software is released, problems are often identified. This also happened with RevMan 4.0/MetaView 4.0. Soon after its release in July 1999, problems were discovered with the statistical calculations in MetaView 4.0 and information on those were immediately sent to the CRGs which warned their reviewers. Any user of RevMan/MetaView is encouraged to subscribe to the RevMan discussion list through which they will receive details about possible problems and known bugs. The Cochrane Collaboration does not in any way want to restrict people outside the Collaboration from using RevMan, but when they do, they should be aware of the risks that it carries. Only by being a member of a CRG is a reviewer guaranteed technical support and information on updates or corrections to the software. Reviewers will have access to a network of people who are dedicated to help them produce good systematic reviews and keep them up-to-date, and a by being a member of a CRG, they will help minimize the risk of preparing a review which may already be in progress elsewhere. 1. Review Manager (RevMan) [Computer program]. Version 4.0 for Windows. Oxford, England: The Cochrane Collaboration, 1999. 2. MetaView [Computer program]. Version 4.0 for Windows. Oxford, England: Update Software, 1999. 3. The Cochrane Database of Systematic Reviews. In: The Cochrane Library, Issue 4, 1999. Oxford: Update Software. Updated quarterly. |
|||
|
|
|||
|
Brendan Delaney, Senior Lecturer Dept. of Primary Care & General Practice, The University of Birmingham, UK.
Send response to journal:
|
Editor -
The conclusions reached by Jaakkimainen et al [1] differ from those that we have reached in a systematic review, addressing the same question, for the UK Health Technology Assessment Programme. We are thus concerned that the Jaakkimainen et al paper may provide a seriously misleading impression of the effect of H.pylori eradication therapy on symptoms of non-ulcer dyspepsia. Our review was conducted using a protocol peer reviewed by the Cochrane Collaboration and will be submitted to The Cochrane Library. Problems we have identified with the meta-anlayisis of RCTs by Jaakkimainen et al include: 1. The search strategy is substantially incomplete with only one electronic database being searched, no text words being used, no 'grey literature' (important for obtaining papers in press in a fast-moving field) and non-English language papers excluded. 2. We believe that there is a potential problem with the selection of trials. 'Single therapy' trials will not have eradicated H.pylori adequately, and may be confounding where the treatment may have benefits on dyspepsia other than H.pylori eradication (e.g. Bismuth and Erythromycin). [2] 3. We are concerned by the exclusion of the trial by McColl at al, [3] and wonder whether this was on account of the trial not excluding all patients with reflux-like symptoms. A list of excluded studies, and reasons for exclusions, should be available on the BMJ website. 4. As already discussed by Cates, [4] the misinterpretation of odds ratios as effect sizes, the handling of tests for heterogeneity, and the application of a fixed effects model are all serious flaws in the analysis. Even on the basis of the studies presented here it is simply not true to say that 'eradication of H.pylori is associated with an almost twofold improvement in dyspeptic symptoms'. 5. The quantitative estimate that is required by clinicians and researchers planning trials is a measure of the likely absolute benefit (or NNT) of H.pylori eradication in non-ulcer dyspepsia. Nowhere in this paper are any figures quoted that could be used to determine what this might be. As four, large, high quality, studies have been published in the past 12 months, and we are in possession of data from a fifth trial currently in press, we would be interested to compare our result with the equivalent estimate from this paper. We expect that the benefit from H.pylori eradication will be modest, and any application in clinical practice would require careful consideration and a supportive cost-effectiveness analysis in comparison with alternative therapies. We would like to extend the comments of Fischer,[5] and emphasise how contact with the relevant Cochrane Review Group is helpful in ensuring quality in all phases of systematic reviews from the search strategy to statistical analysis. Reviewers benefit considerably from the support and peer review procedures of the Collaboration. 1. Jaakkimainen RL, Boyle E, Tudiver F. Is Helicobacter pylori associated with non-ulcer dyspepsia and will eradication improve symptoms? A meta-analysis. BMJ 1999; 319: 1040-1044. 2. Baron JH, Barr J, Batten J, Sidebotham R, Spencer J. Acid, pepsin, and mucus secretion in patients with gastric and duodenal ulcer before and after colloidal bismuth subcitrate (DeNol). Gut 1986; 27: 486-490. 3. McColl K, Murray L, El-Omar E, Dickson A, El-Nujumi A, Wirz A, Kelman A, Penny C, Knill-Jones R, Hilditch T. Symptomatic benefit form eradicating Helicobacter pylori infection in patients with nonulcer dyspepsia. New England Journal of Medicine 1998; 339: 1869-1874. 4. Cates C. Problems with the summary estimates. e-BMJ (18 October 1999) www.bmj.com/cgi/eletters/319/7216/1040#EL1 5. Fischer M. Why being a reviewer in the Cochrane Collaboration is a better deal than working outside it! e-BMJ (20 October 1999). www.bmj.com/cgi/eletters/319/7216/1040#EL4 Brendan Delaney, Senior Lecturer, Dept of Primary Care and General Practice, The University of Birmingham, UK Paul Moayyedi, MRC Training Fellow in Health Services Research, Centre for Digestive Diseases, University of Leeds, UK Jon Deeks, Director, Systematic Review Programme, Centre for Statistics in Medicine, University of Oxford, UK Shelly Soo, Clinical Research Fellow, Centre for Digestive Diseases, University of Leeds, UK David Forman, Co-ordinating editor, Cochrane Upper GI and Pancreatic Diseases Review Group, 30 Hyde Terrace, Leeds, UK |
|||
|
|
|||
|
Liisa Jaakkimainen , Eleanor Boyle, Fred Tudiver
Send response to journal:
|
We thank Dr. Cates for his comments with respect to our meta-analysis and systematic review. With respect to the heterogeneity found in the summary estimate of the association studies, the sensitivity analyses were conducted to explain and explore the robustness of the summary estimate and the reasons for statistical heterogeneity. Statistical heterogeneity did in fact improve in the subgroup analyses (study design and study quality). In addition, the summary estimate remained stable during the sensitivity analysis. We feel non-ulcer dyspepsia is a heterogenous condition. Considering that multiple populations were examined and varying definitions of dyspepsia were used, having statistical heterogenity for the summary estimate is not surprising. The summary estimate from the five eradication trials is not generalizable. We mostly base this argument on the lack of robustness of the estimate when performing the sensitivity analysis and not so much on the issue of heterogeneity (which depending on the level of statistical significance may or may not be a problem). We do not feel that the trials available in the English published literature are representative of the general population of patients with dyspeptic symptoms. We agree that random effect models produce more conservative estimates (1), yet there is also debate on the appropriateness of their use (2,3). More important is the fact that the estimate is based on very few trials and we agree that the inclusion of more studies will produce a more stable estimate. Of note, a recent trial by McColl et al (4) was not included in the estimate for the sole reason that the format of the published data could not be utilized in our analysis. This trial supported the association between eradication and symptomatic improvement. Because of the limited nature of the results of our systematic review, we believe further clinical research necessary before we can make decisions about changing practice with respect to this disease. We feel evidence based research representing a primary care practice is needed. There are many factors that need to be considered when trying to alter or influence clinical practice. Hp eradication is only a part of the larger dilemma of providing evidence-based clinical information about the management of dyspepsia amongst primary care patients. Changing clinical practice for a common condition having multiple therapeutic strategies requires a broad look at the literature that includes results of clinical trials, meta-analyses, cost-effectiveness studies, cost-utility studies and a full understanding of the consequences of treatment. For non-ulcer dyspepsia, this includes fully examining acid suppression therapy, motility agents and lifestyle changes, along with Hp eradication. Liisa Jaakkimainen Eleanor Boyle Fred Tudiver 1 Berlin JA, Laird NM, Sacks HS, Chalmers TC: A comparison of statistical methods for combining event rates from clinical trials. Stat Med 1989;8:141-151. 2 Thompson SG, Pocock SJ. Can meta-analysis be trusted? Lancet 1991;338:1127-1130. 3 Petitti DB. Meta-Analysis, Decision Analysis and Cost- Effectiveness Analysis. Methods for Quantitative Synthesis in Medicine. Oxford University Press 1994 pp90-114 4 McColl K, Murray L, El-Omar E, et al. Symptomatic benefit from eradicating Helicobacter pylori in patients with non-ulcer dyspepsia. N Engl J Med 1998;339:1869-1874. |
|||
|
|
|||
|
Liisa Jaakkimainen , Eleanor Boyle, Fred Tudiver
Send response to journal:
|
Re: Helicobacter pylori and non-ulcer dyspepsia. Dr. Scott raises an interesting point about the timing and relationship between symptom data and endoscopic results. If the process of referral for endoscopy were organized so that patients were investigated shortly after seeing a primary care physician for their dyspeptic symptoms, then this argument would be mute. Often this is not the case. The correlation of dyspeptic symptoms with endoscopic findings is not straightforward. As in the case of NSAID-related gastrointestinal hemorrhage where prior to their presentation to emergency rooms many patients are asymptomatic (1,2). We agree with Dr. Scott and believe further clinical research necessary before we can make decisions about changing practice for the management of dyspepsia. References: (1) Collins AJ, Davies J, Dixon, AJ. Contrasting presentation and findings between patients with rheumatic complaints taking nonsteroidal anti-inflammatory drugs and a general population referred for endoscopy. Br J Rheumatol 1986;25:50-53 (2) Aabakken L. Clinical symptoms, endoscopic findings and histologic features of gastrointestinal non-steroidal anti-inflammatory drug lesions. Ital J Gastroenterol Hepatol 1999;31 Suppl:S19-S22. Liisa Jaakkimainen Eleanor Boyle Fred Tudiver Re: Why exclude foreign language publications? We agree that the inclusion of English-only published studies introduces some selection bias in any review. It is possible to translate original articles, especially the European-based languages. However, it may be more difficult to translate African and Asian-based languages. Unless a researcher is committed to translate all articles, we would be concerned that a differential selection bias may be introduced. Liisa Jaakkimainen Eleanor Boyle Fred Tudiver Re: Why being a reviewer in the Cochrane Collaboration is a better deal than working outside it! We thank Dr. Fischer for her concern about frustrations with respect to the RevMan Cochrane Collaboration package. We were surprised to find out that technical errors existed within the software. Fortunately, we used several software packages for our analyses and did not rely on the Cochrane software. In addition, the data analysis for this paper was started well before July 1999. We in fact used an earlier version of the package (RevMan3). We, regrettably, referenced RevMan 4 because we assumed it was a reliable package to which others interested in doing meta- analyses could use. Liisa Jaakkimainen Eleanor Boyle Fred Tudiver |
|||
|
|
|||
|
Deirdre McNamara, Specialist Registrar in Gastroenterology & Lecturer in Medicine at Trinity Adelaide & Meath Hospitals, Tallaght, Dublin 24
Send response to journal:
|
Dear Editor, We would like to congratulate Jaakkimainen et al for their recent meta- analysis on H.pylori infection and non-ulcer dyspepsia, (BMJ 1999;319:1040 -1044.) Such studies are important and will aid to clarify currently controversial issues. Unfortunately the conclusions drawn regarding the efficacy of H.pylori eradication for non ulcer dyspepsia may be questionable as only five treatment studies were included, a source of bias recognised by the authors in their discussion. There were several key studies omitted, despite apparently meeting the inclusion criteria, namely being randomised control trials, with accepted definitions of dyspepsia and non ulcer dyspepsia, employing accepted and effective eradication regimens and having dyspepsia symptoms as a defined outcome measure. Mc Coll et al, (N Eng J Med 1998;339:1869-74) randomised 160 patients to omeprazole, amoxicillin and metronidazole and 158 to placebo. At one year a significant benefit in symptom resolution was shown, employing a validated Glasgow dyspepsia severity score, for those who became H. pylori negative, 21% versus 7% for those remaining infected. Similarly, an earlier study by Gilvarry et al (Scand J Gastroenterol 1997;32:335-340) reported a significant symptom reduction in patients successfully treated with bismuth, tetracycline and metronidazole compared to bismuth and placebo, symptom score of 14.2 and 9.2 at inclusion and at 1 year follow up respectively. A contradictory and equally valid study by Talley et al (The Optimal Regimen Cures Helicobacter Induced Dyspepsia (O.R.C.H.I.D) Study group) was also not included for analysis. Two hundred and seventy-eight subjects were randomised to PPI based triple therapy or placebo, there was no difference in symptom scores at 1 year between either group. However, an improvement of symptoms with resolution of chronic gastritis was reported. Although these studies are not perfect - assessment of compliance, description of randomisation process and even presentation - as referred to in the meta-analysis, their design is adequate and their findings significant. Lack of inclusion of these studies, could alter the findings and conclusions of this meta-analysis. The Jury in the criminal trial of H.pylori infection is still out, the evidence put forward should include all relevant information lest a hung jury be the end result of long deliberation. Sincerely, Dr Deirdre Mc Namara, Specialist Registrar Dr M Buckley, Consultant Gastroenterologist Prof. C. O’Morain, Consultant Gastroenterologist, Academic Head of Department of Medicine TALLAGHT • DUBLIN 24 • IRELAND |
|||