Jump to: Page Content, Site Navigation, Site Search,
You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.
Rapid Responses to:
|
|
Rapid Responses published:
|
|
|||
|
Ralph Crott, Head Health Economics Unit, EORTC EORTC, Brussels, 1200
Send response to journal:
|
To the editor, Dear Sir, The new BMJ publication policy on economic studies raises a number of practical questions. First, submitting together the clinical trial results and the economic analysis creates the risk of having one rejected and the other not. So what policy will the BMJ then follow ? Second, economic analyses are often performed after the clinical trial is closed and analyzed first. A good reason for this is that the clinical results have to inform a number of methodological choices for the economic analysis (for example if there is no significant treatment difference should one then choose for a cost-minimization analysis ?). Furthermore while analysis of the clinical results can be done relatively rapidly as it uses the data collected in the clinical report forms, an economic analysis often has to collect many additional outside trial data from a variety of sources, therefore lengthening quite a bit the time needed to perform the analysis. On top of this more than often a separate (re)analysis of the trial's clinical data is needed as well. Waiting for the separate economic analysis to be finished would unduly delay the publication of the clinical results and would not be acceptable for most of the clinical investigators. Yours Sincerely, R. CROTT, PhD, MPH. Competing interests: None declared |
|||
|
|
|||
|
John Forbes, economist The University of Edinburgh
Send response to journal:
|
The decision to publish comprehensive evaluations reporting "clinical" and "economic" findings is to be applauded. The artifical and largely arbitrary distinction between different types of treatment effects has tended to undermine the results and, more importantly, the interpretation and application of many economic evaluations in health care. Your policy may mean that fewer economic evaluations are published in the BMJ but at least those that satisfy this new condition will be more powerful and command the attention of those interested in coherent economic analyses based on all of the relevant scientific evidence generated by robust study designs. Competing interests: None declared |
|||
|
|
|||
|
Charles Normand, Professor of Health Economics London School of Hygiene & Tropical Medicine, Keppel Street, London WC1E 7HT
Send response to journal:
|
I understand the reasons for the new policy on publishing economic evaluation studies, but it is not clear how this will apply to many of the best evaluations that are based on reviews of many RCTs and other clinical studies, and use modelling to assess outcomes and cost-effectiveness. There is no reason to exclude such studies. If the new policy is to work it is important also for the BMJ to ensure that its processes of review and decision making are 'joined up' in terms of the different components of studies. Too often in the past when pairs of papers are submitted or when a single paper reports the overall results of a study the reviewing of the economics has been weak. Richard Smith's editorial raises the point that publishing clinical outcomes without economic ones is really incomplete evaluation. I look forward to results of high quality clincal trials being rejected for want of a proper economic evaluation. Competing interests: None declared |
|||
|
|
|||
|
Anna E Bucsics, Hauptverband der österr. SV-Träger A-1030 Vienna, Austria
Send response to journal:
|
Your policy is laudable, because an economic evaluation is incomplete without clinical data, and we all know that "the devil is in the details". However, if the publication of the clinical data is freely available (e.g. on the Net), readers have the information they need. So you might consider accepting economic evaluations of clinical trials if the publication is freely available on the Internet - if their quality is up to standards, of course! yours sincerely, Anna Bucsics Competing interests: None declared |
|||
|
|
|||
|
David P McDaid, Research Fellow LSE Health and Social Care, London School of Economics, Houghton Street, London, WC2A 2AE
Send response to journal:
|
The new BMJ policy to publish economic evaluations alongside clinical trial results is broadly to be welcomed. The policy does though raise a number of questions to be considered. One question concerns the current 2,000 word limit for article submissions. It is difficult enough, given the word limit for authors to include sufficient information to satisfy the current BMJ guidelines on economic evaluations. One possibility may of course be to publish both papers separately in the same issue of the journal, but this would seem to be contrary to the spirit of the new policy. If more combined clinical papers are to be published, then the journal should seriously consider raising the word limit for submissions, or make even greater use of the electronic version of the journal to publish supplemental information. A second issue concerns the type of economic evaluation undertaken. Evidence on outcomes often are generated from systematic reviews of previous studies, and modelling, an important and useful tool in the absence of information from clinical trials, may be used to synthesise data on outcomes and costs. How will the journal treat such studies? This issue is of particular importance given moves in Europe and elsewhere to improve the generalisability of economic evaluations, by adjusting results and methods to take account of local contexts. It may also be the case that an economic evaluation is not conducted alongside a clinical trial but retrospectively; will such studies be excluded in their entirety? There is some justification for at least publishing such papers, where relevant clinical paper has previously been published by the BMJ or one of its sister publications. Consideration needs to be given to these questions, otherwise there is a danger that a well intentioned move by the journal may throw out the baby along with the bath water! Competing interests: None declared |
|||
|
|
|||
|
Tracy E Roberts, Lecturer in Health Economics University of Birmingham, Health Economics Facility, 40 Edgbaston Park Road, Birmingham, B15 2RT
Send response to journal:
|
I agreed with the principles underlying your arguments regarding new policy on economic evaluations – initially (1). But on reflection the implications are not clear. My first concern is that there is often a lag between the clinical results and the economic results and it may not be possible to submit both simultaneously. Clinicians are often eager to get their results published immediately and are not keen to wait until all the costs have been added up. The ECMO trial was among the first multi-disciplinary research projects to incorporate economic evaluation in its design from the outset. But the preliminary clinical results were written up and fast tracked to the Lancet before I was even employed to continue the economic evaluation (2). These clinical results were preliminary but dramatic. The economic results were subsequently published in the BMJ, alongside a commentary by Tom Jefferson suggesting a number of reasons for the time lag (3). But the explanation is simple: the economic evaluation relied on the clinical evidence before its results could be finalised. Whilst the economists on the project were not entirely happy with the situation, would it have been beneficial to anyone to sit on the clinical results which showed the new technology Extra Corporeal Membrane Oxygenation (ECMO) saving twice as many infant lives as the conventional treatment, for the next 12 months waiting for the final economic results? Some clinical results may be more generalisable to an international audience than the concurrent economic results. Input costs and the value of resource used eg labour, and differing policies towards discount rates used in the analysis, all vary across countries and thus have varying implications for economic results. The limitations of the clinical information, in the absence of economic evidence, could (and should) be made explicit. The pertinent question here is not dissemination to clinical audiences but restraint by policy makers. For instance should the Department of Health have adopted any policy with regard to ECMO before the results of the economic evaluation were ready? My second concern is the failure of any incentive in the new policy for clinicians to change their practice. Presumably clinicians send trial results to the Lancet for higher impact factors and wider dissemination. If economists cannot persuade clinical colleagues to submit the clinical paper alongside the economic results to the BMJ, they will resort to targeting results to economic journals. Consequently, economic results will be written up in a different style to suit a specialist audience, ensuring even poorer dissemination of economic evidence to clinical audiences and possibly policy makers. Finally, your editorial presented strong arguments in favour of keeping clinical and economic results together and you state “ Send somebody else your clinical results and us your economic results, and we will send them back, politely”. May I therefore ask, politely, is the converse also true? Will you return clinical trials if submitted without any economic results? 1. Smith R, New BMJ Policy on Economic Evaluations: we won't publish economic evaluations unless offered the clinical results as well. Editorial. BMJ 2002; 325:1124 2. UK Collaborative ECMO (Extracorporeal Membrane Oxygenation) Trial Group. UK collaborative randomised trial of neonatal extracorporeal membrane oxygenation. Lancet 1996; 348:75-82 3. Tracy E Roberts and the Extracorporeal Membrane Oxygenation Economics Working Group on behalf of the Extracorporeal Membrane Oxygenation Trial Steering Group. BMJ 1998; 317:911-6 Competing interests: None declared |
|||
|
|
|||
|
Michael V Hurley, Reader in Physiotherapy Rehabilitation Research Unit
Send response to journal:
|
Dear Sir, I read with mixed feelings the BMJ's new policy about publishing papers of economic evaluation only if the clinical trial is also sent. I fully support your assertion that people can only gain the fullest information for decision-making if all the relevant facts are presented together, and believe it will strengthen the BMJ’s standing in the field of applied clinical research. However, your policy of integrating the clinical and economic analysis results in one paper is likely to create problems for the editors, publishers, reviewers, researchers and readers. Firstly, to incorporate essential methodological and data analysis details of an economic evaluation into a paper reporting a complex clinical trial, and then integrate the clinical and economic inferences in the Discussion will produce a very wieldy paper, which will be extremely difficult to write, review, read and make inferences from, taxing the concentration and fortitude of all but the most interested. Pity the poor reviewer who will require a diverse (and uncommon?) range of skills; clinicians with an up to date understanding of the nuances of health economic evaluations and health economists with an understanding of specific clinical areas. If these criteria are not fulfilled “good” papers may be erroneously rejected and “poor” papers accepted. Your commendable aspirations may work in reverse. What will be your policy if the clinical aspect of a trial is excellent and relevant but the economic aspect flawed, or vice versa? The clinical results wont necessarily be undermined because other “unrelated” aspects of the trial were not (or cannot be) so rigorous, but the results are likely to be tainted in the readers mind. Will you, for clarity or space pressures, publish only the clinical results “suppressing” the other flawed results? Is this tantamount to publishing selective results? In addition, your policy may put another hurdle in the way of clinical researchers (whose fits of pique and petulance surpass even those of BMJ’s editorial board), and hinder our ability to obtain funding to carry out relevant applied clinical research. Already we are battling with a perverted system that demands and rewards quantity rather than quality of research. Adequately designed, complex, clinical trials are lengthy and produce relatively few papers compared with other disciplines, where small “salami-ed” studies produce enviable large publication records, but usually less useful - if not completely redundant – clinical applications. To address these problems and ensure the BMJ continues to receive and publish important relevant papers, perhaps the Editorial board would consider publish separate papers back to back in one issue one covering the clinical aspects of the trial, the other the economic aspects. These could be reviewed and read separately easing the burden on reviewers, ensuring that appropriate comments are raised and addressed pre- publication, presenting to the reader all the pertinent information in bite-size digestible chunks, allowing them to concentrate on certain aspects of a trial, improving clarity and understanding. And the clinical researchers will have two “legitimate” publications since they report on two substantive aspects of the research that need and deserve top be considered carefully and separately. Then the inferences form both aspects can be incorporated to make an informed decision. This solution would allow us to thumb our noses at the system and partially satisfy the monster that insists on quantity of research. Yet simultaneously you will be presenting to our colleagues and patients the complete and comprehensible package of results, enabling them to make informed clinical and policy decisions based on the quality research. Your worthy aspirations fulfilled. Yours truly, Dr Mike Hurley
Competing interests: None declared |
|||
|
|
|||
|
Dawn Craig, Research Fellow in Health Economics NHS Centre for Reviews and Dissemination, University of York, York YO10 5DD, UK.,, John Nixon, Nigel Armstrong, Julie Glanville, Michael Drummond, and Jos Kleijnen.
Send response to journal:
|
The NHS Economic Evaluation Database (NHS EED) research team agree that economic evaluations should contain comprehensive reporting of both clinical effectiveness and economic analysis, and that the BMJ is right to implement this new policy. How the clinical trial results (which inform the economic evaluation) are obtained is often paramount to the understanding and quality of the economic analysis conducted.(1) Research reports are included and abstracted in full on the NHS Economic Evaluation Database -if they, inter alia, explicitly report costs and clinical outcomes for an intervention and at least one comparator.(2) However, in order to appropriately critique the methodology adopted in the effectiveness study underpinning the economic evaluation, our template requires information that is often omitted in the report of the economic evaluation. To overcome this problem, where the ‘parent’ clinical study has been previously published elsewhere we obtain this study and use that alongside the economic research when writing the abstract – this can then provide information on sample selection, study design, method of analysis and so on.. When this occurs, NHS EED abstracts indicate that the relevant information is cited from the parent study. Adhering to published guidelines, such as those provided by the BMJ(3) should produce the highest quality publications, but, it is likely that authors may still feel the need to be selective in their reporting, given word limits. If authors are required to report more effectiveness data other vital aspects of the economic evaluation might receive less attention. The focus for BMJ editors should be to ensure that reporting of both important components of economic evaluations receive appropriate attention from authors. If the BMJ policy will result in full reporting of both clinical and economic results in one place (e.g. two papers in one issue of the journal) this constitutes an improvement. If, however, the new policy results in the combination of clinical and economic results in one short paper, this may be a step backwards. References 1) Hoffmann C, Stoykova B, Nixon J, Glanville J, Misso K and Drummond M (2002) Do Health-Care Decision Makers Find Economic Evaluations Useful? The Findings of Focus Group Research in UK Health Authorities, Value in Health and, 5, 71-78. 2) NHS CRD (2001) Improving Access to Cost-effectiveness Information for Health Care Decision-making: The NHS Economic Evaluation Database, CRD Report No 6 (2nd Ed), University of York. 3) Drummond MF and Jefferson TO (1996) Guidelines for authors and peer reviewers of economic submissions to the BMJ, BMJ, 313, 275-83. Competing interests: None declared |
|||
|
|
|||
|
Christopher J McCabe, Senior Lecturer in Health Economics University of Sheffield, Regent Court, 30 Regent Street Sheffield, S1 4DA, Jennifer Roberts
Send response to journal:
|
Dear Sir We have read with interest the emerging debate ensuing from the new policy on publication of economic evaluations. A number of important practical points have been made, however we feel that some broader strategic issues still remain. Thankfully major research funders in the United Kingdom increasingly require that economic evaluation is an integral part of a clinical trial design. In many ways the BMJ's decision is the natural extension of this philosophy. However, there is a risk that unless other major journals follow suit the policy will damage the dissemination of cost-effectiveness information. It has to be acknowledged that the collaboration between clinicians and health economists is often a delicate one. The pressures that this new policy will place upon these relationships will have implications for long -term cooperation. Given the pressure to publish rapidly in high impact journals, to secure long-term funding, the interests of clinical and economic researchers will not always coincide. An immediate effect of this policy for those of us involved in multi- disciplinary research is the need to agree publication strategies at the outset of a project. It may be that a process similar to the Lancet's protocol pre-approval could facilitate these discussions by providing confidence that high quality clinical trials including an economic evaluation will be acceptable to the major journals. Unless all researchers accept the need for the simultaneous publication of clinical and economic results, cost effectiveness information may be confined to more specialist journals, which are rarely seen by the clinical community. The best solution would be for the major journals to agree that clinical trials designed to inform policy decisions must include a high quality economic evaluation. We would be interested to know the Lancet's thoughts on this issue. Competing interests: It is important for our research careers to be able to publish in high impact journals. |
|||
|
|
|||
|
Satyajit Nag, Research Fellow in Diabetes James Cook University Hospital,Marton Road,Middlesbrough TS4 3BW
Send response to journal:
|
Smith's editorial on the new BMJ policy on economic evaluations discusses the merits of publishing clinical results and economic studies from randomized controlled trials together in the same journal(1). Between the lines however an undercurrent of resentment is obvious. Smith's admission of 'petulance' on the part of the BMJ appears to stem from the fact that authors are not sending the clinical results from high quality randomized controlled trials to the BMJ but are using the journal to report the results of secondary studies like economic evaluations. Is the BMJ peeved that a premier medical journal like the Lancet is getting to publish the clinical results of landmark trials? A number of questions will test how robust the new policy is. Would the BMJ now require authors to make a declaration whether an economic evaluation is underway? Is the BMJ going to refuse publication of the clinical paper if the economic evaluation is sent elsewhere? Authors surely, should have the freedom to submit papers for peer review with no strings attached. It is debatable whether editorial policy should dictate this. Conflict of interests: None References 1.Smith R, New BMJ Policy on Economic Evaluations: we won't publish economic evaluations unless offered the clinical results as well. Editorial. BMJ 2002; 325:1124 Competing interests: None declared |
|||
|
|
|||
|
Alastair M Gray, Professor of Health Economics University of Oxford OX3 7LF, Andrew Briggs and Philip Clarke
Send response to journal:
|
Health economists have been grateful for the BMJ's hitherto supportive stance towards the publication of economic evaluations. The proposed new policy not to publish economic evaluations unless also offered the clinical results is disappointing and misjudged. First, this policy denies the fact that, while clinical and economic results from a trial are both components of an overall evaluation, they also have many differences, frequently including the funding agencies supporting them, the researchers involved, and the timescale over which they are performed and published. Perhaps most importantly, important trials are often prepared for an international audience, but economic evaluations normally relate to specific health care systems; indeed large trials may generate the need for several country-specific economic evaluations. These differences justify researchers in choosing to submit clinical and economic results to different journals, and entitle journals to use different criteria when deciding whether to publish or reject. Consequently, as in all other fields, research findings that are closely related and possibly interdependent often appear in different journals. That poses no great problem to readers, especially in the era of electronic publication championed by the BMJ . Second, what is the likely effect of this policy? Researchers aim to publish where they judge they make most impact. Surely no-one will forego an opportunity to publish trial results in the Lancet simply because the BMJ will not then consider publishing an economic evaluation. Richard Smith's editorial included no positive proposals to make the BMJ a more attractive outlet for trial results. Instead, this policy will inevitably mean turning away well-conducted empirical research - such as the economic analysis of the multicentre aneurysm screening study that occasioned this announcement - on strictly non-scientific grounds. Arguably these are precisely the more scientifically important papers, leaving the BMJ with a greater preponderance of non-trial based economic analyses and data-free "think-pieces". This is hardly the route to improving the journal's impact on decisions regarding the adoption of new therapies or technologies. Richard Smith admits this new policy owes something to petulance, but nevertheless defends it as reasonable. We think it is unreasonable and ask him to reconsider. Competing interests: We are currently involved in economic analyses of several large trials whose clinical results have recently been published in other journals including the Lancet. If adopted, this policy will deny us the opportunity to have our scientific research results considered for publication by the BMJ. |
|||
|
|
|||
|
Richard Smith, Editor BMJ
Send response to journal:
|
We are grateful to everybody who has responded to our proposal to consider for publication economic evaluations that accompany clinical papers only if we are sent both the economic and the clinical papers. More respondents are in favour than against, but people have raised important questions that we must answer. This is a further clarification of our policy. 1. If both the clinical and the economic paper are submitted to us we might accept one and not the other. 2. We will be willing to consider for publication economic papers that are submitted some time after the clinical paper if the clinical paper was also submitted to us. It will not matter if we decided not to publish the clinical paper. We might still be willing to publish the economic paper. 3. We will not reject clinical papers if they are not accompanied by an economic evaluation. There would be a logic to such a policy, but we are first and foremost a clinical journal. 4. We will be willing to consider either papers that combine clinical and economic results or pairs of papers. Pairs of papers will usually be better. Our ELPS (electronic long, paper short) policy means that we can publish long papers on bmj.com-longer than 2000 words. (1) We prepare the shorter version for the paper edition of the BMJ. Authors approve it before publication. If we take two papers then we will publish them together. 5. We will continue to consider for publication economic papers, perhaps modelling papers, that are not related to particular clinical papers. 1 Müllner M, Groves T. Making research papers in the BMJ more accessible. BMJ 2002; 325: 456. Competing interests: RS is the editor of the BMJ and the chief executive of the BMJ Publishing Group. Journals make substantial sums from selling reprints. If the BMJ is sent more trials as a result of this policy the BMJ, but not RS, will benefit financially. If researchers take umbrage and send us fewer trials then the BMJ will lose financially. |
|||
|
|
|||
|
Richard Smith, Editor BMJ
Send response to journal:
|
Respondents to this debate, most of them economists, ask if we can provide incentives for clinical authors to publish in the BMJ rather than the Lancet despite the Lancet having a higher impact factor. This allows me to indulge in a favourite pastime, but before I do so I want to make clear that I'm a fan of the Lancet and a friend of Richard Horton, its editor. We are having lunch together today (Friday). Ten reasons to publish in the BMJ rather than the Lancet 1. The BMJ reaches virtually all doctors in Britain. The Lancet reaches only a small fraction. 2. Virtually no general practitioners read the Lancet. Anything that has a message that is for general practitioners or primary care should be in the BMJ. 3. The BMJ has many more readers than the Lancet, even in the United States, where the Lancet has traditionally been stronger. The print run of the BMJ is 110 000. bmj.com has almost a million visitors a month, many of them from the US. BMJUSA, a monthly compilation from the weekly BMJ, is sent to 100 000 American doctors. Local editions of the BMJ--in China, South Asia, the Middle East, West Africa, and other countries--reach many tens of thousands more. 4. bmj.com is free to everybody everywhere. 5. Original research published in the BMJ goes straight into Pubmed Central, where people can access the full text directly. 6. The BMJ has a better peer review system. This is a bold claim and not very evidence based, but as well as being open the BMJ peer review process concentrates on adding value to the papers we publish. At least six people read every word of every paper and concentrate on how the paper can be improved. 7. Our ELPS (electronic long, paper short) process means that authors can simultaneously provide large amounts of material for research colleagues and reach out to a more general audience through the short paper version prepared by editors. 8. The BMJ allows authors without asking to post their papers on their own websites or the websites of their institution and to use them for any non-commercial purpose--for example, teaching. 9. If we sell reprints for more than a thousand pounds we give 10% of the revenue to the authors. Some reprints sell for hundreds of thousands of pounds. 10. The profits of the BMJ go back to the BMA, which is owned by 120 000 British doctors. The profits of the Lancet go to Reed-Elsevier, a large multinational who are unpopular with librarians and last year made an adjusted operating profit of £990m. Richard Smith, Editor, BMJ 1 Müllner M, Groves T. Making research papers in the BMJ more accessible. BMJ 2002; 325: 456. Competing interests: RS is the editor of the BMJ and the chief executive of the BMJ Publishing Group. Journals make substantial sums from selling reprints. If the BMJ is sent more trials as a result of this policy the BMJ, but not RS, will benefit financially. If researchers take umbrage and send us fewer trials then the BMJ will lose financially. |
|||
|
|
|||
|
Richard Horton, Editor The Lancet
Send response to journal:
|
In Confidence Dear Richard - I hope you got home safely. After that bottle of wine, most of which seemed to pass me by (but thanks for the fizzy water), the last I saw of you was a faint figure wobbling into the mists of London on a rickety bike. Is that what the BMJ means by fast-track? I can't remember what we talked about now, but I'm sure one of us tried to bribe the other. I think I offered you £1000 if you promised to launch a BMJ poll when the Queen offers you a knighthood. You agreed didn't you? You thought £1000 was rather pathetic, but then think of the glory. I like your reasons to publish in the British Medical Journal. Perhaps we can send you all the papers we cannot publish. Here are 10 reasons why you might like to publish your next research paper in The Lancet rather than the BMJ. I'd prefer it if you kept this list private for the time being. Authors might flock to us if they knew, and I don't want to hurt the BMJ. 1. The Lancet is a truly global journal, publishing far more work that reflects the real burden of disease among the world's population - so it's a great place to hang out if you are a serious scientist. 2. The BMJ is distributed, but The Lancet is used - it has articles downloaded more often than any other journal on ScienceDirect, the largest database of journals that libraries currently can hold. 3. The Lancet provides the best service to authors for peer-reviewing and publishing research quickly (fast track) and in multiple usable formats (early on line). 4. We try to eliminate publication bias by committing ourselves to publication before a study is completed - our protocol reviews facility (see www.thelancet.com). 5. Editors make mistakes. We are the only journal to have instituted a formal appeals system to challenge our original decisions. 6. Editors have too much power and they sometimes mis-use it. We were the first journal to appoint an independent ombudsperson to act as an arbiter when authors and readers are unhappy with the way editors have handled a complaint. About half of all complaints against us are upheld. This helps us do a better job. 7. Impact factor - it's a crazy thing to measure and compare, but The Lancet is higher than the BMJ. I'm sorry. 8. Our peer review meetings are much better, and certainly more fun, than those at the BMJ. I cannot prove this statement, of course, but anybody can come along and see for themselves. Just let me know if you do want to come -they are on a Thursday at 2pm. You can visit in the morning to read papers and stay for the debate in the afternoon. 9. Editors at The Lancet don't publish their own research in their own journal - this seems to us a fatal conflict of interest. But we can understand why you do it... 10. The owners of journals - mmmm. The Lancet is owned by Reed- Elsevier, while the BMA has something to do with the BMJ - although, Richard, you have done well to keep the out-of-touch (consultant contract etc) buffoons of BMA House out of your editorial office. Which is the more difficult? Being owned by a political organisation that is a gravy train for the great and the good, or being owned by a publisher that lets us do what we need to do and recognises that a profitable journal is our best guarantee of editorial independence? Anyway, have a great holiday and New Year - lunch is on me in 2003! Best wishes, Richard Competing interests: RH works at The Lancet. He could benefit from publishing clinical papers, but that only means that he can then afford to buy RS lunch next time round. |
|||