Intended for healthcare professionals

Education And Debate Getting research findings into practice

Finding information on clinical effectiveness

BMJ 1998; 317 doi: (Published 18 July 1998) Cite this as: BMJ 1998;317:200
  1. Julie Glanville (jmg1{at}, information service managera,
  2. Margaret Haines, principal adviserb,
  3. Ione Auston, librarianc
  1. aNHS Centre for Reviews and Dissemination, University of York, York Y01 5DD
  2. bLibrary and Information Commission, London W1V 4BH
  3. cNational Information Center on Health Services Research and Health Care Technology, National Library of Medicine, 8600 Rockville Pike, Bethesda, MD 20894, USA
  1. Correspondence to: Ms Glanville

    This is the third in a series of eight articles analysing the gap between research and practice

    Series editors: Andrew Haines and Anna Donald

    There is increasing pressure on healthcare professionals to ensure that their practice is based on evidence from good quality research, such as randomised controlled trials or, preferably, systematic reviews of randomised controlled trials and trials of other study designs. This pressure comes from various sources. The evidence based healthcare movement encourages a questioning and reflective approach to clinical practice and emphasises the importance of lifelong learning. Thus, good access to research based evidence is necessary. Many governments are encouraging the development of evidence based medicine because its advantages are understood, especially in terms of improved efficiency in the delivery of health care through the identification of effective treatments.12 There are also indications that legal decisions may take account of whether research evidence and clinical guidelines were adhered to.34 Better informed consumers may provide another incentive for clinicians to be more aware of research findings. Clinicians will need to be able to access information on clinical effectiveness in order to improve the quality of care and to stay well informed on developments in specialist areas. We examine the resources that are already available to clinicians, strategies for finding and filtering information, and ways of improving dissemination.

    • Information alone is often not sufficient to encourage changes in practice

    • A national dissemination strategy for important research messages combined with local support mechanisms may increase the uptake of changes in practice

    • All healthcare decision makers need to know how to filter research for quality and how to appraise evidence from research

    • Extensive information on clinical effectiveness is already available, and computer based systems are being developed that will present clinicians with evidence based information when they need it

    • Good library and information support provided to doctors has been proved to make a positive impact on clinical decision making

    Evidence based information already

    In the 1990s evidence from research has become more easily available. In part this has been due to the development of programmes for assessing health technology and to the growth in systematic reviews. Systematic reviews evaluate primary evidence and the effectiveness of particular interventions. They necessarily take time to complete but a useful compilation of reviews is available in The Cochrane Library and there are also reports from technology assessment agencies such as the Agency for Health Care Policy and Research in the United States or in England the Department of Health's health technology assessment programme. The publications and databases in the box present evidence on effectiveness, often in a summarised form suitable for the busy clinician or policymaker. However, important problems remain, such as how to increase awareness of what information is available and how to provide clinicians with information when they need it.

    Selected resources

    The Cochrane Library
    • A collection of databases including the full text of the Cochrane Database of Systematic Reviews, critical commentaries on selected systematic reviews that have been assessed for quality by the NHS Centre for Reviews and Dissemination, and brief details of more than 170 000 randomised controlled trials.

    • Available from: Update Software, Summertown Pavilion, Middle Way, Summertown, Oxford, OX2 7LG, or and

    Clinical Guidelines from the US Agency for Health Care Policy and Research
    • A series of clinical guidelines based on thorough reviews of research evidence. The agency is now focusing on producing evidence reports (reviews and analyses of scientific literature designed to provide the basis for guidelines, measures of performance, and other tools for quality improvement), as well as working with the American Medical Association and the American Association of Health Plans to develop an online clearing house for practice guidelines; the online service will have electronic mailing lists to keep users informed about the implementation of guidelines.

    • Available from: and

    Best Evidence Database on CD ROM
    • Abstracts of primary and review articles that have been published in the American College of Physicians Journal Cluband Evidence-Based Medicine, with assessments of quality by clinical experts.

    • Available from: BMJ Publishing, London WC1H 9JR, or

    Best Evidence Database on CD ROM
    • Reports of systematic reviews presented in a readable and accessible format, produced by the NHS Centre for Reviews and Dissemination.

    • Available from: Subscriptions Department, Pearson Professional, PO Box 77, Fourth Avenue, Harlow CM19 5BQ, or

    Guide to Clinical Preventive Services, 2nd ed
    • US Preventive Services Task Force. Baltimore: Williams and Wilkins, 1996

    • Evidence based recommendations on preventive services.

    • Available from:

    Canadian Guide to Clinical Preventive Health Care
    • Ottawa: Health Canada, 1994

    • Evidence based recommendations on preventive services.

    Drug and Therapeutics Bulletin
    • Independent assessments of drugs and other treatments.

    • Available from: Consumers' Association, Castlemead, Gascoyne Way, Hertford, SG14 1LH

    Effectiveness Matters
    • Summaries of published research on a single topic which emphasise presenting clear messages on effectiveness.

    • Available from: NHS Centre for Reviews and Dissemination, University of York, York Y01 5DD, or

    MeReC Bulletin
    • Reviews of new drugs compiled for general practitioners, with discussion of effectiveness, safety, appropriateness, acceptability, and cost.

    • Available from: Medicines Resource Centre, Hamilton House, 24 Pall Mall, Liverpool L3 6AL

    NHS Economic Evaluation Database
    • Critical assessments of published economic evaluations, produced by the NHS Centre for Reviews and Dissemination.

    • Available from: NHS Centre for Reviews and Dissemination, University of York, York Y01 5DD, or

    Collections of systematic reviews and critical appraisals of primary research are valuable sources of evaluated research. The proliferation of these collections is creating its own information explosion; this is a serious problem that needs to be addressed. Because there is no single comprehensive index to all the material described in the box several searches through both paper journals and electronic services may be required to locate relevant information. It may also be necessary to obtain copies of the original publication. These are disincentives to searching for and obtaining research evidence. Information technology may eventually provide a more streamlined way of dealing with this explosion of information, perhaps in the form of world wide web interfaces that provide links to a range of evidence based information services that filter publications for quality, or by providing access to the full text of publications. Biomednet is one model of this type of service. It offers a range of full text resources with free Medline access, discussion facilities, and virtual meeting rooms. Biomednet is beginning to highlight important papers that have been cited and evaluated by expert reviewers as a means of filtering papers for quality.5

    The resources in the box provide information that has been evaluated and filtered–that is, they highlight the best quality studies from the mass of available literature. However, research based answers to many questions of effectiveness are not yet available in such time saving, value added forms. Clinicians may still need to search indexes and abstracts of published literature. For several years it has been possible for clinicians to search Medline using software such as Grateful Med, and its world wide web interface, internet Grateful Med. This has provided access to a large body of peer reviewed studies that are mostly unsynthesised and unevaluated. There are tools to help searchers identify the types of studies that are more likely to provide high quality information on clinical effectiveness, such as systematic reviews or randomised controlled trials.67 Once the original papers have been retrieved there are checklists that, together with training in critical appraisal skills, can be used to assess the rigour and validity of such studies.810

    Although Medline is a rich resource, access is increasingly required to a wider range of material than it presently indexes. The US National Library of Medicine and the American Hospital Association have recently launched the HealthSTAR database which seeks to provide expanded access to both non-clinical information (on topics such as healthcare administration, economics, and planning) and non-journal information (such as reports, meeting abstracts, and chapters from books) that is not available in journals.11 The National Library of Medicine has recently announced that access to Medline and HealthSTAR through internet Grateful Med and access to Medline through the PubMed interface will be free.11 Other databases that cover specific clinical areas, specific types of publications, and non-English language material should also be used. Tools such as search strategies and single interfaces, like PubMed, are required to enhance access to a range of such databases.

    Strategies for finding and filtering information

    Training and practice are required to search information services and navigate the internet effectively, but other options are available which may help clinicians cope with the challenges of finding information. Locating, appraising, and exploiting resources, both print and electronic, has typically been the role of the librarian or information professional. Increasingly, clinicians are finding that librarians can not only help them locate information in answer to a particular question but also can help to keep their knowledge up to date by presenting selections of important new evidence in the form of paper or electronic bulletins.

    The value of library and information support has been demonstrated on both sides of the Atlantic. Trained librarians are often more effective than physicians in filtering papers for quality.12 Some American studies have shown that library support not only contributes to lower patient care costs by decreasing the number of admissions, length of stay, and number of procedures but also contributes to a higher quality of care in terms of patient advice, improved decision making, and savings in time. 1314 A similar study in the United Kingdom found that library services had a positive impact on the continuing education of hospital doctors.15

    Not all clinicians have the time to visit libraries, and new models have emerged for delivering library support directly to hospital wards and departments.1617 In the United States, the National Network of Libraries of Medicine provides outreach services to general practitioners (and, more recently, to public health professionals); in the United Kingdom the BMA library offers an electronic outreach service to members. 1819 Also in the United Kingdom, the Oxford PRISE (primary care sharing the evidence) project is developing a model whereby general practitioners' computers are linked to a central computer that provides access to a range of databases; in this model the general practitioners can also request librarians to follow up particular questions in more detail.20 Librarians are increasingly asked to provide training in information skills as part of courses in evidence based medicine offered to NHS staff.

    The development of primary care based services presents a challenge to librarians; they must become better trained to deal with a wider range of inquiries, to evaluate and synthesise evidence, and to present selected information through innovative delivery systems. Clearly initiatives such as the Oxford Health Libraries' training programme, known as the “librarian of the 21st century,” is a model for other library networks.21 Similar initiatives under development in the United States include the National Information Center on Health Services Research and Health Care Technology, web based training materials that are not copyrighted and can be modified to suit the user,22 and training programmes for librarians sponsored by the National Library of Medicine in subjects such as medical informatics.

    Improving dissemination

    Embedded Image

    For information to be accessible it must be packaged and published in formats that promote easy identification and encourage use. Evidence based information is becoming easier to find: structured abstracts in articles in journals make it easier to identify the methodology of a study and its potential reliability. Innovations, such as the BMJ's key messages boxes, make it easier to identify the important points of research. Journal editors have an important role in encouraging authors to provide informative abstracts and in ensuring that researchers' conclusions are supported by their paper's results. However, the benefits of clearer labelling may be undermined if current buzz words, such as “effective” and “evidence based,” are adopted and used incorrectly or inaccurately so that previously useful labels become meaningless.

    Organisations that produce recommendations on policy and clinical guidelines are finding it necessary to make their guidelines more explicitly evidence based, both by using research evidence to develop their guidelines and in stating the level of evidence on which the guidance is based.2324 It would be easier and quicker to assess guidelines if the types of evidence used in their development were stated as clearly as possible, for example on the front cover of published guidelines there could be a statement to the effect that “this guideline is based on a Cochrane review.” The guideline appraisal project of the Health Information Research Unit at McMaster University is an example of efforts to help practitioners identify and critically evaluate clinical guidelines, and to determine their applicability to local practice.25

    Information from research needs to be presented in forms that are appropriate for the target audience. Guidelines from the Agency for Health Care Policy and Research have been packaged in different ways for different users; they have been packaged as a detailed report of the review with a full exposition of the evidence for researchers and decision makers, as a briefer guideline for clinicians, and as a leaflet for patients. In the United Kingdom, the Midwives Information and Resource Service has produced a series of leaflets aimed at both pregnant women and their professional carers using, when possible, evidence from Cochrane reviews.26

    Simply presenting research evidence to clinicians is often insufficient to ensure that it is incorporated into practice. Government directives and direct incentives such as payments can increase the speed of uptake. Sometimes powerful research findings will have an immediate effect; swift changes in practice followed the publication of research findings that sleeping position could affect mortality from the sudden infant death syndrome.2728 However, even when findings are packaged, summarised, and made relevant to clinicians further action will be needed to ensure their implementation.

    A complex set of factors influences the uptake of research findings, and a variety of dissemination methods need to be used to encourage clinicians to make informed changes in their practice.29 Much research on effective implementation is currently under way, but a nationally coordinated strategy to disseminate and promote important evidence from research and systematic reviews could improve implementation among healthcare professionals. National campaigns to distribute information packs, briefings, and videos of important points from research findings could speed the wider adoption of changes in practice. Such national campaigns would need to be complemented by a variety of other activities at a local level.30 Local implementation strategies involving continuing education programmes, patient education programmes, and library and information outreach services could be coordinated to ensure that key research evidence is not only accessible but also acted on.

    • The articles in this series are adapted from Getting research findings into practice, edited by Andrew Haines and Anna Donald, which is published by the BMJ Publishing Group.


    Helpful comments were provided by Olwen Jones, Susan Mottram, Ian Watt, Trevor Sheldon, Andrew Jones, and the two referees for this paper.

    Funding: None.

    Conflict of interest: None.


    View Abstract